Article Sidebar

Download
PDF
Statistic
Read Counter : 72 Download : 9

Main Article Content

Abstract

Purpose: This study compares the performance of conventional machine learning algorithms for multiclass emotion prediction in reviews of Large Language Model (LLM)-powered applications using a unified TF-IDF representation.


Research Method: User reviews were collected from the Google Play Store for eight popular LLM applications through web scraping. After text preprocessing, reviews were labeled into five emotion categories (Happy, Sadness, Anger, Fear, and Surprise). Ten machine learning algorithms, namely Perceptron, KNN, Naïve Bayes, Logistic Regression, MLP, SVM (Linear and RBF), Random Forest, XGBoost, and LightGBM, were evaluated using Accuracy, Balanced Accuracy, Precision, F1-score, MCC, and ROC-AUC.


Result and Discussion: XGBoost achieved the best performance, with 87.30% Accuracy, 77.84% Balanced Accuracy, 64.28% F1-score, and 0.5626 MCC. ROC-AUC values between 0.91 and 0.96 demonstrate excellent discrimination across all emotion classes. The superior performance of XGBoost indicates that gradient boosting effectively captures informative patterns from sparse TF-IDF features.


Implication: The findings provide practical guidance for selecting efficient machine learning models for emotion analysis of LLM application reviews and support AI application improvement through user emotion understanding.


Originality: This study presents a comprehensive benchmark of ten conventional machine learning algorithms using a large-scale multi-application LLM review dataset and a unified five-emotion classification framework.

Keywords

Emotion prediction Large language models Machine learning Text classification TF-IDF

Article Details

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite
Taju, S. W., William Tangka, G. M., & Chrisanti, I. R. . (2026). Comparative Analysis of Machine Learning Models for Emotion Prediction in Reviews of Large Language Model-Powered Applications. Advances in Human Resource Management Research, 4(3), 404–422. https://doi.org/10.60079/ahrmr.v4i3.1008
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX

References

  1. Acheampong, F. A., Wenyu, C., & Nunoo-Mensah, H. (2020). Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports, 2(7), e12189. https://doi.org/10.1002/eng2.12189
  2. Adinath, D., & IS, S. (2025). ChatGPT-5 and Grok-4: A comparative analysis of advancements in NLP. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.5547358
  3. Ahmed, I., Islam, S., Datta, P. P., Kabir, I., Chowdhury, M. N. U. R., & Haque, A. (2025). Qwen 2.5: A comprehensive review of the leading resource-efficient LLM with potential to surpass all competitors. https://doi.org/10.36227/techrxiv.174060306.65738406/v1
  4. Al Maruf, A., Khanam, F., Haque, M. M., Jiyad, Z. M., Mridha, M. F., & Aung, Z. (2024). Challenges and opportunities of text-based emotion detection: A survey. IEEE access, 12, 18416-18450. https://doi.org/10.1109/ACCESS.2024.3356357
  5. Al-Obaydy, W. I., Hashim, H. A., Najm, Y. A., & Jalal, A. A. (2022). Document classification using term frequency-inverse document frequency and K-means clustering. Indonesian Journal of Electrical Engineering and Computer Science, 27(3), 1517-1524.
  6. Alam, M. S., Mrida, M. S. H., & Rahman, M. A. (2025). Sentiment analysis in social media: How data science impacts public opinion knowledge integrates natural language processing (NLP) with artificial intelligence (AI). American Journal of Scholarly Research and Innovation, 4(01), 63-100. https://doi.org/10.63125/r3sq6p80
  7. Ali, Z. A., Abduljabbar, Z. H., Tahir, H. A., Sallow, A. B., & Almufti, S. M. (2023). eXtreme gradient boosting algorithm with machine learning: A review. Academic Journal of Nawroz University, 12(2), 320-334. 10.25007/ajnu.v12n2a1612
  8. Asniwati, A., & Latief, F. (2026). The Influence of Interpersonal Communication and Workplace Friendships on Employee Job Satisfaction in Public Organizations. Advances in Human Resource Management Research, 4(2), 185-200. https://doi.org/10.60079/ahrmr.v4i2.748
  9. Bahasoan, S., Asniwati, A., & Surianto, S. (2026). Synergy Between Training and Digital Capabilities: Transforming Competencies to Achieve Competitive Advantage for Retail Apparel SMEs. Advances in Human Resource Management Research, 4(1), 89-111. https://doi.org/10.60079/ahrmr.v4i1.754
  10. Bentéjac, C., Csörgő, A., & Martínez-Muñoz, G. (2019). A comparative analysis of XGBoost. arXiv. https://arxiv.org/abs/1911.01914
  11. Berrar, D. (2018). Bayes' theorem and naive Bayes classifier. In Encyclopedia of Bioinformatics and Computational Biology (pp. 403–412).
  12. Chatterjee, A., Gupta, U., Chinnakotla, M. K., Srikanth, R., Galley, M., & Agrawal, P. (2019). Understanding emotions in text using deep learning and big data. Computers in Human Behavior, 93, 309-317. https://doi.org/10.1016/j.chb.2018.12.029
  13. Chen, Q., Chen, C., Hassan, S., Xing, Z., Xia, X., & Hassan, A. E. (2021). How should i improve the ui of my app? a study of user reviews of popular apps in the google play. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(3), 1-38. https://doi.org/10.1145/3447808
  14. Christian, H., Agus, M. P., & Suhartono, D. (2016). Single document automatic text summarization using term frequency-inverse document frequency (TF-IDF). ComTech: Computer, Mathematics and Engineering Applications, 7(4), 285-294.
  15. Cui, J., Wang, Z., Ho, S. B., & Cambria, E. (2023). Survey on sentiment analysis: evolution of research methods and topics. Artificial intelligence review, 56(8), 8469-8510. https://doi.org/10.1007/s10462-022-10386-z
  16. Deng, J., & Ren, F. (2021). A survey of textual emotion recognition and its challenges. IEEE Transactions on Affective Computing, 14(1), 49-67. https://doi.org/10.1109/TAFFC.2021.3053275
  17. Fakhri, M., & Silvianita, A. (2026). Servant Leadership and Employee Engagement Among Generation Z: Examining the Mediating Role of Mental Well-Being. Advances in Human Resource Management Research, 4(2), 219-238. https://doi.org/10.60079/ahrmr.v4i2.772
  18. Fu, B., Lin, J., Li, L., Faloutsos, C., Hong, J., & Sadeh, N. (2013, August). Why people hate your app: Making sense of user feedback in a mobile app store. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1276-1284).
  19. Gibney, E. (2025). Chinese AI model Kimi K2 excites researchers. Nature, 643, 889.
  20. Hartati, H., Setiawan, A., Alyyasa-Gan, S. S., & Machmud, M. (2026). The Influence of Leadership and Motivation on Performance in High-Discipline Organizations: A Study of Brimob Personnel. Advances in Human Resource Management Research, 4(1), 55-71. https://doi.org/10.60079/ahrmr.v4i1.749
  21. Hidayat, N., Mojolou, D. N., Madli, F., Lada, S., & Loong, A. H. (2026). The Role of Trust and Knowledge Sharing in Encouraging Innovative Work Behavior: An Empirical Study of SMEs in Tarakan City. Advances in Human Resource Management Research, 4(1), 1-13. https://doi.org/10.60079/ahrmr.v4i1.733
  22. Jakkula, V. (2006). Tutorial on support vector machine (SVM). School of EECS, Washington State University.
  23. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., ... & Liu, T. Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30.
  24. Kosasih, K., Yesika, M., Lustina, F. A., Rusdiana, E., & Tansil, A. R. (2026). Artificial Intelligence in Hospital Human Resource Management: A Systematic Review and Bibliometric Analysis. Advances in Human Resource Management Research, 4(2), 267-285. https://doi.org/10.60079/ahrmr.v4i2.776
  25. Latif, R. M. A., Abdullah, M. T., Shah, S. U. A., Farhan, M., Ijaz, F., & Karim, A. (2019). Data scraping from Google Play Store and visualization of its content for analytics. In Proceedings of the 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET) (pp. 1–8). IEEE.
  26. Liu, J., Zhao, X., Shang, X., & Shen, Z. (2026). Dive into Claude Code: The design space of today's and future AI agent systems. arXiv. https://arxiv.org/abs/2604.14228
  27. Liu, Y., Wang, Y., & Zhang, J. (2012, September). New machine learning algorithm: Random forest. In International conference on information computing and applications (pp. 246-252). Berlin, Heidelberg: Springer Berlin Heidelberg.
  28. Mäntylä, M. V., Graziotin, D., & Kuutila, M. (2018). The evolution of sentiment analysis—A review of research topics, venues, and top cited papers. Computer science review, 27, 16-32. https://doi.org/10.1016/j.cosrev.2017.10.002
  29. Nanli, Z., Ping, Z., Weiguo, L., & Meng, C. (2012). Sentiment analysis: A literature review. In Proceedings of the International Symposium on Management of Technology (ISMOT) (pp. 572–576). IEEE.
  30. Naseem, S., Mahmood, T., Asif, M., Rashid, J., Umair, M., & Shah, M. (2021). Survey on sentiment analysis of user reviews. In Proceedings of the 2021 International Conference on Innovative Computing (ICIC) (pp. 1–6). IEEE.
  31. Nurinaya, N., & Marhumi, S. (2025). Leadership strategy evaluation and succession planning in preparing the company’s future leaders. Advances in Human Resource Management Research, 3(1), 1-14. https://doi.org/10.60079/ahrmr.v3i1.405
  32. Nusinovici, S., Tham, Y. C., Yan, M. Y. C., Ting, D. S. W., Li, J., Sabanayagam, C., ... & Cheng, C. Y. (2020). Logistic regression was as good as machine learning for predicting major chronic diseases. Journal of clinical epidemiology, 122, 56-69. https://doi.org/10.1016/j.jclinepi.2020.03.002
  33. Saeidnia, H. R. (2023). Welcome to the Gemini era: Google DeepMind and the information industry. Library Hi Tech News.
  34. Sharma, N. A., Ali, A. S., & Kabir, M. A. (2025). A review of sentiment analysis: tasks, applications, and deep learning techniques. International journal of data science and analytics, 19(3), 351-388. https://doi.org/10.1007/s41060-024-00594-x
  35. Stratton, J. (2024). An introduction to Microsoft Copilot. In Copilot for Microsoft 365: Harness the power of generative AI in the Microsoft apps you use every day (pp. 19–35). Springer.
  36. Tan, K. L., Lee, C. P., & Lim, K. M. (2023). A survey of sentiment analysis: Approaches, datasets, and future research. Applied Sciences, 13(7), 4550. https://doi.org/10.3390/app13074550
  37. Venkatakrishnan, S., Kaushik, A., & Verma, J. K. (2020). Sentiment analysis on google play store data using deep learning. In Applications of Machine Learning (pp. 15-30). Singapore: Springer Singapore.
  38. Vinodhini, G., & Chandrasekaran, R. M. (2016). A comparative performance evaluation of neural network based approach for sentiment classification of online reviews. Journal of King Saud University Computer and Information Sciences, 28(1), 2-12. https://doi.org/10.1016/j.jksuci.2014.03.024
  39. Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023). A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5), 1122–1136.
  40. Yue, L., Chen, W., Li, X., Zuo, W., & Yin, M. (2019). A survey of sentiment analysis in social media: L. Yue et al. Knowledge and information systems, 60(2), 617-663. https://doi.org/10.1007/s10115-018-1236-4
  41. Zhai, Y., Song, X., Chen, Y., & Lu, W. (2022). A study of mobile medical app user satisfaction incorporating theme analysis and review sentiment tendencies. International journal of environmental research and public health, 19(12), 7466. https://doi.org/10.3390/ijerph19127466
  42. Zhang, J., Li, D., Wang, X., Wu, Z., & Lyu, Q. (2025). Application and challenges of DeepSeek in primary care in China. Frontiers in public health, 13, 1721308. http://dx.doi.org/10.3389/fpubh.2025.1721308
  43. Zhang, Z. (2016). Introduction to machine learning: k-nearest neighbors. Annals of translational medicine, 4(11), 218.