Imbalance-Aware Evaluation and Hyperparameter Optimization of Supervised Machine Learning Models for Credit Card Fraud Detection

The financial sector is one of the industries where credit card fraud detection is a critical issue because the number of legitimate transactions is by far outnumbered by the number of fraud transactions carried out. This paper performs an imbalance-sensitive analysis and hyperparameter optimization of three supervised machine learning (SML) models (Logistic Regression, Random Forest, and XGBoost) on the European credit card fraud dataset (n = 284,807; fraud rate = 0.172%). It embraced the CRISP-DM process model as the data lifecycle model to guide it. The training partition was only subjected to SMOTE after 80/20 stratified split to avoid data leaking and the hyperparameters are optimized using stratified 3-fold cross-validation. Each tuned model was further probability threshold tuned with probability threshold set to 0.70 to maximize Precision-Recall operating point. All the experiments were executed in Google Colaboratory on Python 3.10. Precision, Recall, F1-Score, ROC-AUC and the Area Under the Precision-Recall Curve (AUPRC) were used to evaluate model performance, and AUPRC was chosen as the ultimate measure due to extreme imbalance in the classes. XGBoost developed as the most effective model in general, having the highest AUPRC (0.817), ROC-AUC (0.970) and the perfect combination of Precision = Recall = F1 = 0.81, which was achieved by tuning the probability threshold to 0.70. Random Forest had the best Precision (0.93) with AUPRC of 0.805 and hence it is the most appropriate model in the minimum false positive. Logistic Regression achieved maximum Recall (0.86) but had low Precision (0.10) which restricted its feasibility of operation even with threshold modification. These results indicate that XGBoost, together with SMOTE, systematic hyperparameter optimization, and threshold calibration, offers the best and balanced fraud detection at extreme imbalance in the classes.

credit card fraud detection; class imbalance; SMOTE; machine learning; XGBoost; Random Forest; AUPRC; hyperparameter optimization

1. Bergstra, J., & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13(10), 281–305. https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf [Google Scholar] [Crossref]

2. Bisong, E. (2019). Google Colaboratory. In Building machine learning and deep learning models on Google Cloud Platform: A comprehensive guide for beginners (pp. 59–64). Apress. https://doi.org/10.1007/978-1-4842-4470-8_7 [Google Scholar] [Crossref]

3. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324 [Google Scholar] [Crossref]

4. Carneiro, T., Da NóBrega, R. V. M., Nepomuceno, T., Bian, G.-B., De Albuquerque, V. H. C., & Reboucas Filho, P. P. (2018). Performance analysis of Google Colaboratory as a tool for accelerating deep learning applications. IEEE Access, 6, 61677–61685. https://doi.org/10.1109/ACCESS.2018.2874767 [Google Scholar] [Crossref]

5. Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953 [Google Scholar] [Crossref]

6. Cheah, P. C. Y., Yang, Y., & Lee, B. G. (2023). Enhancing financial fraud detection through addressing class imbalance using hybrid SMOTE-GAN techniques. International Journal of Financial Studies, 11(3), Article 110. https://doi.org/10.3390/ijfs11030110 [Google Scholar] [Crossref]

7. Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. https://doi.org/10.1145/2939672.2939785 [Google Scholar] [Crossref]

8. Chung, J., & Lee, K. (2023). Credit card fraud detection: An improved strategy for high recall using KNN, LDA, and linear regression. Sensors, 23(18), Article 7788. https://doi.org/10.3390/s23187788 [Google Scholar] [Crossref]

9. Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE Publications. [Google Scholar] [Crossref]

10. Dantas, R. M., Firdaus, R., Jaleel, F., Mata, P. N., Mata, M. N., & Li, G. (2022). Systemic acquired critique of credit card deception exposure through machine learning. Journal of Open Innovation: Technology, Market, and Complexity, 8(4), Article 192. https://doi.org/10.3390/joitmc8040192 [Google Scholar] [Crossref]

11. Davis, J., & Goadrich, M. (2006). The relationship between Precision-Recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning, 233–240. https://doi.org/10.1145/1143844.1143874 [Google Scholar] [Crossref]

12. Du, H., Lv, L., Guo, A., & Wang, H. (2023). AutoEncoder and LightGBM for credit card fraud detection problems. Symmetry, 15(4), Article 870. https://doi.org/10.3390/sym15040870 [Google Scholar] [Crossref]

13. Du, H., Zhang, Y., Li, X., & Wang, Q. (2024). A novel method for detecting credit card fraud problems. PLOS ONE, 19(3), Article e0294537. https://doi.org/10.1371/journal.pone.0294537 [Google Scholar] [Crossref]

14. Fernández, A., García, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research, 61(1), 863–905. https://doi.org/10.1613/jair.1.11192 [Google Scholar] [Crossref]

15. Herland, M., Bauder, R. A., & Khoshgoftaar, T. M. (2018). Approaches for identifying U.S. medicare fraud in provider claims data. Health Care Management Science, 23(1), 2–19. https://doi.org/10.1007/s10729-018-9460-8 [Google Scholar] [Crossref]

16. Ileberi, E., Sun, Y., & Wang, Z. (2021). Performance evaluation of machine learning methods for credit card fraud detection using SMOTE and AdaBoost. IEEE Access, 9, 165286–165294. https://doi.org/10.1109/ACCESS.2021.3134330 [Google Scholar] [Crossref]

17. Ileberi, E., Sun, Y., & Wang, Z. (2022). A machine learning based credit card fraud detection using the GA algorithm for feature selection. Journal of Big Data, 9, Article 24. https://doi.org/10.1186/s40537-022-00573-8 [Google Scholar] [Crossref]

18. Nilson Report. (2022). Card fraud losses worldwide [Issue 1209]. Nilson Report. https://nilsonreport.com [Google Scholar] [Crossref]

19. Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), Article e0118432. https://doi.org/10.1371/journal.pone.0118432 [Google Scholar] [Crossref]

20. Schroer, J., Kruse, F., & Gómez, J. M. (2021). A systematic literature review on applying CRISP-DM process model. Procedia Computer Science, 181, 526–534. https://doi.org/10.1016/j.procs.2021.01.199 [Google Scholar] [Crossref]

21. Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. arXiv. https://doi.org/10.48550/arxiv.1206.2944 [Google Scholar] [Crossref]

22. Strelcenia, E., & Prakoonwit, S. (2023). Improving classification performance in credit card fraud detection by using new data augmentation. AI, 4(1), 172–198. https://doi.org/10.3390/ai4010008 [Google Scholar] [Crossref]

23. Tripathy, N., Nayak, S. K., Godslove, J. F., Friday, I. K., & Dalai, S. S. (2022). Credit card fraud detection using logistic regression and synthetic minority oversampling technique (SMOTE) approach. International Journal of Computer and Communication Technology, 8(4), 38–45. https://doi.org/10.47893/ijcct.2022.1438 [Google Scholar] [Crossref]

24. ULB Machine Learning Group. (2013). Credit card fraud detection [Dataset]. Kaggle. https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud [Google Scholar] [Crossref]

25. Wirth, R., & Hipp, J. (2000). CRISP-DM: Towards a standard process model for data mining. Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining, 29–39. http://cs.unibo.it/~danilo.montesi/CBD/Beatriz/10.1.1.198.5133.pdf [Google Scholar] [Crossref]

26. Zhu, M., Zhang, Y., Gong, Y., Xu, C., & Xiang, Y. (2024). Enhancing credit card fraud detection: A neural network and SMOTE integrated approach. Journal of Theory and Practice of Engineering Science, 4(02), 23–30. https://doi.org/10.53469/jtpes.2024.04(02).04 [Google Scholar] [Crossref]

• Performance Analysis of Silicon PV Array Using Infrared Thermography and Detecting Temperature Non-Uniformities
• Dynamic Capabilities and Digital Transformation Performance: The Role of Organizational Agility in the Public Sector
• Comparative Effectiveness of Neural Mobilization Versus Positional Release Technique in Cervical Radiculopathy – An Experimental Study
• Soil Moisture Mapping of Solano Nueva Vizcaya: A Comparison and Review
• Silent Assistance for Emergencies (SAFE): A Mobile-Based Emergency Reporting Application for the Deaf and Mute Community