Model-Dependent Calibration Effects of Imbalance Handling in Credit Card Fraud Detection

Class imbalance correction methods such as class weighting and synthetic oversampling are widely used to improve fraud detection, but they can distort the probability estimates that risk scoring depends on. This study evaluates the calibration cost of two common imbalance handling strategies, inverse frequency class weighting and SMOTE-NC, applied to three model families: Logistic Regression, Random Forest, and XGBoost. Using the Sparkov credit card transaction dataset, comprising 1,296,675 training transactions and 555,719 future test transactions, the analysis follows a chronological four way split that separates model training, calibration fitting, threshold selection, and future testing, so that no evaluation period leaks information into model fitting or calibration. Across the three models, mean raw expected calibration error (ECE) rose from 0.0012 without imbalance handling to 0.0715 with class weighting and 0.0742 with SMOTE-NC, driven mainly by Logistic Regression. Mean raw F1 score declined from 0.680 to 0.648 and 0.622, respectively, showing that neither strategy consistently improved detection performance. The effects were model dependent: class weighting improved the F1 score of XGBoost from 0.7878 to 0.8106 while substantially increasing its calibration error, whereas Random Forest with class weighting retained near ideal calibration. For the five configurations whose raw ECE exceeded 0.01, post hoc isotonic calibration, fitted on a dedicated chronological period, reduced ECE by 96.6% to 99.4% while keeping future test F1 score differences below 0.008 across all nine raw versus calibrated pairs. The calibration gains were robust across multiple ECE binning schemes, including 10, 20, and 50 equal width bins and adaptive equal frequency binning. These results indicate that imbalance handling should be evaluated separately for each model family rather than assumed to help uniformly, and that post hoc calibration should be applied conditionally, only when diagnostics indicate substantial probability distortion, rather than as a default step in every fraud detection pipeline.

credit card fraud detection, probability calibration, class imbalance, SMOTE-NC, class weighting

1. K. G. Dastidar, O. Caelen, and M. Granitzer, “Machine learning methods for credit card fraud detection: A survey,” IEEE Access, vol. 12, pp. 158939–158965, 2024. [Google Scholar] [Crossref]

2. A. Cherif, A. Badhib, H. Ammar, S. Alshehri, M. Kalkatawi, and A. Imine, “Credit card fraud detection in the era of disruptive technologies: A systematic review,” J. King Saud Univ. Inf. Sci., vol. 35, no. 1, pp. 145–174, 2023. [Google Scholar] [Crossref]

3. S. Makki, Z. Assaghir, Y. Taher, R. Haque, M.-S. Hacid, and H. Zeineddine, “An experimental study with imbalanced classification approaches for credit card fraud detection,” IEEE Access, vol. 7, pp. 93010–93022, 2019. [Google Scholar] [Crossref]

4. E. Ileberi, Y. Sun, and Z. Wang, “A machine learning based credit card fraud detection using the GA algorithm for feature selection,” J. Big Data, vol. 9, no. 1, p. 24, 2022. [Google Scholar] [Crossref]

5. H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263–1284, 2009. [Google Scholar] [Crossref]

6. N. V Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, 2002. [Google Scholar] [Crossref]

7. P. Hajek, M. Z. Abedin, and U. Sivarajah, “Fraud detection in mobile payment systems using an XGBoost-based framework,” Inf. Syst. Front., vol. 25, no. 5, pp. 1985–2003, 2023. [Google Scholar] [Crossref]

8. I. D. Mienye and Y. Sun, “A deep learning ensemble with data resampling for credit card fraud detection,” IEEE Access, vol. 11, pp. 30628–30638, 2023. [Google Scholar] [Crossref]

9. E. Strelcenia and S. Prakoonwit, “Improving classification performance in credit card fraud detection by using new data augmentation,” AI, vol. 4, no. 1, pp. 172–198, 2023. [Google Scholar] [Crossref]

10. M. Piccininni, M. Wechsung, B. Van Calster, J. L. Rohmann, S. Konigorski, and M. van Smeden, “Understanding random resampling techniques for class imbalance correction and their consequences on calibration and discrimination of clinical risk prediction models,” J. Biomed. Inform., vol. 155, p. 104666, 2024. [Google Scholar] [Crossref]

11. A. Mishra, R. L. McClelland, L. Y. T. Inoue, and K. F. Kerr, “Recalibration methods for improved clinical utility of risk scores,” Med. Decis. Mak., vol. 42, no. 4, pp. 500–512, 2022. [Google Scholar] [Crossref]

12. M. Kull, T. M. Silva Filho, and P. Flach, “Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration,” Electron. J. Stat., vol. 11, no. 2, pp. 5052–5080, 2017. [Google Scholar] [Crossref]

13. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International conference on machine learning, PMLR, 2017, pp. 1321–1330. [Google Scholar] [Crossref]

14. G. Menardi and N. Torelli, “Training and assessing classification rules with imbalanced data,” Data Min. Knowl. Discov., vol. 28, no. 1, pp. 92–122, 2014. [Google Scholar] [Crossref]

15. A. Dal Pozzolo, O. Caelen, R. A. Johnson, and G. Bontempi, “Calibrating probability with undersampling for unbalanced classification,” in Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI), 2015, pp. 159–166. [Google Scholar] [Crossref]

16. J. L. Leevy, J. M. Johnson, J. Hancock, and T. M. Khoshgoftaar, “Threshold optimization and random undersampling for imbalanced credit card data,” J. Big Data, vol. 10, no. 1, p. 58, 2023, doi: 10.1186/s40537-023-00738-z. [Google Scholar] [Crossref]

17. T. Silva Filho, H. Song, M. Perello-Nieto, R. Santos-Rodriguez, M. Kull, and P. Flach, “Classifier calibration: a survey on how to assess and improve predicted class probabilities,” Mach. Learn., vol. 112, no. 9, pp. 3211–3260, 2023, doi: 10.1007/s10994-023-06336-7. [Google Scholar] [Crossref]

18. G. Douzas, F. Bacao, and F. Last, “Improving imbalanced learning through a heuristic oversampling method based on k-means and SMOTE,” Inf. Sci. (Ny)., vol. 465, pp. 1–20, 2018. [Google Scholar] [Crossref]

19. C. Elkan, “The foundations of cost-sensitive learning,” in Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2001, pp. 973–978. [Google Scholar] [Crossref]

20. A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” in Proceedings of the 22nd international conference on Machine learning, 2005, pp. 625–632. doi: 10.1145/1102351.1102430. [Google Scholar] [Crossref]

21. M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in Proceedings of the AAAI conference on artificial intelligence, 2015. [Google Scholar] [Crossref]

22. G. W. Brier, “Verification of forecasts expressed in terms of probability,” Mon. Weather Rev., vol. 78, no. 1, pp. 1–3, 1950. [Google Scholar] [Crossref]

23. J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019, pp. 38–41. [Google Scholar] [Crossref]

24. J. C. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, MIT Press, 2000, pp. 61–74. [Google Scholar] [Crossref]

25. B. Zadrozny and C. Elkan, “Transforming classifier scores into accurate multiclass probability estimates,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002, pp. 694–699. [Google Scholar] [Crossref]

26. S. Rajaraman, P. Ganesan, and S. Antani, “Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks,” PLoS One, vol. 17, no. 1, p. e0262838, 2022. [Google Scholar] [Crossref]

27. T. Guilbert, O. Caelen, A. Chirita, and M. Saerens, “Calibration methods in imbalanced binary classification,” Ann. Math. Artif. Intell., vol. 92, pp. 1319–1352, 2024. [Google Scholar] [Crossref]

28. Z. Liu, “The hidden cost of resampling: How imbalance correction degrades probability calibration in tree ensembles,” 2026. doi: 10.48550/arXiv.2606.29720. [Google Scholar] [Crossref]

29. F. Lin, Y. Dai, and Y. Liu, “On probability estimation for unbalanced classification,” J. Stat. Theory Pract., vol. 20, p. 57, 2026, doi: 10.1007/s42519-026-00572-5. [Google Scholar] [Crossref]

30. kartik2112, “Credit Card Transactions Fraud Detection Dataset.” Kaggle, Aug. 13, 2020. [Online]. Available: https://www.kaggle.com/datasets/kartik2112/fraud-detection [Google Scholar] [Crossref]