Vectorwall: A Hybrid Multi-Layer API Security Gateway with Ensemble Machine Learning and Adaptive Threshold Optimization
Authors
HSE University Moscow, Russia (Russia)
HSE University Moscow, Russia (Russia)
Article Information
DOI: 10.51583/IJLTEMAS.2026.151000080
Subject Category: Computer Science
Volume/Issue: 15/10 | Page No: 1001-1013
Publication Timeline
Submitted: 2026-10-25
Accepted: 2026-10-30
Published: 2026-11-10
Abstract
Signature-based web application firewalls (WAFs) remain the usual defence for web APIs, but they cannot recognise attacks for which no rule exists, they need constant rule maintenance, and they do not inspect newer threats such as prompt injection against applications built on large language models. This paper presents VectorWall, a self-hosted API security gateway that combines deterministic filtering with a machine-learning (ML) anomaly-detection layer. Each request passes a 16-level sequential pipeline: rate limiting, allow and block lists, custom rules, account-takeover detection, geo-blocking, IP reputation, behavioral scoring, prompt-injection and personally identifiable information (PII) inspection, a signature gate covering 18 attack classes, the OWASP Core Rule Set, OpenAPI schema validation and, last, an ensemble of three unsupervised detectors operating on an eight-dimensional request feature vector. A detector may vote only when its threshold lies inside its measured benign score range, and the ML layer ships in an observing mode that records but does not block. On CSIC 2010, under stratified 5-fold cross-validation with exact duplicates removed before splitting, a supervised XGBoost reference classifier reached a ROC-AUC of 0.769 ± 0.003; with the 58% duplicate rows kept, the same protocol gives 0.913, which shows how strongly duplicates inflate results on this dataset. We also show that unsupervised detectors trained on CSIC 2010 score ordinary REST traffic as maximally anomalous even after threshold calibration. In two load-test runs on one development machine, before and after removing a per-request connection overhead, the gateway blocked every attack probe and saturated at about 45–48 requests per second both times, so that overhead was not the bottleneck. The evaluation is limited to one dated dataset and one test host.
Keywords
Web application firewall, machine learning, anomaly detection, ensemble learning, threshold calibration, Isolation Forest, API security gateway, prompt injection, zero-day attack, adaptive security
This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
Downloads
References
1. OWASP Foundation, “OWASP API Security Top 10,” 2019. [Online]. [Google Scholar] [Crossref]
2. Available: https://owasp.org/www-project-api-security/ [Google Scholar] [Crossref]
3. OWASP Foundation, “OWASP ModSecurity Core Rule Set,” 2023. [Online]. Available: https://coreruleset.org/ [Google Scholar] [Crossref]
4. F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in Proc. 8th IEEE Int. Conf. Data Mining (ICDM), 2008, pp. 413–422. [Google Scholar] [Crossref]
5. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 785–794. [Google Scholar] [Crossref]
6. I. Ristić, ModSecurity Handbook. London, U.K.: Feisty Duck, 2010. [Google Scholar] [Crossref]
7. C. Kruegel, G. Vigna, and W. Robertson, “A multi-model approach to the detection of web-based attacks,” Computer Networks, vol. 48, no. 5, pp. 717–738, 2005. [Google Scholar] [Crossref]
8. Spanish National Research Council (CSIC), “CSIC 2010 HTTP dataset,” 2010. [Online]. Available: http://www.isi.csic.es/dataset/ [Google Scholar] [Crossref]
9. I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in Proc. 4th Int. Conf. Information Systems Security and Privacy (ICISSP), 2018, pp. 108–116. [Google Scholar] [Crossref]
10. F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” in NeurIPS ML Safety Workshop, 2022. [Google Scholar] [Crossref]
11. G. Baye, F. Hussain, A. Oracevic, R. Hussain, and S. M. A. Kazmi, “API security in large enterprises: Leveraging machine learning for anomaly detection,” in Proc. Int. Symp. Networks, Computers and Communications (ISNCC), 2021, pp. 1–6, doi: 10.1109/ISNCC52172.2021.9615638. [Google Scholar] [Crossref]
12. J. Liang, W. Zhao, and W. Ye, “Anomaly-based web attack detection: A deep learning approach,” in Proc. 2017 VI Int. Conf. Network, Communication and Computing (ICNCC), 2017, pp. 80–85. [Google Scholar] [Crossref]
13. M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “LOF: Identifying density-based local outliers,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2000, pp. 93–104. [Google Scholar] [Crossref]
14. B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, “Estimating the support of a high-dimensional distribution,” Neural Computation, vol. 13, no. 7, pp. 1443–1471, 2001. [Google Scholar] [Crossref]
15. ONNX Runtime Developers, “ONNX Runtime: Cross-platform machine learning model accelerator,” 2023. [Online]. Available: https://onnxruntime.ai/ [Google Scholar] [Crossref]
Metrics
Views & Downloads
Similar Articles
- Explainable 3D Convolutional Neural Networks Spatiotemporal Learning for Human Handshake Interaction Recognition
- Comprehensive Empirical Benchmarking of Twelve Sorting Algorithms Across Comparison-Based, Non-Comparison-Based, and Hybrid Paradigms: A Multi-Dimensional Performance Model for Algorithm Selection at Practical Data Scales (n up to 100,000)
- Reinforcement Learning for Personalized Insulin Dosing: A Comparative Study of A2C, SAC and PPO on Real-World Clinical Data
- Faculty Milestone Manager and Appraisal System: A Web-Based Solution Part 1
- Improving Diabetes Prediction Through IQR Outlier Treatment an Evaluation of XGBoost And LightGBM Models
This work is licensed under CC BY 4.0