Vectorwall: A Hybrid Multi-Layer API Security Gateway with Ensemble Machine Learning and Adaptive Threshold Optimization | IJLTEMAS
00
Days
00
Hrs
00
Min
00
Sec
Submit Your Paper

Vectorwall: A Hybrid Multi-Layer API Security Gateway with Ensemble Machine Learning and Adaptive Threshold Optimization

Authors

Muhammad Hammad ur Rehman

HSE University Moscow, Russia (Russia)

Syed Haris Hussain

HSE University Moscow, Russia (Russia)

Article Information

DOI: 10.51583/IJLTEMAS.2026.151000080

Subject Category: Computer Science

Volume/Issue: 15/10 | Page No: 1001-1013

Publication Timeline

Submitted: 2026-10-25

Accepted: 2026-10-30

Published: 2026-11-10

Abstract

Signature-based web application firewalls (WAFs) remain the usual defence for web APIs, but they cannot recognise attacks for which no rule exists, they need constant rule maintenance, and they do not inspect newer threats such as prompt injection against applications built on large language models. This paper presents VectorWall, a self-hosted API security gateway that combines deterministic filtering with a machine-learning (ML) anomaly-detection layer. Each request passes a 16-level sequential pipeline: rate limiting, allow and block lists, custom rules, account-takeover detection, geo-blocking, IP reputation, behavioral scoring, prompt-injection and personally identifiable information (PII) inspection, a signature gate covering 18 attack classes, the OWASP Core Rule Set, OpenAPI schema validation and, last, an ensemble of three unsupervised detectors operating on an eight-dimensional request feature vector. A detector may vote only when its threshold lies inside its measured benign score range, and the ML layer ships in an observing mode that records but does not block. On CSIC 2010, under stratified 5-fold cross-validation with exact duplicates removed before splitting, a supervised XGBoost reference classifier reached a ROC-AUC of 0.769 ± 0.003; with the 58% duplicate rows kept, the same protocol gives 0.913, which shows how strongly duplicates inflate results on this dataset. We also show that unsupervised detectors trained on CSIC 2010 score ordinary REST traffic as maximally anomalous even after threshold calibration. In two load-test runs on one development machine, before and after removing a per-request connection overhead, the gateway blocked every attack probe and saturated at about 45–48 requests per second both times, so that overhead was not the bottleneck. The evaluation is limited to one dated dataset and one test host.

Keywords

Web application firewall, machine learning, anomaly detection, ensemble learning, threshold calibration, Isolation Forest, API security gateway, prompt injection, zero-day attack, adaptive security

Downloads

References

1. OWASP Foundation, “OWASP API Security Top 10,” 2019. [Online]. [Google Scholar] [Crossref]

2. Available: https://owasp.org/www-project-api-security/ [Google Scholar] [Crossref]

3. OWASP Foundation, “OWASP ModSecurity Core Rule Set,” 2023. [Online]. Available: https://coreruleset.org/ [Google Scholar] [Crossref]

4. F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” in Proc. 8th IEEE Int. Conf. Data Mining (ICDM), 2008, pp. 413–422. [Google Scholar] [Crossref]

5. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 785–794. [Google Scholar] [Crossref]

6. I. Ristić, ModSecurity Handbook. London, U.K.: Feisty Duck, 2010. [Google Scholar] [Crossref]

7. C. Kruegel, G. Vigna, and W. Robertson, “A multi-model approach to the detection of web-based attacks,” Computer Networks, vol. 48, no. 5, pp. 717–738, 2005. [Google Scholar] [Crossref]

8. Spanish National Research Council (CSIC), “CSIC 2010 HTTP dataset,” 2010. [Online]. Available: http://www.isi.csic.es/dataset/ [Google Scholar] [Crossref]

9. I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in Proc. 4th Int. Conf. Information Systems Security and Privacy (ICISSP), 2018, pp. 108–116. [Google Scholar] [Crossref]

10. F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” in NeurIPS ML Safety Workshop, 2022. [Google Scholar] [Crossref]

11. G. Baye, F. Hussain, A. Oracevic, R. Hussain, and S. M. A. Kazmi, “API security in large enterprises: Leveraging machine learning for anomaly detection,” in Proc. Int. Symp. Networks, Computers and Communications (ISNCC), 2021, pp. 1–6, doi: 10.1109/ISNCC52172.2021.9615638. [Google Scholar] [Crossref]

12. J. Liang, W. Zhao, and W. Ye, “Anomaly-based web attack detection: A deep learning approach,” in Proc. 2017 VI Int. Conf. Network, Communication and Computing (ICNCC), 2017, pp. 80–85. [Google Scholar] [Crossref]

13. M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “LOF: Identifying density-based local outliers,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2000, pp. 93–104. [Google Scholar] [Crossref]

14. B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, “Estimating the support of a high-dimensional distribution,” Neural Computation, vol. 13, no. 7, pp. 1443–1471, 2001. [Google Scholar] [Crossref]

15. ONNX Runtime Developers, “ONNX Runtime: Cross-platform machine learning model accelerator,” 2023. [Online]. Available: https://onnxruntime.ai/ [Google Scholar] [Crossref]

Metrics

Views & Downloads

Similar Articles

Engineering,
Management &
Applied Science
Journal
(IJLTEMAS)
Published by RSIS International
Tower B, Suncity Success Tower
Golf Course Ext Rd, Sector 65
Gurugram, Haryana 122101, India
© 2026 IJLTEMAS · RSIS International. ISSN 2278-2540.