Safe-ARCS: Risk-Budget Shielding for Hierarchical Reinforcement Learning in Automated Cybersecurity Incident Response
DOI:
https://doi.org/10.54097/yv21p342Keywords:
Cyber incident response, Hierarchical reinforcement learning, Action shielding, Constrained decision making, Reproducible simulationAbstract
Automated incident response should contain malicious activity without imposing unacceptable disruption on legitimate services. We study this conflict in a reproducible six-step stochastic simulator with noisy alerts, five latent incident classes, immediate containment actions, and persistent strategic defenses. Safe-ARCS builds on two mechanisms of the ARCS framework: independently learned immediate and strategic decision streams with exponentially decayed alert urgency, and temporal-difference prioritized replay. Its new budget shield admits an action only when a verified upper envelope on induced operational cost fits the remaining episode budget. For the specified bounded-cost simulator, this filtering yields a simple zero-overspend invariant. Across five independent training seeds and 59,500 held-out evaluation episodes, Safe-ARCS achieves 86.4% attack resolution and zero budget exceedances in nominal tests, versus 88.2% resolution and 0.5% exceedances for its unshielded dual-stream counterpart. Under shifted alert distributions and weakened mitigation probabilities, the corresponding results are 77.3% versus 80.4% resolution, with zero versus 1.9% exceedances. Ablations expose component effects. A separate 16,200-episode cost-misspecification study shows that under 50% inflation of modeled intervention costs, the default shield exceeds the budget in 3.7% of shifted-test episodes, whereas a calibrated 1.5-fold reserve eliminates these violations at a 2.1-point resolution cost. The findings define an auditable, conditional safety–effectiveness trade-off rather than an estimate of operational-network performance.
Downloads
References
[1] Buczak, A. L., & Guven, E. (2016). A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys & Tutorials, 18(2), 1153-1176. https://doi.org/10.1109/COMST.2015.2494502
[2] Ferrag, M. A., Maglaras, L., Moschoyiannis, S., & Janicke, H. (2020). Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. Journal of Information Security and Applications, 50, Article 102419. https://doi.org/10.1016/j.jisa.2019.102419
[3] Nguyen, T. T., & Reddi, V. J. (2023). Deep reinforcement learning for cyber security. IEEE Transactions on Neural Networks and Learning Systems, 34(8), 3779-3795. https://doi.org/10.1109/TNNLS.2021.3121870
[4] Adawadkar, A. M. K., & Kulkarni, N. (2022). Cyber-security and reinforcement learning: A brief survey. Engineering Applications of Artificial Intelligence, 114, Article 105116. https://doi.org/10.1016/j.engappai.2022.105116
[5] Ren, S., Jin, J., Niu, G., & Liu, Y. (2025). ARCS: Adaptive reinforcement learning framework for automated cybersecurity incident response strategy optimization. Applied Sciences, 15(2), Article 951. https://doi.org/10.3390/app15020951
[6] Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2), 181-211. https://doi.org/10.1016/S0004-3702(99)00052-1
[7] Dietterich, T. G. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13, 227-303. https://doi.org/10.1613/JAIR.639
[8] Ghanem, M. C., Chen, T. M., & Nepomuceno, E. G. (2023). Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks. Journal of Intelligent Information Systems, 60, 281-303. https://doi.org/10.1007/s10844-022-00738-0
[9] Oh, S. H., Kim, J., Nah, J. H., & Park, J. (2024). Employing deep reinforcement learning to cyber-attack simulation for enhancing cybersecurity. Electronics, 13(3), Article 555. https://doi.org/10.3390/electronics13030555
[10] Hore, S., Shah, A., & Bastian, N. D. (2023). Deep VULMAN: A deep reinforcement learning-enabled cyber vulnerability management framework. Expert Systems with Applications, 221, Article 119734. https://doi.org/10.1016/j.eswa.2023.119734
[11] Vávra, J., Hromada, M., Lukáš, L., & Dworzecki, J. (2021). Adaptive anomaly detection system based on machine learning algorithms in an industrial control environment. International Journal of Critical Infrastructure Protection, 34, Article 100446. https://doi.org/10.1016/j.ijcip.2021.100446
[12] Shaukat, K., Luo, S., Varadharajan, V., Hameed, I. A., & Xu, M. (2020). A survey on machine learning techniques for cyber security in the last decade. IEEE Access, 8, 222310-222354. https://doi.org/10.1109/ACCESS.2020.3041951
[13] García, J., & Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42), 1437-1480.
[14] Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70 (pp. 22-31). PMLR.
[15] Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., & Topcu, U. (2018). Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2669-2678. https://doi.org/10.1609/aaai.v32i1.11797
[16] Chow, Y., Ghavamzadeh, M., Janson, L., & Pavone, M. (2018). Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167), 1-51.
[17] Tessler, C., Mankowitz, D. J., & Mannor, S. (2019). Reward constrained policy optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
[18] Altman, E. (1999). Constrained Markov decision processes. Chapman & Hall/CRC.
[19] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized experience replay. In Proceedings of the International Conference on Learning Representations (ICLR).
[20] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533. https://doi.org/10.1038/nature14236
[21] van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), 2094-2100. https://doi.org/10.1609/aaai.v30i1.10295
[22] Sewak, M., Sahay, S. K., & Rathore, H. (2023). Deep reinforcement learning in the advanced cybersecurity threat detection and protection. Information Systems Frontiers, 25, 589-611. https://doi.org/10.1007/s10796-022-10333-x
[23] Tamar, A., Glassner, Y., & Mannor, S. (2015). Optimizing the CVaR via sampling. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), 2993-2999. https://doi.org/10.1609/aaai.v29i1.9561
[24] Moustafa, N., & Slay, J. (2015). UNSW-NB15: A comprehensive data set for network intrusion detection systems. In 2015 Military Communications and Information Systems Conference (MilCIS) (pp. 1-6). IEEE. https://doi.org/10.1109/MilCIS.2015.7348942
[25] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv, arXiv:1707.06347. https://doi.org/10.48550/arXiv.1707.06347
[26] Sarker, I. H. (2021). Deep cybersecurity: A comprehensive overview from neural network and deep learning perspective. SN Computer Science, 2, Article 154. https://doi.org/10.1007/s42979-021-00535-6
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Computing and Electronic Information Management
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
