Adaptive Conservation-Consistent Multi-Agent Policy Optimization for Tail-Risk-Aware Inventory Networks
DOI:
https://doi.org/10.54097/vbke0c45Keywords:
Multi-agent policy optimization, Supply chain inventory, Conservation constraints, CVaR, Adaptive risk control, Differentiable simulationAbstract
Capacity-limited inventory networks must reconcile locally requested shipments with executable warehouse flows while controlling losses under supply and demand disruptions. We introduce a decentralized, parameter-shared policy trained through a differentiable, mass-balanced simulator. A nodewise command-to-realization residual, derived from discrete inventory conservation, penalizes infeasible requested flows. An online-adjusted risk weight couples mean episode cost with the upper-tail conditional value-at-risk (CVaR). Ten independent seeds compare seven policies across four synthetic disturbance regimes (400 held-out trajectories per method, seed and regime). Under combined stress, mean cost is 4906.8 versus 4979.7 for risk-neutral policy optimization, while CVaR90 is 8464.8 versus 8580.9. Against a fixed-risk comparator with a closely matched final risk weight, the adaptive formulation further reduces stress CVaR90 by 39.4 cost units. Requested-but-unshipped flow drops from 15.80 to 2.27 units/day. Realized transitions conserve inventory under every policy: the residual regulates planning consistency, not physical conservation itself. Results reflect a reproducible synthetic benchmark rather than field validation.
Downloads
References
[1] Clark, A. J., & Scarf, H. (1960). Optimal policies for a multi-echelon inventory problem. Management Science, 6(4), 475-490. https://doi.org/10.1287/mnsc.6.4.475
[2] Chaharsooghi, S. K., Heydari, J., & Zegordi, S. H. (2008). A reinforcement learning model for supply chain ordering management: An application to the beer game. Decision Support Systems, 45(4), 949-959. https://doi.org/10.1016/j.dss.2008.03.007
[3] Oroojlooyjadid, A., Snyder, L. V., & Takáč, M. (2020). Applying deep learning to the newsvendor problem. IISE Transactions, 52(4), 444-463. https://doi.org/10.1080/24725854.2019.1632502
[4] Gijsbrechts, J., Boute, R. N., Van Mieghem, J. A., & Zhang, D. J. (2022). Can deep reinforcement learning improve inventory management? Performance on lost sales, dual-sourcing, and multi-echelon problems. Manufacturing & Service Operations Management, 24(3), 1349-1368. https://doi.org/10.1287/msom.2021.1064
[5] Meisheri, H., Sultana, N. N., Baranwal, M., Baniwal, V., Nath, S., Verma, S., Ravindran, B., & Khadilkar, H. (2022). Scalable multi-product inventory control with lead time constraints using reinforcement learning. Neural Computing and Applications, 34(3), 1735-1757. https://doi.org/10.1007/s00521-021-06129-w
[6] Hubbs, C. D., Perez, H. D., Sarwar, O., Sahinidis, N. V., Grossmann, I. E., & Wassick, J. M. (2020). OR-Gym: A reinforcement learning library for operations research problems. arXiv, arXiv:2008.06319. https://arxiv.org/abs/2008.06319
[7] Yang, X., Liu, Z., Jiang, W., Zhang, C., Zhao, L., Song, L., & Bian, J. (2023). A versatile multi-agent reinforcement learning benchmark for inventory management. arXiv, arXiv:2306.07542. https://arxiv.org/abs/2306.07542
[8] Sultana, N. N., Meisheri, H., Baniwal, V., Nath, S., Ravindran, B., & Khadilkar, H. (2020). Reinforcement learning for multi-product multi-node inventory management in supply chains. arXiv, arXiv:2006.04037. https://arxiv.org/abs/2006.04037
[9] Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems, 30, 6379-6390.
[10] Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018). Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2974-2982. https://doi.org/10.1609/aaai.v32i1.11794
[11] Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., & Whiteson, S. (2018). QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), PMLR 80 (pp. 4295-4304). PMLR.
[12] Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., & Wu, Y. (2022). The surprising effectiveness of PPO in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35, 24611-24624. https://doi.org/10.52202/068431-1787
[13] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686-707. https://doi.org/10.1016/j.jcp.2018.10.045
[14] Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics, 3, 422-440. https://doi.org/10.1038/s42254-021-00314-5
[15] Cuomo, S., Di Cola, V. S., Giampaolo, F., Rozza, G., Raissi, M., & Piccialli, F. (2022). Scientific machine learning through physics-informed neural networks: Where we are and what's next. Journal of Scientific Computing, 92, Article 88. https://doi.org/10.1007/s10915-022-01939-z
[16] Wang, S., Yu, X., & Perdikaris, P. (2022). When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449, Article 110768. https://doi.org/10.1016/j.jcp.2021.110768
[17] Bellemare, M. G., Dabney, W., & Munos, R. (2017). A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70 (pp. 449-458). PMLR.
[18] Dabney, W., Rowland, M., Bellemare, M. G., & Munos, R. (2018). Distributional reinforcement learning with quantile regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2892-2901. https://doi.org/10.1609/aaai.v32i1.11791
[19] Rockafellar, R. T., & Uryasev, S. (2000). Optimization of conditional value-at-risk. The Journal of Risk, 2(3), 21-41. https://doi.org/10.21314/JOR.2000.038
[20] Chow, Y., Ghavamzadeh, M., Janson, L., & Pavone, M. (2018). Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167), 1-51.
[21] Tamar, A., Chow, Y., Ghavamzadeh, M., & Mannor, S. (2015). Policy gradient for coherent risk measures. Advances in Neural Information Processing Systems, 28, 1468-1476.
[22] Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1), 1-122. https://doi.org/10.1561/2200000016
[23] Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning (ICML), PMLR 70 (pp. 22-31). PMLR.
[24] Liu, J., Wang, J., & Lin, H. (2025). Coordinated physics-informed multi-agent reinforcement learning for risk-aware supply chain optimization. IEEE Access, 13, 190980-190993. https://doi.org/10.1109/ACCESS.2025.3629716
[25] Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
[26] Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1057-1063.
[27] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 3207-3214. https://doi.org/10.1609/aaai.v32i1.11694
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Computing and Electronic Information Management
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
