Bi-level Agentic Reinforcement Learning for Web Decision-Making under Multi-Cost and Failure Risk Constraints
PDF

Keywords

Bi-level optimization
agentic reinforcement learning
web decision models
multi-cost constraints
failure risk control

Abstract

Web agent decision-making naturally involves nested objectives, where high-level task completion depends on lower-level interaction strategies that incur heterogeneous costs and failure risks. This study formulates agentic web reinforcement learning as a bi-level optimization problem. The upper-level objective optimizes task success and global failure tolerance, while the lower-level problem minimizes cumulative operational costs subject to interaction feasibility. The two levels are coupled through shared policy parameters and budget signals. A differentiable bi-level training scheme is introduced to approximate the lower-level optimal response during policy updates. The proposed model is evaluated on 1,000–1,800 web tasks spanning transactional, informational, and configuration studyflows. Empirical results demonstrate improved cost efficiency and reduced catastrophic failure frequency compared with single-level constrained reinforcement learning, particularly in tasks requiring long sequences of dependent actions.

PDF

References

Mao, Y., Ma, X., & Li, J. (2025). Research on Web System Anomaly Detection and Intelligent Operations Based on Log Modeling and Self-Supervised Learning.

Ma, Q., Yue, L., Xu, S., Shi, Y., & Liu, H. (2026). Web Agent Agentic Reinforcement Learning Decision Model Under Multi-Cost and Failure Risk Constraints.

Wachi, A., Shen, X., & Sui, Y. (2024). A survey of constraint formulations in safe reinforcement learning. arXiv preprint arXiv:2402.02025.

Ceusters, G., Camargo, L. R., Franke, R., Nowé, A., & Messagie, M. (2023). Safe reinforcement learning for multi-energy management systems with known constraint functions. Energy and AI, 12, 100227.

Qiu, Y., & Wang, J. (2023, October). A machine learning approach to credit card customer segmentation for economic stability. In Proceedings of the 4th International Conference on Economic Management and Big Data Applications, ICEMBDA (pp. 27-29).

Trapasso, A. (2025). Integrating multi-agent planning and reinforcement learning through reward and exploration machines.

Zhu, W., Yao, Y., & Yang, J. (2025). Real-Time Risk Control Effects of Digital Compliance Dashboards: An Empirical Study Across Multiple Enterprises Using Process Mining, Anomaly Detection, and Interrupt Time Series.

Lim, S. H., & Malik, I. (2022). Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35, 30977-30989.

Li, T., Xia, J., Liu, S., & Jiang, Y. (2025). Digital Transformation of Human Resources: From Consulting Frameworks to AI-Enabled Learning Management Systems.

Danish, M. S. S. (2023). A framework for modeling and optimization of data-driven energy systems using machine learning. IEEE Transactions on Artificial Intelligence, 5(5), 2434-2443.

Gu, X., Liu, M., & Yang, J. (2025). Application and Effectiveness Evaluation of Federated Learning Methods in Anti-Money Laundering Collaborative Modeling Across Inter-Institutional Transaction Networks.

Fernández, J. G., Ahmad, N., & van Gerven, M. (2025). A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation. arXiv preprint arXiv:2510.18812.

Cai, B., Bai, W., Lu, Y., & Lu, K. (2024, June). Fuzz like a Pro: Using Auditor Knowledge to Detect Financial Vulnerabilities in Smart Contracts. In 2024 International Conference on Meta Computing (ICMC) (pp. 230-240). IEEE.

Almasi, M., & Anahideh, H. (2025). Bi-Level Contextual Bandits for Individualized Resource Allocation under Delayed Feedback. arXiv preprint arXiv:2511.10572.

Liu, S., Feng, H., & Liu, X. (2025). A Study on the Mechanism of Generative Design Tools' Impact on Visual Language Reconstruction: An Interactive Analysis of Semantic Mapping and User Cognition. Authorea Preprints.

Wessel, M., Vargovich, J., Gerosa, M. A., & Treude, C. (2023). Github actions: the impact on the pull request process. Empirical Software Engineering, 28(6), 131.

Du, Y. (2025). Research on Deep Learning Models for Forecasting Cross-Border Trade Demand Driven by Multi-Source Time-Series Data. Journal of Science, Innovation & Social Impact, 1(2), 63-70.

Thiyagalingam, J., Shankar, M., Fox, G., & Hey, T. (2022). Scientific machine learning benchmarks. Nature Reviews Physics, 4(6), 413-420.

Mao, Y., Ma, X., & Li, J. (2025). Research on API Security Gateway and Data Access Control Model for Multi-Tenant Full-Stack Systems.

Mystakidis, A., Koukaras, C., Koukaras, P., Kaparis, K., Stavrinides, S. G., & Tjortjis, C. (2024, December). Optimizing nurse rostering: a case study using integer programming to enhance operational efficiency and care quality. In Healthcare (Vol. 12, No. 24, p. 2545). MDPI.

Chowdhury, R., & Ara, J. (2022). Optimization Algorithms for Enhancing High Dimensional Biomedical Data Processing Efficiency. Review of Applied Science and Technology, 1(04), 98-145.

Zhu, W., Yang, J., & Yao, Y. (2025, October). How Compliance Maturity Translates to Risk Reduction: A Multi-Case Comparison of Global Operations Using fsQCA and Hierarchical Bayesian Methods. In Proceedings of the 2025 2nd International Conference on Digital Economy and Computer Science (pp. 672-676).

Li, T., Xia, J., Liu, S., & Hong, E. (2025). Strategic Human Resource Leadership in Global Biopharmaceutical Enterprises: Integrating HR Analytics and Cross-Cultural.

Petrulionytė, I., Mairal, J., & Arbel, M. (2024). Functional bilevel optimization for machine learning. Advances in Neural Information Processing Systems, 37, 14016-14065.

Gu, X., Yang, J., & Liu, M. (2025). Research on a Green Money Laundering Identification Framework and Risk Monitoring Mechanism Integrating Artificial Intelligence and Environmental Governance Data.

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright (c) 2026 James Thompson, Emily Carter, Daniel Wright (Author)

Downloads

Download data is not yet available.