InfraLib: Enabling Reinforcement Learning and Decision Making for Large Scale Infrastructure Management
Pranay Thangeda Trevor S. Betz Michael N. Grussing Melkior Ornik



Bi-capacity Choquet Integral for Sensor Fusion 
 with Label Uncertainty



This material is based upon work supported by the National Science
Foundation under Grant IIS-2153171-CRII: III: Explainable Multi-Source Data Integration with Uncertainty.

    Hersh Vakharia
University of Michigan 

Ann Arbor, MI 

hershv@umich.edu
Xiaoxiao Du
University of Michigan 

Ann Arbor, MI 

xiaodu@umich.edu

    Accepted XXX. Received YYY; in original form ZZZ
========================================================================================================================================================================================================================================================



§ ABSTRACT 
    Efficient management of infrastructure systems is crucial for economic stability, sustainability, and public safety. However, infrastructure management is challenging due to the vast scale of systems, stochastic deterioration of components, partial observability, and resource constraints. While data-driven approaches like reinforcement learning (RL) offer a promising avenue for optimizing management policies, their application to infrastructure has been limited by the lack of suitable simulation environments. We introduce InfraLib, a comprehensive framework for modeling and analyzing infrastructure management problems. InfraLib employs a hierarchical, stochastic approach to realistically model infrastructure systems and their deterioration. It supports practical functionality such as modeling component unavailability, cyclical budgets, and catastrophic failures. To facilitate research, InfraLib provides tools for expert data collection, simulation-driven analysis, and visualization. We demonstrate InfraLib's capabilities through case studies on a real-world road network and a synthetic benchmark with 100,000 components. 



§ INTRODUCTION


Infrastructure systems are the backbone of modern society, encompassing a wide array of essential services including transportation networks, utility systems, and public facilities. Efficient infrastructure management is crucial for modern society's functioning, influencing economic stability <cit.>, <cit.>, environmental sustainability <cit.>, and public safety <cit.>. Managing modern infrastructure systems is a complex and multifaceted task, involving the maintenance, repair, and replacement of numerous components distributed across facilities and networks <cit.>, <cit.>. The challenges of infrastructure management are further compounded by their vast scale, the stochastic nature of component deterioration <cit.>, <cit.>, stringent operational constraints <cit.>, limited resources <cit.>, and extreme weather events due to climate change <cit.>, <cit.>, <cit.>.

some dicussion on how infra management is usually done and end with downsides of it

Traditional approaches to infrastructure management typically involve rule-based methodologies that rely on deterministic models. These methods, while useful in controlled environments, often struggle to capture the inherent uncertainties and dynamic variations present in real-world scenarios <cit.>. The complexity is further compounded by the need for strategic allocation of resources and budgetary considerations, which are critical yet challenging aspects of effective infrastructure management <cit.>, <cit.>.

talk about how decision making under uncertainty and RL, learning-based approaches work

In recent years, there has been a significant shift towards data-driven methodologies, particularly with the advent of machine learning techniques like reinforcement learning (RL) and imitation learning (IL) <cit.>, <cit.>. These approaches offer a promising avenue for decision-making under uncertainty, allowing for adaptive and proactive infrastructure management strategies <cit.>, <cit.>. RL, in particular, has shown remarkable success in various domains, thanks to its ability to learn optimal policies through interaction with an environment <cit.> ,<cit.>. However, the application of these techniques in infrastructure management is still in its infancy, primarily due to the lack of suitable simulation environments that can accurately model the complexity and scale of these systems <cit.>, <cit.>. There is a strong need for a unified framework modeling infrastructure management problems. Such a framework should provide a natural and intuitive way to represent the uncertainties and limitations in observability while ensuring scalability to handle large-scale, real-world problems. This tool would enable rapid progress in the field, providing a simulation environment for training and realistic benchmarks for comparing learning-based approaches, traditional optimization-based approaches, and rule-based methods.




Talk about what's needed for RL/IL/learning approaches. Also talk about how this need evolves in setting of infra management
Also talk here about existing work of RL and infra management a little bit.
talk about how infralib standardizes and provides unified framework - begin by talking about the need for one.


To address these challenges, we introduce InfraLib, a comprehensive and versatile simulation framework designed for modeling and analyzing large-scale infrastructure management problems. InfraLib provides a realistic and granular representation of infrastructure systems by integrating a hierarchical model that captures the intricate relationships between different components and facilities <cit.>. Moreover, InfraLib employs a stochastic approach to mimic the real-world uncertainties and partial observability inherent in infrastructure systems <cit.>, enabling the development and validation of infrastructure management strategies that are robust to the challenges faced in real-world scenarios.

Beyond serving as a simulation tool, InfraLib includes features for analysis, expert data collection, and modeling real-world budget schedules and failure modes. These features make it a valuable resource for both researchers looking to develop and test new management strategies and practitioners aiming to optimize operational efficiencies in real-world settings. 

In this paper, we present a detailed overview of the architecture and capabilities of InfraLib, highlighting its capabilities and potential applications. We demonstrate the ability of InfraLib to create realistic scenarios for the deployment and evaluation of learning-based approaches, showcasing its ability to model the complexities and challenges encountered in real-world infrastructure systems. Furthermore, we provide a series of benchmarks and environments to illustrate the utility and scalability of InfraLib in facilitating the development and comparison of novel management strategies.

The rest of this paper is organized as follows. Section <ref> introduces key concepts and background information on infrastructure management, Partially Observable Markov Decision Processes, and data-driven approaches for decision making. Section <ref> formalizes the infrastructure management problem as a POMDP and discusses the various research challenges that arise in this context. We then present InfraLib in Section <ref> and Section <ref>, detailing its structure, component dynamics, and key functionalities. Section <ref> delves into the human interface aspects of InfraLib, including tools for expert data collection and analysis. Finally, in Section <ref>, we showcase example environments and benchmarks to demonstrate InfraLib's utility, versatility, and scalability. 



§ PRELIMINARIES AND BACKGROUND



In this section, we provide background on the infrastructure management domain, including the hierarchical nature of infrastructure systems, the metrics used to quantify component health, the dynamics of component deterioration, and the budget constraints that govern infrastructure management. We also introduce the concepts of Partially Observable Markov Decision Processes and data-driven approaches for decision-making. We start by defining the notation used in the paper. 

Given a finite set , || denotes its cardinality and Δ() denotes the set of all probability distributions over the set . ℕ_0 denotes the set of natural numbers including 0 i.e. ℕ_0 = {0,1,2, …}. 



 §.§ Infrastructure Hierarchy and Management


Infrastructure management is inherently hierarchical, comprising multiple layers of organization. At the base level, we have individual components which are the smallest units of individually managed infrastructure elements. These components are grouped into units, which are collections of components that are managed together. Multiple units collectively form a facility, which is the highest level of organization in the infrastructure hierarchy. This hierarchical structuring in inherent in real-world infrastructure systems and is crucial for systematic management and decision-making processes.

The condition of each component in the infrastructure is characterized by a Condition Index (CI) <cit.>, a metric that reflects its health status. The CI of a component quantitatively represents the health of a component and it deteriorates over time due to environmental factors, wear-and-tear, and in some cases catastrophically due to a manufacturing defect or an external event. This deterioration is typically stochastic, arising from unpredictable environmental interactions and the complex nature of infrastructure materials. Moreover, the CI is not always directly observable, necessitating periodic inspections to estimate its current state. These inspections, while essential, incur additional costs. Even when the CI is observed by inspection, the observation is subjective depending on the inspector and the inspection method, and can be noisy.


Management of infrastructure systems at the component level include inspection, repair, and replacement. Inspection provides an estimate of the CI at a cost, replacement involves completely substituting the component at a higher expense, and repair, a more cost-effective option, aims to improve the CI. These actions are fundamental to maintaining the overall health of the infrastructure system.




 §.§ Partially Observable Markov Decision Process

A discrete-time finite-horizon POMDP M is specified by the tuple (, , , T, Z, R, H), where  denotes a finite set of states,  denotes a finite set of actions, and  denotes a finite set of observations. T :  × A →Δ() denotes the transition probability function, where Δ() is the space of probability distributions over . Furthermore, and Z : ××→Δ() denotes the observation probability function where Δ() is analogous to Δ(). Finally, R : ×→ [-R_min, R_max] denotes the reward function and H ∈ℕ_0 denotes the finite planning horizon.


For the above POMDP, at each time step, the environment is in some state s ∈ and the agent interacts with the environment by taking an action a ∈. Doing so results in the environment transitioning to a new state s̅∈ in the next time step with probability T(s,a,s̅). Simultaneously, the agent receives an observation o ∈ regarding the state of the environment with probability Z(o|s̅,a) which depends on the new state of the environment and the action taken by the agent. In a POMDP the agent doesn't have access to the true state of the environment. However, the agent can update it's belief about the true state of the environment using this observation. The agent also receives a reward R(s,a). 

The problem of optimal policy synthesis for a finite-horizon POMDP is that of choosing a sequence of actions which maximizes the expected total reward. 
[∑_t=0^Hr_t] where r_t is the reward earned at time instant t.  
Hence the optimal behavior may often include actions which are taken simply because they improve the agent's belief about the true state. After reaching the state s^', the agent receives observation o ∈ with probabily Z(o|s^',a). Let the belief b be a probability distribution over . Then, b(s) denotes the belief state and the agent updates the belief state according to Bayes' rule. 




 §.§ Data-Driven Approaches for Decision Making


Data-driven approaches, such as reinforcement learning (RL), inverse reinforcement learning (IRL) <cit.>, and imitation learning (IL), have emerged as powerful tools for learning optimal decision-making policies in sequential decision-making problems. In RL, an agent learns to make decisions by interacting with an environment modeled as a (PO)MDP. The agent's goal is to learn a policy π: ×→ [0,1], which maps states to a probability distribution over actions, that maximizes the expected cumulative reward over a horizon H:

    π^* = _π_π[∑_t=0^H R(s_t, a_t)]

where R(s_t, a_t) is the reward obtained by taking action a_t in state s_t at time step t, and the expectation is taken over the trajectories generated by following policy π. RL algorithms can be broadly classified into value-based methods <cit.>, which learn a value function that estimates the expected cumulative reward from each state or state-action pair, and policy-based methods <cit.>, which directly learn a parametrized policy. RL's success in various domains can be attributed to its ability to adaptively learn optimal strategies through trial and error.

Inverse Reinforcement Learning (IRL) addresses the problem of learning a reward function that explains the behavior of an expert demonstrator. Given a set of expert demonstrations 𝒟 = (s_0, a_0), (s_1, a_1), …, (s_T, a_T), where (s_t, a_t) represents the state-action pair at time step t, the goal of IRL is to find a reward function R(s,a) that rationalizes the expert's behavior:

    R^* = _Rℒ(R |𝒟)

where ℒ(R |𝒟) is a likelihood function that measures how well the reward function R explains the expert demonstrations 𝒟. Common approaches to IRL include maximum entropy IRL <cit.> and Bayesian IRL <cit.>.

Imitation Learning (IL) focuses on learning a policy that mimics the behavior of an expert demonstrator. Given a set of expert demonstrations 𝒟, the goal of IL is to learn a policy π that generates behavior similar to the expert. IL can be approached through behavioral cloning <cit.>, which treats IL as a supervised learning problem and learns a mapping from states to actions by minimizing a loss function between the predicted actions and the expert actions:

    π^* = _π∑_(s,a) ∈𝒟ℓ(π(s), a)

where ℓ(·, ·) is the chosen loss function. Alternatively, apprenticeship learning seeks to learn a policy that achieves a similar expected cumulative reward as the expert policy under some unknown reward function, often by iteratively solving an RL problem with a reward function learned via IRL based on the expert demonstrations.

The effectiveness of RL and IL is heavily contingent on the availability of accurate and comprehensive simulation environments and expert demonstrations, which presents unique challenges in the infrastructure domain. Benchmarks and baselines are essential in the realm of RL and IL as they provide standard metrics and methods for comparing different approaches. Benchmarks offer predefined problems with set parameters and goals, allowing for a consistent and fair evaluation of various strategies. Baselines, typically consisting of established methods or algorithms, serve as a reference point to gauge the performance of new approaches. 



§ PROBLEM FORMULATION


Infrastructure management is a complex problem influenced by environmental, manufacturing, and operational factors. In real-world infrastructure systems, control over the environment and manufacturing is limited. Therefore, the space of possible decisions spans over the operational factors. We focus on optimizing management decisions under budget constraints, while ensuring that our model captures the stochastic nature of component deterioration and the partial observability of infrastructure condition.

In this section, we formalize the infrastructure management problem as a Partially Observable Markov Decision Process (POMDP) closely following the multi-component POMDP with shared budget formulation in <cit.> and proceed to discuss the challenges and research questions arising when optimizing infrastructure management decisions.



 §.§ Modeling Infrastructure Systems as POMDPs


We model a large-scale, hierarchical infrastructure system as a single collection of components. Let n denote the total number of components in the infrastructure system. We model the condition index (CI) dynamics of each component as an independent POMDP with the assumption that the detoriation dynamics of individual components are not related. Specifically, let M^i denote a POMDP representing the dynamics of component i for i ∈{1,2,…,n}. For each component, the state space ⊂ℕ_0 is given by = {0,1,2, …, s_max}, where s_max∈ℕ_0. The state at any time step k denotes the CI of the component at that time step. The observation space is given by 𝒪 = ∪{e}, where e ∈ℕ_0 is a null observation that does not provide any information regarding the true state of the system. 

The action space for each component is given by = {d,q,r,m} where (i) action d, the do nothing action, lets the component's CI transition to a new state following the deterioration dynamics, (ii) action q, the inspection action, follows similar state transition dynamics as action d and provides the true state as the observation, (iii) action r, the repair action, improves the CI of the component to a new state s' where s < s' ≤ s_max and provides the true state as the observation, and (iv) action m, the replace action, drives the component state to s_max, and also provides the true state as the observation.

The transition probability function for each component, governed by its deterioration dynamics D, is defined as

    T(s,a,s̅) = 
        1,   if s̅ = s_max and  a = m, 
    
        1,   if s̅ = s'  and  a = r, 
    
        D(s,s̅),   if s̅≤ s  and  a ∈{d, q},
    
        0,   otherwise.


Similarly, the observation probability function for each component is defined as

    Z(s̅,a,o) = 
            1,   if  o = s̅ and  a ∈{q, m, r}
    
            1,   if  o = e  and  a = d 
    
            0,   otherwise.

The reward function for each component depends on the objective and constraints of the research problem considered. We discuss the reward formulation in detail in later sections. 

In addition to the POMDP model M^i, each component is also associated with additional parameters and meta-data that capture the component's importance, maintenance costs, hierarchy, among other attributes. These parameters are essential for modeling the infrastructure system as a whole and are used to define the budget constraints, resource availability, and other operational considerations. Let Ω^i denote the set of additional parameters associated with component i including λ^i ∈ [0,1], the relative importance of component i in the infrastructure system, δ^i ∈ [0, s_max], the failure threshold of component i, and c^i_d, c^i_q, c^i_r, c^i_m, the costs associated with taking actions d, q, r and m respectively for component i.  


We manage the collection of n components {(M^1,Ω^i), (M^2,Ω^2), …, (M^n,Ω^n)} with a shared budget B. The budget B is allocated across the components to perform maintenance, repair, and replacement actions. Assume that the number of d^i,q^i, m^i and r^i actions taken for component i for a horizon H are n^i_d,n^i_q, n^i_m and n^i_r respectively. Then, the total cost incurred for the all the components for the horizon H is given by:

    C_H = ∑_i=1^|| (n^i_dc^i_d + n^i_qc^i_i + n^i_rc^i_r + n^i_mc^i_m).




 §.§ Problem Statement

For an infrastructure system {(M^1,Ω^1), (M^2,Ω^2), …, (M^n,Ω^n)} with a shared budget B, we study a series of research problems that aim to find an optimal policyπ^* that maximizes the time before the components reach their failure thresholds while operating under the budget and other operational constraints. Formally, our goal is to find an optimal policy π^* under objective functions of the form

    π^* = _π[∑_t=0^H∑_i=1^nλ^i ·𝕀(s^i_t > δ^i)]

while ensuring that, at minimum, the total cost incurred over the time horizon H does not exceed the total budget B i.e. C_H = ∑_i=1^n (n^i_dc^i_d + n^i_qc^i_q + n^i_mc^i_m + n^i_rc^i_r) ≤ B.




 §.§ Research Problems in Infrastructure Management

In this section we present some of the interesting research questions that arise while solving the infrastructure management formulation presented in the previous section.



  §.§.§ Hierarchical Decision Making

Infrastructure systems exhibit an inherent hierarchical structure, including components, units, and facilities. This hierarchy adds complexity to decision-making processes, as in many real-world applications the decisions are often made at different levels of the hierarchy. Further, the decisions made at lower levels can have cascading effects on higher levels. 



  §.§.§ Stochastic Component Deterioration

The deterioration of infrastructure components is inherently stochastic, influenced by environmental factors, wear-and-tear, and unexpected events. Accurately modeling this stochastic deterioration and integrating these models into decision-making processes is crucial, especially to ensure that the data-driven approaches trained in simulation environments work robustly in real-world deployments. Some additional challenges include how RL can be adapted to operate in environments with high levels of uncertainty and variability.



  §.§.§ Budget Constraints

Management of infrastructure systems often involves operating under strict budget constraints. Key research questions involve optimal resource allocation for maintenance, repair, and replacement actions, and balancing short-term costs against long-term infrastructure health and functionality. Further, the budget is often not fixed and can vary over time, requiring adaptive policies and policies that plan over a long horizon to ensure optimal resource utilization.



  §.§.§ Partial Observability

Most learning-based approaches including state-of-the-article reinforcement learning algorithms often assume a full observability of the environment. However, in infrastructure management, the condition of components is often only partially observable, requiring costly inspections to estimate the true state. Research is needed to develop algorithms that can effectively handle partial observability and make informed decisions based on uncertain or incomplete information.



  §.§.§ Interpretability

Unlike other applications of learning-based decision making, interpretability and explainability are crucial for the adoption of intelligent approaches in real-world infrastructure management. Infrastructure management often involves critical decisions that impact public safety and economic stability and therefore it is essential to understand why learned-policies make certain decisions and how to ensure that these decisions align with domain-specific perceptions, requirements and constraints.



  §.§.§ Sim2Real Gap

Simulation environments are essential for training and evaluating RL models. However, the gap between simulated environments and real-world dynamics can lead to ineffective policies when deployed in the real world. Research is required to develop simulation environments that accurately reflect the complexity and stochasticity of the real world, as well as algorithms that can bridge the Sim2Real gap.



  §.§.§ Sparsity and Time Scales

Unlike typical environments modeled as (PO)MDPs, infrastructure management decisions are made over long time horizons and often involve sparse actions where the agent often has to stay idle and wait for the environment to evolve before taking an action. Further, the decisions often have long-term impacts, with rewards or consequences of actions not immediately observable. Research into approaches and reward shaping techniques capable of handling sparse rewards is crucial for effective infrastructure management.



  §.§.§ Scalability and Computational Efficiency

Real-world infrastructure systems are large-scale, often involving millions of individual components distributed across facilities. The simulation environment and the decision-making algorithms need to be scalable, capable of handling large-scale problems efficiently. Further, policies trained using transfer-learning and meta-learning methods should be able to generalize across different infrastructure systems and scenarios while maintaining computational efficiency. Research into scalable solution methods and efficient computation techniques is vital for practical applicability.





§ INFRALIB


InfraLib is a comprehensive modeling, simulation, and analysis framework designed to enable research into data-driven, learning-based decision making for infrastructure management under uncertainty. It provides predefined, structured environments while also allowing users to flexibly define custom scenarios and constraints. The code, documentation, example environments, and tutorials are available at <https://infralib.github.io/>.



 §.§ InfraLib Structure

InfraLib framework adopts a modular architecture, which enables separation of concerns and easy extensibility. The core infrastructure model is designed to be highly configurable, allowing users to define custom components, deterioration models, objectives, constraints, and management actions. The hierarchical structure of infrastructure systems is also configurable, enabling users to group components into units and facilities in domain-specific ways.

InfraLib is implemented as a Python library, leveraging popular scientific computing packages like NumPy and Numba for efficient computation. The framework is designed to be user-friendly, with a simple and intuitive API that abstracts the underlying complexity. This makes InfraLib accessible to a wide range of users, from researchers and practitioners to students and educators. The functionalities of InfraLib library are organized into different modules, with the Core module providing the foundational capabilities of modeling and simulating large-scale infrastructure systems. Additional modules, including the analysis module, visualization module, and expert data collection module, offer advanced tools for understanding infrastructure dynamics and assessing policy performance. The input-output module ensures that all data is stored and retrieved in a standardized format, facilitating seamless integration with external tools and libraries and enabling reproducibility and collaboration.

A key emphasis in InfraLib's design is scalability and computational efficiency. Through a scalable software architecture and efficient algorithms, the framework can simulate infrastructure systems comprising millions of components and spanning long time horizons. This massive scale is crucial for bridging the gap between research and the complexity of real-world infrastructure networks.



 §.§ Component Condition and Cost Dynamics


In InfraLib, the Condition Index of each component, used to quantitatively represent the component's current state of degradation or functionality, takes values in the range [0, 100]. The CI evolves stochastically over time, and the dynamics of component deterioration are modeled as a Markov chain with transition probability function D(s,s'). Following the literature <cit.>, we model the CI dynamics as a Weibull distribution tailored to each component's deterioration pattern. The Weibull distribution is a flexible model that can capture a wide range of real-world deterioration behaviors, from early-life failures to wear-out failures. The Weibull distribution is parameterized by shape parameter k and scale parameter λ, with the CDF given as:

    F(x; k, λ) = 1 - e^-(x/λ)^k.


For every component, based on real-world data, we assume access to the mean and variance of the shape and scale parameters its Weibull distribution. To generate the transition function D^i(s,s') for component i, we collect multiple samples of k and λ from their respective distributions:

    k ∼ N(μ_k, σ_k^2)


    λ∼ N(μ_λ, σ_λ^2)

and then compute the transition probabilities by estimating them from scaled Weibull CDF values where in each sample trajectory, the CI at time step k is given by:

    CI(t) = ⌊ 100 × (1 - F(t; k, λ)) ⌋.

This equation ensures that when t is 0 (representing a new or fully functional component), CI(t) is 100 (least degraded), and as t increases towards the end of the component's expected lifecycle, CI(t) approaches 0 (most degraded).

The cost of repairing a component in InfraLib is designed to reflect the degree of degradation and the urgency of intervention, based on the condition index. The cost is dynamically calculated based on the state of the component at the time of repair and the effectiveness of the repair action, and is given as:

    c_r^i = (100 - s^i/100 - δ^i)^α^i× c_m^i

where s^i is the current CI of component i, δ^i is the failure threshold, α^i is a parameter that adjusts the sensitivity of repair costs to damage, and c_m^i is replacement cost of component i. This formulation ensures that repairing severely damaged components is proportionally more expensive, aligning repair costs with the component's condition and the urgency of repairs.




§ INFRALIB FUNCTIONALITY



InfraLib supports resource allocation problems under several constraints and scenarios that are common in real-world infrastructure management. In this section, we highlight some of the key functionalities of InfraLib and discuss how they can be used to address critical research questions in infrastructure management.



 §.§ Optimal Budget Allocation

InfraLib is fundamentally designed to enable optimal budget allocation for large-scale infrastructure systems comprising numerous components. As discussed in <ref>, the modeling framework allows users to optimize actions while considering budget constraints, importance scores, and component deterioration dynamics. Figure <ref> illustrates a simulation of this model through the visualization of condition indices for different components.




In a given instance, InfraLib can simulate the evolution of millions of component instances of tens of thousands of component types over a long time horizon. The simulation accepts the actions taken on each component at each time step as input, and transitions the components to new states after verifying that the actions are feasible under the budget constraints. 



 §.§ Intermittent Component Availability

All the components in an infrastructure systems may not always be available for management actions. This is especially relevant in the case of real-world infrastructure systems that are in remote or inaccessible locations, or critical infrastructure components that cannot be taken offline for maintanance. InfraLib supports modeling intermittent component availability, where components can be marked as unavailable for certain time periods. Figure <ref> illustrates the condition index of a component that is intermittently unavailable for inspect, repair, and replacement actions.




In addition to enabling users to simulate and analyze scenarios where components are only available for inspection, repair, or replacement during specific time windows, InfraLib also allows users to evaluate the impact of these constraints on their management policies. 



 §.§ Cyclic Budget

InfraLib can model scenarios with a cyclic budget, where the total budget allotted for infrastructure management is reset to a fixed amount periodically. For instance, the budget could be replenished annually to a predetermined value. Under a cyclic budget schedule, the user can specify either a fixed budget and cycle length or a budget profile with replenished Budget and cycle length that varies over time. Any resources that are not utilized within the current cycle are forfeited and do not carry over.

In addition to the cyclic budget schedules that are directly supported, users can also define custom budget profiles that reflect the budget allocation patterns in their system. Modeling such budget schedules allows testing management policies under real-world resource constraints where budget allocations tend to be more complex than simple cyclic schedules. 



 §.§ Catastrophic Failures

InfraLib enables modeling of unexpected catastrophic failure events that severely impact infrastructure components instantly. Users can specify failure events to occur at predefined time steps during a simulation or optionally let the library generate random failure events based on a built-in, predetermined distribution. The catastrophic failures can affect one or more components, and can be configured based on the component metadata such as the component type, location, or facility to introduce spatial and temporal dependencies. Figure <ref> illustrates how the condition indices of components change in the event of a catastrophic failure.




Simulating catastrophic failures provides a mechanism to stress-test infrastructure systems and management policies. It is particularly crucial for evaluating the resiliency of management policies to extreme weather events, natural disasters, or other unforeseen circumstances.



 §.§ RL Environments

InfraLib generates standardized reinforcement learning environments that encapsulate the complexities of infrastructure management problems while maintaining compatibility with popular RL libraries. Given an infrastructure system modeled in InfraLib, we generate an RL environment E = (𝒮, 𝒜, 𝒫, ℛ, γ). The state space 𝒮 = ∏_i=1^n 𝒮_i ×ℝ^+ incorporates the condition indices of all components and the remaining budget. The action space 𝒜 = ∏_i=1^n 𝒜_i represents all possible combinations of actions across components. The transition probability function 𝒫(s' | s, a) = ∏_i=1^n T_i(s'_i | s_i, a_i) ·𝕀(b' = b - c(a)) is derived from component-level transition probabilities, where T_i is the transition function for component i, c(a) is the total action cost, and 𝕀 is the indicator function. The generic reward function ℛ(s, a, s') = ∑_i=1^n w_i · f_i(s_i, s'_i) - λ· c(a) balances the change in component conditions with action costs based on user specification.

To model partial observability, InfraLib can generate POMDP environments E' = (𝒮, 𝒜, 𝒫, ℛ, 𝒪, Z, γ), where the observation space 𝒪 = ∏_i=1^n (𝒮_i ∪u) ×ℝ^+ includes an unknown state u for uninspected components. The observation function Z(o | s, a) = ∏i=1^n Z_i(o_i | s_i, a_i) reflects the inspection history and recent actions. These environments provide standardized interfaces (, , ) compatible with popular RL libraries, facilitating the application and evaluation of RL algorithms to infrastructure management problems. 


So far, we have discussed InfraLib from the perspective of modeling and simulating infrastructure systems for learning decision-making policies and evaluating them under various constraints. In the following section, we will delve into the tools provided by InfraLib for analysis and data collection from experts.



§ INFRALIB HUMAN INTERFACE


Analysis of existing infrastructure management policies in a unified framework is crucial for enabling decision-makers to compare and evaluate different policies and their impact on different aspects of the infrastructure system. In addition, the ability to collect expert data from human decision-makers is essential for training imitation learning approaches that can leverage these demonstrations without specific reward functions or extensive exploration. In this section, we discuss the tools provided by InfraLib for expert data collection and analysis.

At the core of InfraLib's human interface for analysis and data collection is an intuitive web-based dashboard interface. The interface is powered by a simulation process running in the background and provides experts with detailed information about the current state of a simulated infrastructure system, including component condition indices, recent observations, and historical management actions. The experts can inspect components and allocate resources for maintenance, repair, or replacement based on their domain knowledge. A snapshot of the dashboard with the options available for the user for analysis is shown in Figure <ref>.



Behind the scenes, InfraLib logs the full trajectory of expert actions, observations, and environment states, and the metadata of the scenario analyzed by the expert. These demonstrations can be used to train imitation learning algorithms to mimic expert behavior or to infer expert's preferences and priorities using inverse reinforcement learning. The expert data collection process is designed to be seamless and user-friendly, allowing experts to focus on demonstrating their management strategies without worrying about the technical details of the simulation environment. In addition to the expert demonstrations collected using the dashboard, InfraLib also supports batch uploads of expert demonstrations generated externally.

InfraLib allows injection of expert knowledge directly into the simulations. Experts can specify replacement thresholds, priority rules, or repair strategies for different components. This expert knowledge can help make the simulations more realistic and guide the agent's exploration in reinforcement learning approaches revealing areas that require tighter mimicking of experts versus allowances for more agent creativity. The modular design also enables the use of expert sub-policies for managing specific components alongside learning-based controllers.











§ EXAMPLE ENVIRONMENTS AND BENCHMARKS


This section provides some sample problems and scenarios that can be modeled using InfraLib. The goal of this section is two fold: (i) to provide a set of ready-to-start scenarios where other researchers can directly test their approaches and use similar templates to design their own custom environments, (ii) to provide baseline benchmarks that other researchers can use to compare the performance of their approach.










 §.§ Champaign-Urbana Road Network Management

We model the road network in Champaign-Urbana, a metropolitan area in Illinois, United States, using InfraLib and simulate its deterioration without intervention. The road network data is sourced from OpenStreetMap (OSM), which provides detailed attributes for each road segment. These attributes are used to parameterize the deterioration dynamics in the simulation.

The key OSM attributes utilized in the model include the road type (specified by the highway tag), number of lanes, maximum speed limit, and surface material. The highway tag is particularly important as it classifies the road into categories such as motorway, trunk, primary, secondary, tertiary, residential, and service, which have distinct deterioration characteristics.

In the InfraLib model, each road segment is treated as a separate component with its own Weibull deterioration dynamics. The shape and scale parameters of the Weibull distribution for each segment are determined based on a combination of the OSM attributes. For instance, segments with a higher speed limits are assumed to have higher deterioration rates compared to lower-class roads <cit.>. Similarly, the surface material affects the deterioration, with asphalt and concrete roads having slower deterioration compared to gravel roads <cit.>. 

The number of lanes and road width also influence the deterioration dynamics, as wider and multi-lane roads typically have higher construction standards and are more resilient to wear and tear. We note that the generated deterioration dynamics utilizing road attributes are illustrative, and serve to showcase InfraLib's capabilities in modeling and simulating infrastructure systems. However, it's important to emphasize that these generated dynamics are not intended to be an accurate representation or predictive model of real-world system behavior.

Figure <ref> illustrates the simulated deterioration of the Champaign-Urbana road network over a 50-year period using InfraLib. The condition index of each road segment is visualized on a color scale, with blue indicating good condition and red indicating poor condition. As seen in the figure, the road network progressively deteriorates over time, with different segments deteriorating at different rates based on their attributes.

This realistic simulation of the Champaign-Urbana road network showcases InfraLib's ability to model large-scale infrastructure systems with heterogeneous components having unique deterioration characteristics. 



 §.§ LargeSys-100K - Large-Scale infrastructure System Management

To demonstrate InfraLib's scalability and ability to handle large-scale infrastructure systems, we introduce the LargeSys-100K benchmark. This synthetic dataset consists of a massive network with 100,000 component instances spanning 1000 different component types. Each component type has 100 instances, resulting in a total of 100,000 components.

The deterioration dynamics and cost parameters for each component type in LargeSys-100K are synthesized based on realistic ranges observed in real-world infrastructure data. The Weibull distribution shape and scale parameters, inspection costs, repair parameters, and replacement costs for each component type are randomly generated while ensuring they fall within these practicable ranges.

LargeSys-100K serves as a standardized benchmark for comparing the performance and scalability of different infrastructure management approaches. By utilizing a large number of components and component types, LargeSys-100K aims to test the scalability and computational efficiency of infrastructure management algorithms. The vast scale of this benchmark poses challenges in terms of memory usage and computation time, pushing the boundaries of optimization and learning-based approaches.

Moreover, the diversity of component types in LargeSys-100K adds an additional layer of complexity. With varying deterioration dynamics and costs across component types, algorithms must be able to effectively prioritize and allocate resources considering the heterogeneity of the infrastructure system. 







   





§ CONCLUSION AND FUTURE WORK

This paper introduced InfraLib, a comprehensive and versatile simulation framework designed to model and analyze large-scale infrastructure management problems. By providing a realistic and granular representation of infrastructure systems, InfraLib enables the application of reinforcement learning and other learning-based decision-making techniques to the complex domain of infrastructure management. The framework's hierarchical and stochastic approach accurately captures the nuances of real-world systems, including budget constraints, resource availability, and the geographical distribution of components. Through a variety of realistic scenarios and benchmarks, InfraLib demonstrates its potential to significantly impact the field, offering researchers and practitioners the means to develop, test, and refine strategies for efficient and effective infrastructure maintenance and allocation.

Looking ahead, there are several promising avenues for future work and enhancements to InfraLib. One direction is to expand the framework's capabilities to model finite crew allocation and scheduling problems, which go hand-in-hand with infrastructure management. Another direction is to integrate transfer learning and meta-learning algorithms to enable the rapid adaptation of learned policies to new infrastructure systems or changing environmental conditions. Furthermore, integrating InfraLib with other tools and platforms commonly used in infrastructure management, such as geographic information systems (GIS) and asset management software, would streamline the data exchange process and facilitate the adoption of learning-based approaches in practice. Finally, building a vibrant community around InfraLib is crucial for its long-term success and impact. Encouraging researchers and practitioners to contribute new components, deterioration models, and management strategies will ensure that the framework remains up-to-date and relevant. 


unsrtnat


