Title: Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints. Abstract: Many real-world settings involve costs for performing actions; transaction costs
in financial systems and fuel costs being common examples. In these settings,
performing actions at each time step quickly accumulates costs leading to vastly
suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and
ultimately, damage. Determining when to act is crucial for achieving successful
outcomes and yet, the challenge of efficiently learning to behave optimally when
actions incur minimally bounded costs remains unresolved. In this paper, we intro-
duce a reinforcement learning (RL) framework named Learnable Impulse Control
Reinforcement Algorithm (LICRA), for learning to optimally select both when
to act and which actions to take when actions incur costs. At the core of LICRA
is a nested structure that combines RL and a form of policy known as impulse
control which learns to maximise objectives when actions incur costs. We prove
that LICRA, which seamlessly adopts any RL method, converges to policies that
optimally select when to perform actions and their optimal magnitudes. We then
augment LICRA to handle problems in which the agent can perform at most k < ∞
actions and more generally, faces a budget constraint. We show LICRA learns the
optimal value function and ensures budget constraints are satisfied almost surely.
We demonstrate empirically LICRA’s superior performance against benchmark
RL methods in OpenAI gym’s Lunar Lander and in Highway environments and a
variant of the Merton portfolio problem within finance.