Control-limit policies for a class of stopping time problems with termination restrictions
We consider a class of stopping time problems in which the state of the process evolves according to a discrete-time Markov chain and the action which terminates the process is available only in a certain subset of the state space. For the objective of maximizing total expected discounted reward we develop an infinite-horizon Markov decision process formulation and derive two families of sufficient conditions under which the optimal policy exhibits a control-limit structure. We present numerical examples to illustrate the two families of conditions we derive are satisfiable and do not overrule each other. We also demonstrate that a family of our conditions is weaker than those in the literature which guarantee the optimality of control-limit policies for a special case of the problem we study.