Take the 2-minute tour ×

Reward R(S) in a Markov Decision Process is a mapping from a State S -> Bounded number. I want to know how a Utility Function is defined for an MDP.

I think it has to be a mapping from a sequence of states to a bounded number - where the utility function is just the Summation of the rewards R(S) of each of the states visited by an agent. Would that be correct?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

Your notation is not completely clear to me (especially with regards to bounded number).

Markov Decision Process (MDP) is a Markov process (MP) where (probabilistic) control is allowed, that name usually refers to discrete-time processes. Probabilistic control means that at each step you choose just a distribution of the next value from the class of admissible distributions. Again, $$\text{MDP} = \text{MP} + \text{probabilistic control}.$$

Talking about the reward $R:S\to\mathbb R$ the name for the model I met in the literature (Computer Science as well) is Markov Reward Model (MRM): that is MP + reward function, not necessary with a control in general. The underlying process for MRM can be just MP or may be MDP.

Utility function can be defined e.g. as $U=\sum\limits_{i=0}^nR(X_i)$ given that $X_0,X_1,...,X_n$ is a realization of the process. In the case when the underlying process is uncontrolled, one studies distribution of this function. In the case when control is allowed, optimal control problems are solved. On the other hand, nothing will prevent you from defining $U$ is another form and studying its properties.

share|improve this answer
add comment

Your Answer


