In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the specific case of RL? What are examples of features in RL? Which RL algorithms require the specification of features? Why do we need "features" in RL?


What are "features" in the specific case of RL?

In RL, the supervised learning component is used to learn a function approximation for either a value function or a policy. This is usually a function of the state, and sometimes of the state and action. The features are therefore explanatory variables relating to the state (or state and action) and need to be sufficient to explain the value or optimal action of that state.

In RL literature, there is often no difference between terms "state representation" or "features". Although you may need to process the state representation using feature engineering, into suitable features for whatever supervised learning you are using. With neural networks, this may involve normalising values that might otherwise cause problems.

What are examples of features in RL?

In OpenAI's gym, CartPole-v1 has the following features describing state:

  • Cart Position
  • Cart Velocity
  • Pole Angle
  • Pole Velocity At Tip

The positions and speeds are measured in arbitrary units in this case, but you might consider them to be in SI units - metres along the track, metres per second, radians, metres per second.

Typically in RL, you will want features that:

  • Describe the state (and possibly the action, depending on implementation)

  • Are feasible to collect by observation of the environment

  • Together with the action choice, can adequately explain expected rewards and next state. In RL terms, they should in aggregate possess the Markov property

Which RL algorithms require the specification of features?

None of them*. However, without descriptive features of state, you are limited to enumerating all possible states, and using tabular methods. Working with tabular methods constrains you in practice by the time it takes to fully explore all the states, and the memory space required to represent all the individual estimates of value.

If you are looking into solving your problem using DQN, A3C or other "deep" RL method that uses neural networks, then you need to be thinking in terms of a state representation that is composed of features, and to treat those features as if they were inputs to the same neural networks used in supervised learning.

Why do we need "features" in RL?

To work with complex environments with large state spaces that cannot be solved without some form of approximation and (hopefully) generalisation.

* This is slightly more complex with policy gradient methods, as they do require that you use a function approximator, such as a neural network. However, it is possible to make the "features" in that case a one-hot coded vector of the state enumeration, which is the same representation as tabular value-based methods. There is little reason to do that in practice, but it is possible in principle.

  • $\begingroup$ So, "features", in RL, are used to build the dataset to train the function approximator? $\endgroup$ – nbro Sep 24 '18 at 8:48
  • $\begingroup$ @nbro: Sort of. In RL you don't have a static dataset, and learning can be entirely online. In DQN you might store observations in an experience replay table which is more similar to a dataset, but again changes over time, so it's not 100% the same. Also, in both cases, the target/label is entirely dynamic, you don't store the equivalent of train_y . . . $\endgroup$ – Neil Slater Sep 24 '18 at 8:53
  • $\begingroup$ In the case the learning process is online, e.g. without using the experience replay technique, how would you then use the features? Would the dataset only contain one value for each feature, that is, the dataset would be of length $1$? $\endgroup$ – nbro Sep 24 '18 at 9:32
  • $\begingroup$ @nbro: Yes, essentially you have repeated datasets of length 1, as in SGD. Although you probably don't see the term "dataset" used in that case. In RL, there isn't really stable dataset even with experience replay, as you are continuously sampling new experience. It is much the same as any online supervised learning technique - those won't usually describe the data as being in a "dataset" either. $\endgroup$ – Neil Slater Sep 24 '18 at 10:18

Your Answer

