A "value function" tells an agent how much reward it will get following a policy formula_1 starting from state formula_5. It represents how "desirable" it is to be in a certain state. Since the value function isn't given to the agent directly, it needs to come up with a good guess or estimate based on the reward it's gotten so far. Value function estimation is the most important part of most reinforcement learning algorithms.