Crucial Considerations Fri, 08 May 2020 15:02:41 +0000 en-US hourly 1 78421812 Why Evolution Has No Goal Tue, 02 Aug 2016 18:23:20 +0000 Share

How evolutionary biology is being misunderstood

The notion that evolution has a goal/purpose is a widespread misconception that comes in several variations. Many people falsely believe that evolution was working towards humans as a final product. But from a biological perspective, there is really no reason to single out the human species specifically. Evolution produced a “tree of life”, with humans being an ordinary twig among many others. It did not produce a “ladder of life” where humans would throne at the top. Homo sapiens is one species out of many, and all currently existing organisms have an evolutionary history of the same length, leading back to the same origin most likely.

Even if one doesn’t view humans as some kind of “end product”, one might still believe that evolution is operating according to some higher purpose, or that it necessarily strives into a particular direction. More complexity, harmony, progress or  some kind of plan that needs to be adhered to. But all of that, too, is mistaken. At most, evolution has certain tendencies – but no goal or purpose.

Why evolutionary biology is being misunderstood

Evolutionary biology is difficult to grasp because it strongly violates some of our basic intuitions about how things work. We have the tendency to favor teleological (“goal-directed”) explanations over causal explanations (Kelemen & Rosset, 2009) even though with regard to nature, teleological explanations are wrong. For instance, when children are asked whether rain happens “for plants to grow” or “because water condenses in the clouds,” they prefer the first explanation. This likely explains, in combination with religious motivations and perhaps also vanity or wishful thinking, why people have such difficulty to properly understand the evolutionary mechanism.

The mechanism behind evolution

Evolution’s mechanism is surprisingly simple. Those variations (alleles) of a gene which give an individual a comparative advantage in terms of biological fitness – in being beneficial for successful reproduction – will automatically become more prevalent in the genepool. Whether a given mutation leads to a fitness advantage depends on the environment of the organism in question. Out of the random variation, which is the result of mutations/copying-errors (which can be the result of exposure to radiation, metals or chemical substances), only a small percentage actually increases the fitness of an individual. Those mutations tend to prevail and become widespread, whereas mutations that lead to a disadvantage will likely be weeded out of the genepool. Even though the variation is originally random, a non-random subset of it – the fitness-benefitting components – ends up conserved through natural selection. This mechanism results in organisms adapting to better survive and reproduce in their environment.

No Foresight

It’s important to realize that this process is always happening in small steps. For large changes in the phenotype of an organism, many mutations are required. Most of the mutations which happened throughout the evolutionary past had to give their first bearer a fitness advantage – otherwise they would likely not have become the prevalent allele in the genepool. It could be that a first mutation is good insofar as it would pave the way for a subsequent mutation, which then, together with the first one, proves to be extremely beneficial. However, if the first mutation does not provide a fitness advantage on its own, or worse, if it proves to be harmful in standalone, without the subsequent fitness-enhancing mutation, then it is very unlikely that it will stay around long enough for the fitness-enhancing mutation to come along too, by chance. This is the case despite the great benefits the two mutations could provide in combination. There is simply no way for natural selection to anticipate such future benefits.

Natural selection has no foresight. In each round, the individuals that die early or are comparatively bad at reproducing are weeded out. There have likely been many situations in a lineage’s evolutionary history where small, slightly fitness-reducing steps would have provided a benefit in the long run, perhaps leading to a lineage culminating in the dominant species across many continents. However, evolution never “goes down” in fitness, because organisms with a lower fitness (by definition) will be more likely to lose the race for reproduction, being outcompeted by their conspecifics.

Conversely, there is also no mechanism that prevents short-sighted improvements that will turn out to carry a design disadvantage in the long run. Once a lineage, following the selection pressure of the current environment, went through a given adaptation, there is no convenient way for evolution to go back to the sketchpad to pursue a different branch. The sole exception would be if the environment changes in a way that exactly reverses the selection pressures and the past mutations – but that, too, is of course very unlikely.

The evolutionary biologist Richard Dawkins (1986) used the analogy of a “blind watchmaker” to describe natural selection: “blind” because it is a mechanical process lacking any sort of foresight, and “watchmaker” because nonetheless, it manages to produce structures (living organisms!) that look as though they were deliberately designed.

Stuck in local optima

Evolution being blind implies that lineages only ever strive towards local optima in regard to the fitness landscape of all possible gene combinations. Global optima, where reaching them would require foresight or planning, are thus not to be expected as a result of evolution.

An example for being stuck in local optima is eye design. Different kinds of eye designs have evolved more than forty times independently of each other (Land & Nilsson, 2002). Not all their basic designs for visual organs had the same potential for sight improvements in the organism’s evolutionary future. Because evolution lacks foresight, all kinds of designs evolved, as long as there was a continuous pathway of beneficial mutations leading up to them.


Contrary to expectation, it is also not the case that evolution is always at work with the same pace. Evolutionary changes in a population happen when the relative frequency of alleles in the genepool is changing. In large populations where no migration happens and where the environment remains constant, it is well possible that very few such changes happen and that evolution thus remains static for long periods of time. This phenomenon contradicts the notion that evolution is continuously striving towards a specific direction.

Changes of Direction

It was explained further above that, in terms of reproductive fitness, the evolutionary trend is always upwards. This should not be confused with claims about complexity, size or abilities having to go “upwards” as well: Evolutionary history is full of examples where lineages became smaller. One example would be the common sparrow, whose ancestors were predatory dinosaurs.

Further, depending on the environmental conditions, it is even possible that abilities are lost or stunted. Cave fishes, living in completely dark caves under the surface, have eyes that, unlike their ancestors’, are permanently degenerated. As their dark environment lacks any selection pressures for sight, mutations that harmed eyesight (and thereby maybe saved energy for other organs) where not to the fishes’ disadvantage.


Evolution has no purpose; it simply happens. There is no reason to assume that evolution comes with some objective  “improvement”. The only thing that is constantly improving is the adaptedness of individuals to their given environment. Because the environment changes, this “progress” always remains relative – progress relative to the adaptedness to an environment.

In case we want to metaphorically talk about a “goal” of evolution, then the goal is simply the copying success of genes. But that, too, does not happen in a strategically planned manner, for evolution is – metaphorically speaking again – “blind and indifferent.”


Kelemen, D. & Rosset, E. (2009). The human function compunction: Teleological explanation in adults. Cognition, 111(1), 138-143.

Land, M. F. & Nilsson, D. E. (2002). Animal Eyes. Oxford University Press. New York.

Dawkins, R. (1986). The Blind Watchmaker. Longman, Harlow.

The Ethics of Self-Driving Cars Wed, 13 Jan 2016 10:31:30 +0000 Share

This comic was made by Anum Yoon, who kindly allowed us to post it here.


Self-driving cars, autonomous cars, driverless cars – regardless of what you want to call them – are expected to revolutionize the entire automobile industry. For over a century, cars have consisted of a fairly straightforward combination of wheels, steering system, engine, and driver. It’s no wonder that the announcement of this new technology has launched a global race that has automakers and tech companies scrambling to develop the best autonomous vehicle technology. And according to Morgan Stanley, self-driving cars will be commonplace by 2025.

So talks about technology aside, let’s dive into the ethics and philosophy behind these vehicles, which is what this infographic is about.

The Laws of Robotics

In 1942, Isaac Asimov, sci-fi author and professor introduced the three laws of robotics.

  • The First Law states that a robot may not injure a human being or, through inaction, allow a human being to come to harm.
  • The Second Law outlines that a robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
  • The Third Law states that a robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
  • He later added a Fourth Law, also called the Zeroeth Law; a robot may not harm humanity, or, by inaction, allow humanity to come to harm.

Can the clear rules-based code of a computer handle the nuances of ethical dilemmas?

Let’s take a look at a few hypothetical scenarios:

The Trolley Problem

You are the driver of a trolley that has broken brakes. Fortunately you still have the ability to steer the train from the main track to an alternate track. You can see the two tracks right ahead of you:

  • The main track has five workers
  • The alternate track has one worker

Both tracks are in narrow tunnels so whichever direction you choose, anyone on that track will be killed. Which way will you go? Would you let the train continue down the main track and kill five, or will you switch it onto the alternate track to kill one?
Most people respond to the Trolley Problem by saying they would steer the train onto the alternate track because their moral intuition tells them that it’s better to kill only one person rather than five.

Now for a little modification to this hypothetical scenario.

The runaway trolley is speeding down a track about to hit five people. But this time, you’re on a bridge that the train is about to pass under. The only thing that could stop the trolley is a very heavy object. It just so happens that you are standing next to a very large man. Your only hope to save the five people on the tracks would be to push the large man over the bridge and onto the track. How would you proceed?
Most people strongly oppose this version of the problem – even the ones who had previously said they would rather kill one person as opposed to five. These two scenarios reveal the complexity of moral principles.

The Tunnel Problem

You are traveling on a single lane mountain road in a self-driving car that is quickly approaching a narrow tunnel. Right before you enter the tunnel, a child tries to run across the road but trips right in the center of the lane, blocking the entrance to the tunnel. The char only has two options:

  • To hit and kill the child
  • Swerve into the wall, thus killing you

How should the car react?

Now that the age of self-driving cars have dawned upon us, this new technological innovation has given ethical dilemmas such as the tunnel and trolley problems a new relevance.

Hypothetical scenarios like the Tunnel Problem present some of the real difficulties of programming ethics into autonomous vehicles. In a survey asking how people would like their car to react in the Tunnel Problem scenario, 64% of respondents would continue straight and kill the child. 36% would swerve and kill the passenger.
But who should get to decide?

44% of those surveyed felt that the passenger should make major ethical decisions. 33% felt that lawmakers should be the ones who decide, 12% felt that the decision should lie with the manufacturers and designers. The remaining 11% responded with “other.”

Ethics is a matter of sharing a world with others, so building ethics into autonomous cars is a lot more complex than just formulating the “correct” response to a set of data inputs.

Here’s one last ethical scenario for driverless cars.

The Infinite Trolley Problem

The Infinite Trolley Problem, introduced by autonomous vehicle advocate Mitch Turck, is where a single person is on the tracks. This person can easily be saved by simply halting the trolley, but that would inconvenience the passengers. So for this variant, the question is not “would you stop to save someone” but rather, “how many people need to be on board the trolley for their inconvenience to be valued more than a single life.” This variant points out to the fact that given the current number of vehicular fatalities, waiting for self-driving cars to be 99% (if not perfectly) safe disregards the fact that many of these accidents could be prevented once the fatality rate for self-driving vehicles merely dips below that of physically-manned vehicles, even if that is still a nonzero statistic.

Is waiting for perfection worth it?

This work is licensed under a Creative Commons Attribution-NoDerivs 3.0 United States License. Original source:

Thought Experiments Fri, 08 Jan 2016 13:01:14 +0000 Share

“Suppose there was a machine you could connect your brain to, which would simulate the perfect life – everything according to your wishes. You wouldn’t notice that it was all virtual. Surely you would connect yourself?”

“Never! That wouldn’t be real, what an awful thing! Your question make me shiver – I need to get get this off my mind… I’m going to play some World of Warcraft now.”

Interesting here is also the Reversal Test, which suggests that an intuitive rejection of the “Experience Machine” could be motivated by a status quo bias: Would you want to exit the machine, if you discovered that you have been connected to it all along up to now?

But let’s get back to the beginning: In ethics, as well as in philosophy in general, thought experiments have a firm place in the methodological tool box. Some thought experiments – such as Robert Nozick’s “Experience Machine” (the question above) or Philippa Foot’s “Trolley Problem” (see below) – are so famous that they are being discussed frequently at dinner tables outside of academic philosophy. However, often not all the participants in the ensuing discussion will be interpreting the intent of the thought experiment in the same way. The following paragraphs explain how we can use thought experiments to figure out what we value.

In contrast to hypotheses in the natural sciences, ethical theories cannot be tested empirically. Nevertheless, there are some useful approaches for evaluating ethical views. Besides formal criteria such as consistency (if someone claims A and non-A simultaneously, this person is in essence claiming nothing whatsoever), it is also appropriate to ask whether a position and the arguments given in support of it hold up to testing through thought experiments. The purpose of a thought experiment in the context of thinking about goals/values is usually to lead a specific intuition or assumption ad absurdum. For the criterion in question, a (decision-)situation is constructed in which the proponents of a particular position should not be willing to consistently apply their principles anymore. If this is the case, then the principle might not be a good one after all.

Thought experiments should, just like experiments in science, be free from confounding variables – otherwise it is unclear what it is that you are testing. The process of isolating all potentially confounding variables is the reason why the scenarios in thought experiments are often highly abstract or “far fetched”, not like any situations that we would expect to encounter in real life. How realistic the examples are is irrelevant as long as their content is interesting and as long as they are clear enough in terms of the allowed options and outcomes.

How to use thought experiments is best explained with an example. We will therefore analyze the most famous group of thought experiments in ethics, the so-called Trolley-Problem:

Standard Case: An out-of-control trolley is rolling down a slope towards five workers, who are trapped on the tracks. You find yourself standing next to the tracks in front of a lever, which, when you pull it, redirects the trolley to a sidetrack. On this sidetrack, there is one other trapped worker. Should you pull the lever, thereby causing one person to die instead of five?

Fat Man Case: Again, five workers are trapped on the track with the runaway trolley approaching. This time, there is no sidetrack. Instead, you find yourself on a footbridge across the tracks, where a fat man is standing at the edge, looking down. You realize that the man’s weight could stop the trolley. Because the fat man won’t jump by himself, the only way to prevent five other people from dying would be by pushing him onto the tracks in front of the trolley. Should you do that and thereby cause one person to die instead of five?

The first thing that stands out is that the hypothetical scenarios are pretty obscure. Especially the second case seems absurd because we’d expect runaway trolleys to be harder to stop. In order to make the thought experiment more realistic, one could of course make a number of adjustments. This can be done, but it would actually be missing the point: As long as we are very clear about what is specified to happen and what not, any thought experiment, even a highly unrealistic one, will do the trick. The guidelines for fruitful interaction with a thought experiment are the following:

1) Only the courses of action introduced in the thought experiment are possible/allowed. It is prohibited to propose further courses of action that would turn the example into a different one, because that likely wouldn’t be relevant for the principles being tested anymore.

2) Only the outcomes that are actually introduced in the thought experiment are the outcomes that can happen. It is prohibited to think up further future consequences or side-effects from some of the potential courses of action. After the situation in the thought experiment is over, there will be no more relevant consequences, and for all we are concerned, the hypothetical world of the thought experiment might end right there.

These strict guidelines are necessary for the proper isolation of the relevant variables. In the trolley example above, the issue being focused on is whether only the outcomes of an action are relevant (here: minimizing the number of deaths), or whether it makes a significant difference how the outcomes are being brought about.

For instance, some people might believe that using someone as a means to save others is always prohibited. In the Standard Case of the trolley problem, most people are willing to redirect the trolley so that only one person dies. In the Fat Man Case, however, the situation is reversed: Most people would not push the fat man from the footbridge. It thus seems that people believe that good outcomes are not everything that counts. What else is relevant? In the light of the two instances of the thought experiment, we can now try to come up with hypotheses that pin down and explain the exact nature of this difference, and then, we could try to design further thought experiments to confirm whether the suspected explanation, why it would be bad to push the fat man from the footbridge but not bad to cause the death of a worker on the sidetrack, also makes sense to us in other cases.

Someone strictly opposed to pushing the fat man off the bridge might claim that it is categorically wrong to instrumentalize people, to use them “merely as a means” for some other end (even if that end is well-intended). In order to test whether this principle is reasonable, we can design a further thought experiment:

The Loop Case: The situation is the same as in the Standard Case, except that the sidetrack leading to the single worker makes a loop, coming back again to the main track. The sidetrack joins the main track again slightly before the spot where the five workers are trapped. In addition, the worker on the sidetrack weighs a lot, so that his body would stop the trolley. Should you pull the lever, thereby causing one person to die instead of five?

One would think that this third scenario is identical to the Standard Case in all relevant respects. When people who have never encountered trolley problems are confronted with either the Standard Case or the Loop Case, the distribution of answers will likely be quite similar. Interestingly however, the Loop Case violates the principle that one should never treat humans as a mere means, which we used as a potential reason for treating the Fat Man Case differently from the Standard Case. If the fat man weren’t trapped on the sidetrack in the Loop Case, the trolley would kill the five workers regardless of whether we pull the lever or not. If we pull the lever, the trolley will enter the sidetrack for a brief distance, but without an obstacle to stop it, it would soon rejoin the main track. The Loop Case only differs from the Standard Case due to an extra piece of track, the one that links the sidetrack back to the main one. Is it really the case that a mere piece of track (which, incidentally, will never even get used) makes such a huge difference, causing five workers to die instead of one? Given that this conclusion seems absurd, this would prompt us to call into question the “categorical prohibition against instrumentalization”.

This discussion is far from comprehensive. In order to come to a clear answer, one would of course have to go through more thought experiments and considerations (see this paper).

Good thought experiments can be surprisingly fruitful. Fortunately, when two people have differing opinions on some ethical issue, it doesn’t necessarily mean that the discussion is over. The principles/values/goals that we hold dear are not always those that we would want to follow after having thought about the topic more thoroughly. In order to figure out when this is the case, we can use cleverly constructed thought experiments to push the edges of our moral beliefs.

Understanding the reasons behind using thought experiments, we should now understand that thought experiments can be relevant even if a situation is absurd and extremely unlikely to happen in reality. If a principle P gives us results that are obviously preposterous, whether the situation in question is real or hypothetical, it shows that the reasons for adhering to P are bad. And this in turn implies that in all the cases (real or imaginary) where P gives acceptable results, it is doing so for the wrong reasons. Consequently, P may be full of surprises when it comes to applying it to new (real or imaginary) cases – which can be very dangerous. What happens when we refuse to take hypothetical reasoning seriously is best illustrated with the following example:

In the excellent TED talk Why our IQ levels are higher than our grandparents’, James Flynn recounts a discussion on racism he had with his father, where he asked the old man: “How would you feel if tomorrow morning you woke up black?” To this, Flynn’s grandfather replied: “That is the stupidest thing you ever said. Who have you ever known who woke up in the morning that turned black?”

Artificial Free Will Fri, 09 Oct 2015 13:54:01 +0000 Share






You cannot avoid determining goals


Implicit and explicit goals


Implicit goals are not fundamentally different


Thinking things through



Genetic programming


Parallels to human enhancement





]]> 3 602
Is Utilitarianism Too Demanding? Thu, 13 Aug 2015 13:38:40 +0000 Share



There is an immense amount of suffering in the world:



Humans have finite willpower




Why utilitarianism is not excessively demanding








Practical arguments are sufficient




Praise and blame are instrumental







]]> 5 585
Bayes’ Theorem Thu, 18 Jun 2015 12:28:32 +0000 Share


Post last updated: 30.06.2015. Thanks to EP and others for pointing out that the negation signs were not always displayed properly in the formulae.

If some terrible catastrophe destroyed human civilization and humanity lost all scientific knowledge it ever collected – which single piece of information would you pass on to the survivors? Richard Feynman’s answer is the following:


Luke Muelhauser deems this an excellent choice, especially since it takes us a long way to  reductionism. He suggests Bayes’ Theorem as the second piece of information he would choose. As he writes: “Seeing the world through the lens of Bayes’ Theorem is like seeing The Matrix. Nothing is the same after you have seen Bayes.”

Bayes’ Theorem tells us how to rationally assess the probability of a certain statement of interest being true, given some evidence. This could be the probability that a patient has a certain disease, the probability that a startup will be successful, or the probability that your opponent at the poker table has you beat. Insofar as science consists in creating hypotheses, collecting evidence for and against them, and updating our credence in these hypotheses in the face of the collected evidence, Bayes’ Theorem formalizes the very process of doing science.

As humans, we are all already more or less good at applying Bayes’ Theorem unconsciously. Imagine that while waiting in a train station, you notice a bag which has been left unattended for one hour. Could it be a bomb? “I guess people forget their bags on train stations much more often than they try to detonate bombs”, you reason. Cautiously, you move a bit closer to the bag and notice that there is something ticking inside the bag. What now? Are you getting nervous, or is a bomb still too unlikely? How should you rationally assess the probability that the bag actually contains a bomb? Which numbers would you need to know (and how many of them) in order to make a confident assessment? What would you actually do with these numbers?

It has been shown repeatedly that even well-trained people systematically deviate from correct Bayesian reasoning when confronted with examples such as the one given before. By definition, such a systematic deviation constitutes a cognitive bias. Consider the most well-known example:

Only 1% of women at age forty who participate in a routine mammography test have breast cancer. 80% of women who have breast cancer will get positive mammographies, but 9.6% of women who don’t have breast cancer will also get positive mammographies. A woman of this age had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

Can you find the correct answer? In case you can’t, make an estimate. Only 15% of all doctors are able to give the correct answer, and people who aren’t usually give answers above 70%, which is way too high. The correct answer is 7.8%. It is much easier to derive this result if the problem is phrased in terms of natural frequencies instead of percentages:

100 out of 10,000 women at age forty who participate in routine screening have breast cancer. 80 of every 100 women with breast cancer will get a positive mammography. 950 out of 9,900 women without breast cancer will also get a positive mammography. If 10,000 women in this age group undergo a routine screening, about what fraction of these women with positive mammographies will actually have breast cancer?

If the problem is phrased like this, more people are able to find the correct solution. It may also be more evident this way that we really need all three pieces of information to arrive at the solution: the fraction of women with breast cancer; the fraction of women with breast cancer who get a positive mammography; and the fraction of women without breast cancer who get a positive mammography.

People who give answers much higher than 7.8% usually neglect the first piece of information – the chance that a given woman has breast cancer if we don’t know the test result. For example, some people subtract 9.6% from 80%, which is completely wrong. It is important to realize that learning the test result does not simply replace our prior knowledge (in this example, that only 1% of women in the relevant demographic have breast cancer). It merely allows us to update our initial credence by a certain factor. Ignoring the prior probability and only relying on the evidence at hand is known in the psychological literature as base rate neglect.

A special case of base rate neglect is the prosecutor’s fallacy, with Sally Clark probably being the most prominent victim. As an example, suppose a robber was wearing a certain tattoo which only one person in 10,000 is wearing, and then a person wearing that tattoo is found. This does not imply that the chance of the suspect being innocent is 1 in 10,000, since the probability of any given person having been the robber is very small. Again, the correctly updated probability of the suspect being guilty needs to weight in both the prior probability of the suspect being guilty and the strength of the evidence provided by the tattoo.

The prior probability and the strength of the evidence correspond exactly to the two factors in Bayes’ Theorem. Before we can state it in a formal way, we need to introduce conditional probabilities.

Formal Matters

For an arbitrary event A (e.g., A = “it will rain tomorrow”) we denote its probability by p(A). Given two events A and B, we denote the probability of both of them happening by p(A\&B). Finally, we denote by p(A|B) the probability of A given B. For example, if A = “there is a bomb in this bag” and B = “there is something ticking in this bag”, then p(A|B) denotes the probability that there is a bomb in the bag if we’ve noticed that there’s something ticking inside. Conditional probabilities thus allow us to express how the probability of a hypothesis changes in the face of new evidence. In the above example, p(A) would denote the probability of a bomb in the bag before we noticed the ticking, and presumably we have p(A|B)>p(A).

The three quantities p(A), p(A\&B), and p(A|B) are connected through the relation

(1)   \begin{equation*}  p(A\&B) = p(A) \cdot p(B|A) \end{equation*}

In words, the probability that A and B both happen is the probability that A happens times the probability that B happens, given that A has happened. We can also condition on B and write

(2)   \begin{equation*}  p(A\&B) = p(B) \cdot p(A|B) \end{equation*}

Combining these two equations, we find p(A) \cdot p(B|A) = p(B) \cdot p(A|B). After dividing by p(A) and relabeling A and B to E and H, respectively, we finally arrive at

(3)   \begin{equation*}  p(H|E) = \frac{p(H) \cdot p(E|H)}{p(E)} \end{equation*}

This is Bayes’ Theorem, which tells us how calculate the conditional probability p(H|E), where H stands for hypothesis and E stands for evidence. While from a formal point of view, Bayes’ Theorem follows trivially from the definition of conditional probabilities, it is an entirely different thing to internalize it and being able to correctly apply it in real-world problems.

The left-hand-side, p(H|E) tells us the probability of a hypothesis H after updating this probability for the evidence E. The first factor on the right-hand-side, p(H), is the prior probability, i.e., our credence in the hypothesis H prior to the evidence E. The denominator, p(E) is the probability of the evidence we are given, while p(E|H) is the probability of this evidence given our hypothesis.

In many practical examples, the denominator p(E) is not known directly but needs to be calculated first. In order to see how this can be done, we need one further bit of notation. For some event A, we denote by \neg A its negation. Correspondingly, p(\neg A) denotes the probability that A does not happen, and we have the evident relation p(A)+p(\neg A)=1 (either A happens or it doesn’t happen – there’s no third possibility). As a final requisite, we’ll need the relation

(4)   \begin{equation*}  p(B) = p(B\&A) + p(B\&\neg A) = p(A) \cdot p(B|A) + p(\neg A) \cdot p(B|\neg A) \end{equation*}

If we apply this to the denominator p(E) in Bayes’ Theorem, we arrive at its most practically useful form

(5)   \begin{equation*}  p(H|E) = \frac{p(H) \cdot p(E|H)}{p(H) \cdot p(E|H) + p(\neg H) \cdot p(E|\neg H)} \end{equation*}

Let us see how this applies to the breast cancer problem. In this case, H denotes the hypothesis that the woman does have breast cancer and E denotes a positive mammography. The prior probability of breast cancer is given by p(H)=1\%, and consequently p(\neg H)=99\%. If a woman does have breast cancer, the probability of a positive mammography is p(E|H)=80\%, while if she doesn’t have cancer, it is p(E|\neg H)=9.6\%. Inserting all of these quantities in the above form of Bayes’ Theorem, we arrive at

(6)   \begin{equation*}  p(H|E) = \frac{0.01 \cdot 0.8}{0.01 \cdot 0.8 + 0.99 \cdot 0.096} = 0.078 \end{equation*}

Can you apply Bayes’ Theorem to the following problem?

Suppose that a barrel contains many small plastic eggs. Some eggs are painted red and some are painted blue. 40% of the eggs in the bin contain pearls, and 60% contain nothing. 30% of eggs containing pearls are painted blue, and 10% of eggs containing nothing are painted blue. What is the probability that a blue egg contains a pearl? [1]

Besides giving a precise, formal meaning to commonsensical wisdoms such as “never be absolutely certain” or “update your credences when you see new evidence”, there are some less trivial lessons to be learned from Bayes’ Theorem. People often introduce a dichotomy between proofs and evidence that can be ignored. However, this distinction is a useful heuristic at best. Ultimately, all our beliefs should be probabilistic and ideally take into account all available evidence, be it weak or strong. An example is the saying “correlation does not imply causation”. Actually, observing an unexpected correlation between variables A and B does increase the probability that one of them causally influences the other, even if alternatives – such as a third variable C causally influencing both A and B or the data having been manipulated – may have considerable weight, too.

Similarly, the fact that people believed in Zeus’ existence is evidence that Zeus actually exists. If Zeus did exist, this would make it likelier that there are people who believe in him; therefore, people believing in Zeus is evidence for his existence. This statement can in fact be formally derived. The equation

(7)   \begin{equation*}  p(E) \cdot p(H|E) = p(H) \cdot p(E|H) \end{equation*}

can be rearranged to

(8)   \begin{equation*}  \frac{p(H|E)}{p(H)} = \frac{p(E|H)}{p(E)} \end{equation*}

Therefore, p(H|E) > p(H) if and only if p(E|H) > p(E). In words, evidence E increases the probability of hypothesis H if and only if hypothesis H makes evidence E more likely than we would otherwise expect. The prior probability of Zeus’ existence should be so low, however, that even after updating for the fact that myths about Zeus exist, the probability of his existence is still negligible for all practical purposes. (Why should this prior be so low? This will be discussed shortly.)

Similarly, and despite a more well-known aphorism stating the contrary, absence of evidence is evidence of absence. It follows from

(9)   \begin{equation*}  p(H) = p(H\& E) + p(H\&\neg E) = p(H|E) \cdot p(E) + p(H|\neg E) \cdot p(\neg E) \end{equation*}

and p(H) = p(H) \cdot p(E) + p(H) \cdot p(\neg E) that

(10)   \begin{equation*}  0 = \left[p(H|E) - p(H)\right] \cdot p(E) + \left[p(H|\neg E) - p(H)\right] \cdot p(\neg E) \end{equation*}

So p(H|E) > p(H) if and only if p(H|\neg E) < p(H); if some evidence E would increase our credence in hypothesis H, then not observing evidence E has to decrease our credence in H. The fact that centuries of science have not collected any evidence that would allow us to increase our credence in Zeus’ existence should further decrease our credence in his existence.

More generally, any expected increase [p(H|E) - p(H)] \cdot p(E) in our credence for some hypothesis H needs to be counterbalanced by an equally-sized expected decrease [p(H|\neg E) - p(H)] \cdot p(\neg E). We cannot rationally have an expectation value for our future credence that differs from our present credence. If we expected to upshift a hypothesis stronger than we downshift it, this should already affect our current credence.

Moreover, since p(E)+p(\neg E)=1, one of the two summands p(E) and p(\neg E) being small implies the other one being large (i.e., close to 1), and vice versa. Therefore, if we expect a high probability of weak evidence into one direction, this needs to be balanced by a low probability for strong evidence in the other direction. Eliezer Yudkowsky calls this principle the Conservation of Expected Evidence. The breast cancer case discussed in the introduction is an excellent example: We expect with high probability a negative result, which leads to a weak decrease of our credence in the woman having cancer (it was already as low as 1%); this is balanced by the rare event of a positive result, which is strong evidence for the woman having cancer.

Science and Priors

As stated in the introduction, Bayes’ Theorem formalizes the very act of doing science. For example, it gives a precise explanation for many ideas developed by Karl Popper, which are collectively known as falsificationism. At the core of falsificationism is the idea that theories can be definitely falsified, but never definitely confirmed. No matter how many white swans you observe, you can never definitely confirm that “all swans are white”; by contrast, observing a single black swan falsifies it. Formally, this directly follows from Bayes’ Theorem. If your theory T definitely forbids a certain observation Op(O|T) = 0 – then observing O definitely disconfirms your theory – p(T|O) = 0. By contrast, given a finite set of observations O it is impossible to update the credence in theory T to p(T|O)=1 if our prior probability (the probability we had before conditioning on evidence) was p(T)<1. Bayes’ Theorem tells us that we only could have p(T|O)=1 if p(O|\neg T)=0, i.e., if there were no theory other than T that could predict O. This is indeed something that we can never know – we can’t possibly exclude all theories other than T. Newtonian mechanics was incredibly well-confirmed and nevertheless it turned out that it describes only a certain limit of Einstein’s more general theory of relativity.

Recall the Conservation of Expected Evidence rule, which states that an expected increase in the credence of a theory needs to be balanced by an equally sized expected decrease. This explains why it is so crucial that a theory be falsifiable. If it is impossible to decrease the credence in a theory, it is also mathematically impossible to increase it – and vice versa, which is where Popper was wrong. It is possible to probabilistically confirm a theory and thereby increase its likelihood. Confirmation and falsification are not fundamentally different, as Popper argued, but both just special cases of Bayes’ Theorem.

Finally, let us turn to the question of where priors, the value of the prior probabilities, come from, and how we should rationally choose them. Firstly, let us note that the choice of priors is not that important if we are given a sufficiently large set of data. The more updates of our probabilities we perform, the more they become independent of the priors. In practice, what prior to choose is often self-evident if the case we are interested in can be viewed as a random element of a larger group about whose statistics we have some information. For example, in the breast cancer case we can view the woman as picked at random from all the women in her demographic, and can then use the (known) fraction of women with breast cancer in that demographic as the prior probability of her having cancer. If we have more information on the woman, e.g. that she is 43 years old, we can use the cancer rate among 43-year-old women as a more precise prior, should we have this information available.

Even if we have no previous information, symmetries sometimes suggest a reasonable prior. Imagine having six bowling balls in front of you that are indistinguishable by eye. It is then natural to assign each ball a prior of 1/6 of being the heaviest, since there is simply no reason for assigning a higher probability to one ball than to another.

However, in many real-world problems, there is no obvious choice for coming up with a prior probability. What is the prior probability of Zeus existing, or of the universe being finite in size? In the 1960s, Ray Solomonoff developed what is known as the universal prior. The universal prior combines and formalizes ideas of philosophers Epicurus (341–270 BC) and William of Ockham (1287 – 1347). Epicurus’ principle states that one should keep all hypotheses that are consistent with the data. Ockham’s principle, widely known as Occam’s Razor, states that entities should not be multiplied beyond necessity. This is widely understood to mean: Among all hypotheses consistent with the observations, choose the simplest.

Solomonoff’s universal prior unifies these two principles. It gives all hypotheses consistent with the data a non-zero probability, but highest probability to the simplest ones. The technicalities necessary for a precise definition of how “simplest” is to be understood here are well beyond this text. However, we should stress that “simple” does not mean “easy to understand for a human”. Rather, complexity should roughly be thought of as the length of a computer program that is able to reproduce your hypothesis. The universal prior is exponentially small in this length and thus strongly penalizes hypotheses which involve unnecessary complications.

As an example, the explanation that Thor’s wrath is responsible for thunderstorms may sound simple enough to humans, and definitely simpler than atmospheric physics. However, a computer program that simulated Thor throwing lightning bolts needed not only to simulate the lightning bolts themselves, but Thor as well. Viewed from this perspective, Thor appears as an unnecessary complication, which does not in fact have any explanatory power. By contrast, the Standard Model of particle physics explains much, much more than thunderstorms, and its rules could be written down in a few pages of programming code.

We close with a historical example which shows that the choice of priors can sometimes be decisive if observations do not allow to exclude one of two rivaling theories. Both the (geocentric) Ptolemaic and the (heliocentric) Copernican model of the cosmos were able to explain all observations that Copernicus’ contemporaries could make. This means that the ratio of the credences in the two theories could not be changed through observations; it was determined by the priors alone. A prior that punishes ad hoc adjustments (which “increase the length of the program”), such as the proverbial epicycles in the Ptolemaic system, would lead to a strong preference for the Copernican model. Preferring a heliocentric model was thus the rational thing to do for Copernicus, even though he was not able to refute the geocentric model on empirical grounds.


1. The answer is 2/3.↩


Parts of this posting is based on Luke Muelhauser’s An Intuitive Explanation of Eliezer Yudkowsky’s Intuitive Explanation of Bayes’ Theorem

]]> 10 552
The Problem of Evil III Tue, 12 May 2015 13:50:22 +0000 Share

“Why have you abandoned us so completely? […] Have you just remained silent like the darkness that surrounds me? Why? At least tell me why.” – Silence, Shusaku Endo


The argument from evil is perhaps the strongest and most compelling argument against God’s existence. It is part of the debate on the problem of evil, the attempt of reconciling God’s existence with that of tremendous suffering and evil. The argument from evil is intended to show that this can’t be done and, since we observe tremendous suffering and evil, God does not exist. Depending on whether the argument involves inductive or deductive reasoning we can distinguish between the evidential and the logical argument from evil.

In a recent blog-post the philosopher Yujin Nagasawa has suggested that this is only the starting point for a new group of arguments from evil. Instead of asking whether the existence of tremendous suffering and evil can be reconciled with God’s existence, we can ask whether the existence of the unsolved first-order problem of evil and puzzlement among theists concerning God’s reason for allowing suffering and evil can be reconciled with God’s existence. Behind this question lies the intuition that God would let us know why he allowed so much evil if he existed and had good reasons for allowing it. Not doing so might cause unnecessary suffering, doubt, and uncertainty among believers.

The Second-Order Arguments from Evil

Let us see whether we can expand on the rough sketch of the higher-order problems of evil laid out in the last section and turn it into proper arguments. Here is the basic idea: Philosophers have turned the problem of evil into an argument from evil by providing reasons to believe that the existence of (apparently) gratuitous suffering and evil cannot be reconciled with God’s existence. Since we observe such suffering and evil – this is the empirical premise – we conclude that God probably does not exist. Now here is the basic thought: Could we not also turn higher-order problem of evil by the same token into a higher-order argument from evil? Let’s give it a try.

Suppose, for the sake of the argument, that there is some good reason why God allows all this apparently gratuitous suffering. Let us call this reason X. We do not know X, for there is no completely convincing theodicy explaining why God allows all this suffering. But even if we knew X now (perhaps because some philosopher found X in the process of writing his dissertation on the problem of evil), most humans which have existed would not have known X. Let us call this empirical fact gratuitous puzzlement. It will serve as our empirical premiss in the second-order argument from evil.

We can now ask: Is gratuitous puzzlement compatible with God’s existence? it seems that a case can be made that it is not. Furthermore, this case works exclusively with premises that the theist is likely to accept.

Here is how: It is an empirical observation that the problem of evil is one of the most compelling arguments against God’s existence. It is probably the argument which has been most effective in converting theists into atheists or agnostics (although I must admit that I do not have any empirical data beyond my own observations and other people’s anecdotes). This does not seem surprising, both the deductive and the inductive version are closely modeled after arguments widely employed in the natural sciences. The dialectic strength of the argument is a problem: If there is some good to be found in believing in God – a view shared by most theists – then it is bad if very strong arguments in favor of atheism remain unanswered by theists for most of human history. Some theists even believe that such a lack of theistic explanation of evil will lead humans, namely those who endorse atheism after careful and honest contemplation on the argument from evil, to spend eternity in hell. And this is surely something God would have an interest to prevent.

But even if a person doesn’t become an atheist due to the argument from evil God might have reasons to prevent his or her gratuitous puzzlement. The existence of such puzzlement will cause an inner tension or cognitive dissonance, doubt, emotional suffering, and could lead her to adopt bad epistemological practices which have spillover effects to other areas of inquiries and thereby cause further problems and potentially even suffering.

In other words: God has an interest to prevent gratuitous puzzlement. And since he is omnipotent he could prevent gratuitous puzzlement. He could presumably do so without revealing his existence, if that is for some reason a problem, by inspiring a clever specimen of the early human race to discover the right theodicy. Hence it is plausible that if God exists, gratuitous puzzlement does not.

Now we can construct the deductive second-order argument from evil:

  1. There is gratuitous puzzlement.
  2. If God exists (and he is benevolent omniscient, and omnipotent) there is no gratuitous puzzlement.
  3. God does not exist.

But one could also construct a probabilistic or Bayesian second-order argument from evil:

  1. P(E)=1
  2. P(H)=0.5
  3. P(E|H)= 0.2
  4. P(H|E)=P(E|H) x P(H)/P(E)=0.1

Where P(E) stands for the prior probability that there is gratuitous puzzlement, P(E|H) for the conditional probability that there is gratuitous puzzlement given the hypothesis that God exists, and P(H) the prior probability of God’s existence. (This is only a rough sketch of the inductive argument and much more needs to be said to justify the probabilities in the argument. For a bit more details on Bayesian arguments from evil see The Problem of Evil II.)

Moving Up the Ladder: Higher-Order Arguments

But the project does not stop here. Suppose there is a solution to the second-order problem of evil and there is a reason why God does not tell us X, his reason for allowing suffering and evil. Let us call this reason Y. We can now ask: If God is omnipotent and omnibenevolent, why does He not tell us that Y is the reason that he cannot tell us that X is the reason that he has to allow evil? This is the third-order problem of evil, and we may construct deductive and inductive third-order arguments from evil. And so on.


Do these higher-order arguments from evil bring anything new to the table? Do they strengthen the case against God’s existence over and above the ordinary argument from evil? I think they do, for the simple reason that the conjunction

  1. God has reasons for allowing first-order evils and suffering (suffering not resulting from gratuitous puzzlement); God has reasons for allowing second-order and higher-order evils and suffering (suffering resulting from gratuitous puzzlement)

is less probable than

  1. God has reasons for allowing first-order evils and suffering.

This is so because there are conceivable cases where 2 is true without 1 being true but not the other way around. This also shows that theists are not necessarily done with the argument from evil if they can come with a good theodicy. Some theodicies might only be first-order theodicies and show that 2 holds, while doing nothing to show that 1 holds. Hence higher-order arguments from evil raise the bar for a successful theodicy.


I’m grateful to Yujin Nagasawa for feedback on a draft of this post. He also pointed me to the passage in Shusaku Endo’s Silence which inspired his original blog-post.


Nagasawa, Y. (2014). Higher-Order Problems of Evil blog-post on Prosblogion. See also the various comments.

]]> 12 544
Means and Ends Thu, 16 Apr 2015 15:01:05 +0000 Share

Ends and Means

What are your goals in life, what is it that you care about? When people are asked what they value, answers will include things like happiness, freedom, fairness, friendship, biodiversity, truth, pizza, equality, medicine, loyalty, and many other things.

In theory, it may be that someone values all of these directly. However, for most of the examples it seems more probable that people would only value them indirectly, as a means to some other end.

It’s possible to be mistaken about one’s own values. A common instance of it is when we think we care about something, while in fact what we truly (i.e. under reflection) care about is something else, something that merely happens to correlate in most typical situations with the thing that we would care about in all situations.

This is the difference between means and ends, or between instrumental values and terminal values. Valuing x only instrumentally means that there are logically possible situations where an increase in x is, all else being equal, not something we care about. On the other hand, if we value something terminally/intrinsically, an increase of it is, all else being equal, always something we care about.

The “all else being equal” is needed because in some situations, the things which we care about might come with undesired side-effects.

Thought experiments

In order to figure out which values are terminal and which instrumental, we can design thought experiments to isolate any potential values.

Let’s take the example of medicine: If someone cared terminally about medicine, it would always be important to her that there is medicine. However, imagine a world where no one ever gets sick, would medicine be important in this world? Probably not, so medicine is merely instrumentally valuable to us (in this case: instrumental for health or well-being).

A more complicated example is biodiversity. Many people seem to want there to be many different species. Why do they want this? Maybe people only value it instrumentally, because biodiversity correlates with many other things one might find valuable:

  • Biodiversity may often serve human interests.
  • Biodiversity is usually important for the stability of ecosystems.
  • Biodiversity is a source of scientific knowledge.
  • Decreasing biodiversity could mean that more animals die than otherwise.

The important question is whether biodiversity is valuable all else being equal, i.e. if no other plausible variable that might qualify as an end is being affected. In order to properly imagine such a situation, we can design a thought experiment where the indirect effects of biodiversity do not apply as they might apply in more usual, real-life circumstances:

Rare species: Imagine a rare rodent species that is close to extinction. The animal serves no important role in the ecosystem. If it went extinct, the ecosystem would not be perturbed in any way. Further, the animal is completely boring: no one cares about whether it goes extinct or not, and future scientists would never learn anything interesting from the animal even if they proceeded to study it extensively for hundreds of years. Finally, the situation happens to be such that if the animal does not go extinct, the amount of suffering and the number of deaths of sentient individuals remains the same.

In such a scenario, would it be bad if that particular rodent species went extinct? Why?

The thought experiment was designed to separate the variable “biodiversity” from potentially confounding variables such as “human interests”, “ecosystem stability”, “well-being of individuals” and “potential source of knowledge”. If we don’t at all care about biodiversity in this example anymore, then it was only ever an instrumental value for us, not a terminal one, and being aware of this difference can be very important.

As a final example, consider someone who claims that pain is not bad because it serves an important function: protecting the body from injury. People with CIPA, a rare genetic disorder, are incapable of experiencing physical pain. While this may sound great at first, these people unfortunately have a significantly lower than average life expectancy because they often fail to notice when they are ill or injured.

Still, it would of course be silly to conclude that just because pain sometimes serves an important function, we should therefore not try to make people pain-free. If there are ways to keep one’s body working that don’t involve agony (the “all else being equal” part!), then it would certainly be preferable to achieve this without the pain. In that sense, pain is always bad, even if there are situations where it comes with useful side-effects.


Something important to keep in mind is that having more than one terminal value implies that there will have to be tradeoffs. In a decision-situation where two of our terminal values conflict with each other (i.e. where it’s impossible to maximally fulfill both), we need a decision-procedure that determines which quantity of the first value could be traded against some quantity of the second one. Imagine that I value both happiness and equality. Making society more equal usually also leads to increased overall happiness (because of diminishing marginal utility). But this is not always the case. What would I do in a situation where equality could be increased greatly at the cost of making a few people unhappy? If I value equality to a significant extent, then there would be circumstances where I would trade happiness for equality. This is what it means to “value” or to “care for” something: that we would be willing to invest precious time and resources into it.

When we envision that there will have to be such tradeoffs, we may become pickier with declaring things to be our terminal values. For instance, most people consider knowledge to be a good thing in itself, but would they introduce significant suffering into a world where everyone is perfectly happy (and would remain so indefinitely), if it was the only way to generate more knowledge? What is the use of knowledge if all else stays equal, if it doesn’t generate any happiness or doesn’t satisfy any interests? These are the questions we need to ask ourselves when thinking about our goals and values.

In real-life situations, ends and means are often hard to separate. Sometimes particular means almost always correlate with particular ends. In some such cases, if the end is hard to measure directly, it could even be useful to focus primarily on the means as a heuristic. For instance, in order to make people happier on average – which is something we might not immediately know how to measure directly – we could simply focus on trying to make society more egalitarian, a measure that will likely correlate significantly with our actual aim. The important point to keep in mind is that we should immediately abandon this endeavor if we ever discover that the correlation does not work in a specific case!


Because of habit forming and presumably also the way our values evolved, we are sometimes tempted to mistake instrumental values for terminal values. This is unfortunate because having more (stated) values means that there will more tradeoffs, which means that our actual terminal values, the ones that we’d keep under reflection, end up less satisfied in expectation.

In order to properly distinguish instrumental values from terminal values, we can design thought experiments where common, real-life correlations are broken up while all the other factors remain constant. If we then decide that we don’t care about a particular thing after all, then it was only ever an instrumental value. By distinguishing instrumental values from terminal ones, we gain clarity about our own goals and will therefore be able to form better plans to achieve them.

Chronicles of the Zombie Wars II Fri, 03 Apr 2015 14:04:10 +0000 Share


In the first post of the zombie series, we’ve seen that the mere possibility of zombies is enough to refute physicalism about the mind. The anti-physicalist, however, cannot simply start with the possibility of zombies as a premise to refute physicalism. As we have seen, one way to formulate physicalism is to say that every world which is physically identical to ours necessarily also contains the same mental states (and perhaps some additional ones). This entails that zombies are impossible. This means that starting with the possibility of zombies amounts to begging the question against the physicalist (or, at any rate, comes very close to it.)

How can we assess whether zombies are possible on impartial grounds? How do we generally know whether statements about how things might be or could have been are true? How do we know that married bachelors are impossible, whereas extraterrestrial bachelors are possible? Knowledge about which things are necessary, contingent, and impossible, is called modal knowledge, and the question how we can acquire modal knowledge belongs in the domain of modal epistemology, a flourishing branch of theoretical philosophy.

Approaching the Question: Teaching an AI Modal Reasoning

Modal epistemology is not just intellectual gymnastics for philosophers interested in zombies and the philosophy of mind. The AI researcher Terry Winograd was made aware of this when he wrote his famous natural language processing program SHRDLU at MIT in 1968–1970. SHRDLU (the name is inspired by the letter sequence “ETAOIN SHRDLU”, the most frequently used letters in the English language) is a program capable of taking natural language input in a limited vocabulary and responding to it in an intelligent way. Its input can either be an inquiry or a command regarding a block world, a simplified simulated world in which SHRDLU is situated. A block world contains basic elements such as blocks and pyramids of various colors.

To get a feeling for the capabilities of SHRDLU, consider the following dialogue between a user and SHDRLU, with Winograd’s comments in brackets:


1. pick up a big red block.


(The system answers “OK” when it carries out a command.In order to pick up the red block, it had to clear it off by finding a space for the green one and moving the green one away.)

2. grasp the pyramid.


(There are three pyramids in the scene, so the program does nothing.)

3. find a block which is taller than the one you are holding and put it into the box.

(The system has to figure out what is meant by “one” and “it”, as well as use deduction to decide which block it is holding.)


(Printed out to keep us informed.)


4. what does the box contain?

THE BLUE PYRAMID AND THE BLUE BLOCK.” Kobes 1990, p. 432 ff.

In this bit of dialogue SHRDLU answered an inquiry concerning the content of one box and followed a number of commands to manipulate the box world. In another piece of dialogue SHRDLU is asked a modal question, and what happens is interesting:

10. can a pyramid be supported by a block?


(The deductive system finds an actual example, so it knows this is


11. can a pyramid support a pyramid?


(The question makes sense, but there is no example in the scene, so there is no easy way to tell without being able to examine the robot’s programs.” Kobes 1990, p. 435.

This extract contains two modal questions as indicated by the word “can”. Another way to phrase the question would be “is it possible that a block supports a pyramid” which would reveal the modal nature of the question more clearly. SHRDLU answers the first one in the affirmative because there is an actual example of a block supporting a pyramid in the block world. This is an application of a theorem of many modal logics, namely that if P is the case, then P is possible. This points to one plausible principle we may use to evaluate possibility-claims: if something is actually the case, we can infer that it is possible.

The second question points to a trickier part of modal epistemology, and SHRDLU doesn’t handle it well. How can we know whether something is possible, if it isn’t actually the case? How can I know whether a mountain made of solid gold is possible, given that there is no such mountain? SHRDLU displays some modesty and refrains from judgement. (Winograd makes a somewhat cryptic comment about the necessity of SHRDLU being able to analyze its own code to answer the question, but that doesn’t help us much.) How could we modify SHRDLU in order to give it the capability to answer the question? How can we teach modal reasoning to an AI? It turns out that modal epistemology is relevant beyond academic philosophy: to relieve SHRDLU from it’s “modal impairment” (or build a new “modally capable” AI) we need to understand how we can obtain modal knowledge.

Why Modal Knowledge is Important

Modal epistemology is not just important for AI researchers. Modal knowledge is valuable in numerous domains, and having the right modal epistemology is crucial to obtain that knowledge.

When trying to explain a phenomenon P we usually come up with various hypotheses which could serve as explanations. We then evaluate these hypotheses, either based on a loose battery of criteria for theory evaluation, among them simplicity, the probability with which they predict the observed phenomena, fit with background knowledge, etc., or according to some more thoroughly regimented evaluation procedure (e.g. some form of Bayesianism).  But in order to demarcate the space of hypotheses we must know which ones are possible– impossible hypotheses are no good, for they are necessarily false. We can illustrate this point with an example. Suppose I’ve observed my neighbour leaving his house in his car in the morning, and the car is blue. In the evening I see him pull into the driveway, but this time, to my surprise, the car is red. One hypothesis to explain my observation is that he got a paint job for his car during the day. Another one is that his car is both red and not-red. The second hypothesis explains the data just as well as the first, for I can deduce the data from both hypotheses. (The elimination rules for conjunction allow me to deduce a conjunct from the conjunction, and so I can deduce that the car is not-red from the statement “the car is red & the car is not-red”). But this doesn’t mean we should take the latter hypothesis seriously. It is clearly an impossible hypothesis and should be excluded from the space of hypotheses since it is necessarily false. This example shows that modal judgements are crucial to determine the space of hypotheses we should consider when explaining some phenomenon.

Modal knowledge is also crucial for a special type of explanations, namely reductive explanations. When we reductively explain some phenomenon we show that the phenomenon just is some other thing(s), or that a certain phenomenon is nothing over and above something else. We can, for example, reductively explain digestion as a certain chemical process which involves a breakdown of large insoluble food molecules into small water-soluble food molecules. Digestion just is that chemical process, it is nothing over and above that process. What does this have to do with modal knowledge?

Suppose I want to explain to someone what a mass marine rush in Star Craft is. I describe the whole procedure of building a bunch of marines and attacking the opponent early in the game with those marines. It might not be obvious, but I have thereby given a successful reductive explanation of a mass marine rush in terms of more basic entities, namely individual marines, and their behavior.


What makes my explanation a successful reductive explanation? One necessary (although presumably not sufficient) condition is this: all the marine facts I’ve mentioned necessitate the mass-marine-rush facts. In other words: it is necessary that  “if A then B” is true, where A are all the things I’ve said about individual marines and B is the statement that a mass marine rush takes place. Or even shorter: it is impossible that all the facts about individual marines obtain without there being a mass marine rush. The point of this toy example generalizes. For any reduction of B facts on A facts, the reduction is only successful if the conditional “if A then B” is necessarily true, i.e. if it is impossible that the A facts obtain without the B facts. It turns out that offering a reductive explanation of a phenomenon involves judgements about necessity and impossibility.

The upshot of those two cases is that modal epistemology is crucial for our explanatory practices. First, we need modal knowledge to demarcate the space of possible explanations. Second, whenever we want to reductively explain a phenomenon we need to know whether it is impossible that the reduction base obtains without the phenomenon itself obtaining. Only if this is the case we might have a (correct or incorrect) reductive explanation on our hands.


The cases in the last section give us a first impression of what is at stake when we talk about the possibility of zombies and modal epistemology. Modal epistemology and the zombie argument are not isolated pieces of philosophy, they are intimately connected to basic models of scientific explanation. Whatever position we choose with regard to modal epistemology and the zombie argument, e.g. whether conceivability is a good guide to possibility, it has important ramifications for scientific explanation. This means we should tread carefully and avoid rushed judgments. In the next post of our post series on zombies we will explore the (arguably) most promising suggestion for how we can get our hands on modal knowledge: conceivability.


Garson, J. (2014). Modal Logic. The Stanford Encyclopedia of Philosophy (Summer 2014 Edition), Edward N. Zalta (ed.).

Kobes, B. W. (1990). Individualism and artificial intelligence. Philosophical Perspectives, 429-459.

Vaidya, A. (2011). The Epistemology of Modality. The Stanford Encyclopedia of Philosophy (Winter 2011 Edition), Edward N. Zalta (ed.).

Van Inwagen, P. (1998). Modal Epistemology. Philosophical Studies, 67-84.

]]> 3 521
Expected Utility Wed, 25 Mar 2015 16:40:04 +0000 Share

It’s in the interest of agents to achieve their own goals as well as possible. When we implement this in our behavior, we are acting rationally. But what does this mean in an applied setting, acting so as to best achieve our goals?

When we are absolutely certain about the consequences of our actions, the case is clear. We should choose those options which lead to the outcome we value the most. If my goal is to earn as much money as possible and I have a choice to receive either $10 or $100, the choice is clear: I should pick the $100.

However, the world, and especially also our senses, are not structured in a such a way that we can predict the consequences of what we do with certainty. For example, if I’m considering whether or not to buy a lottery ticket, most likely I’ll simply lose the money I paid. But there is a small probability that I will win more than the cost of the ticket. I can calculate these probabilities based on the payout structure.

In other cases, we have no such clear information to calculate the chances of winning or losing. Nevertheless, we still always need to place probabilities on possible outcomes by using our best judgment and all the information and experience that we have. Once we have a clear picture of the probabilities involved, we do best to choose the action that maximizes our expected utility.

What is expected utility?

“Utility” refers to how much we value a particular outcome. This depends on our goals. For instance, if someone’s goal is to get as much money as possible, then the utility of an outcome would simply be the monetary value the person could bring into their possession. People’s real goals are of course much more complex, and it would be difficult to put exact numbers on e.g. how much people value friendship, but nevertheless, people prefer some outcomes to other outcomes, which is what the concept of utility quantifies. If an option gives me twice as much utility as another option, then it means that I, when I’m fully informed and unbiased, would find it twice as good.

The expected utility of an action is composed of the sum of the utilities of all possible outcomes, with each outcome being weighted according to the probability of its occurrence. The sum of all probabilities must equal 1. The formula is as follows:
EU stands for the expected utility,
p stands for the probability that a particular outcome will occur,
and U is the utility of said outcomes, if they occur.

To use an easy example, we will calculate the expected monetary value (EV) for a game of dice. You are offered a bet where you can either

a) receive $10 right now, or
b) you can play a game where you will receive $66 if a fair, six-sided die, shows a six on the first throw, and lose one dollar if the die does not show a six.

The expected value of option a) is $10, only one outcome is given. The expected value of option b) consists of two components we need to add up. It is calculated as follows:

The probability of (die not showing a six) is multiplied by the value of that outcome:

(⅚)*(-1) = -$0.83

The probability of (die will show a six) is multiplied by the value of that outcome:

(⅙)*66 = $11

Then we calculate the sum of both:

-$0.83+$11 = $10.17

It follows that I win $0.17 more in expectation if I opt to participate in the dice game. Does that mean that someone is irrational when she chooses the $10 with certainty? Not necessarily! It depends on her utility function, that is, it depends on what’s (how) important to her.

The difference between value and utility

It may well be that we end up disproportionately disappointed in the five of the six cases where we lose in the above example, in comparison to how happy we’d be if we win. If a player wants to also include the worth of her own well-being in the considerations, then it could easily be the case that the extra disappointment doesn’t outweigh the expected extra $0.17 gained from choosing the riskier option.

There are many reasons why one might rationally choose the safe option. It could for instance be that we are in a situation in which we urgently need $10, e.g. we are in town without money and absolutely must buy a particular birthday gift for someone before the shops close. In this situation, it would be very bad for us to choose the small chance of winning a lot of money (and thus to maximize the expected amount of money we will own), because in five out of every six cases, we would not be able to afford the present and thus would disappoint the person we hoped to buy it for – something we’d consider much worse than forfeiting a mere 17 cents in expectation.

No one simply wants to maximize the amount of money they own. Having twice as much money is not always twice as good. Money (and goods general) usually have diminishing marginal utility.

It is often assumed that maximizing utility always has to do with egoism, or the accumulation of money. This is an unfortunate misconception. In principle, all sorts of things fall under utility, it just comes down to what our goals are. Altruistic goals, too, can perfectly well fall under the term utility, as it simply constitutes all that we care about.

Why maximize expected utility?

Why should we maximize expected utility and not any other relation between probability of occurrence of an outcome and its utility?

Assuming that we are completely altruistic and that our utility increases linearly with every person we help (i.e. it would be n times better to help n people than it would be to help one person), consider the following scenario:

An epidemic has broken out on a small island. All 20,000 inhabitants may die from a disease that leads to agonizing death by suffocation within three days after the infection. Experts estimate that, without intervention, the entire island’s population will become infected and suffocate within a few weeks. The inhabitants of this island do not have medicine to cure themselves, so they are dependent on external aid. We find ourselves on a larger neighboring island and are coordinating a rescue strategy. We are faced with choosing the medicine, we can use SafeRelieve and/or CheapRelieve. Both compounds are curative as well as preventive. SafeRelieve costs $2.04 per person treated, and treated patients recover in 100% of cases. CheapRelieve is a cheaper but less safe product. It heals completely in only 50% of the cases, whereas in the remaining cases it is not effective at all. This treatment costs $1 per person. The budget for the rescue is strictly restricted to $10,000. The packages will be distributed so that each person in the population has an equal chance of receiving treatment. How should we proceed?

Suppose we spend all our money on SafeRelieve, then we save 10,000 / 2.04 or about 4,900 inhabitants. If we spend our money on the second drug, we save 10000/1 * 0.5, i.e, approximately 5,000 inhabitants in expectation. CheapRelieve may seem risky, but it is the better choice in expectation.

Theoretically, it would be possible (the chance of this happening is actually 2.3%, see endnote c. here) that we end up unlucky, with CheapRelieve having no effect in a large number of cases, so that fewer people will be helped by it that would be helped by SafeRelieve. However, luck can go the other way too, and we might save even more people than 5,000.


If we split the money fifty-fifty, buying both SafeRelieve and CheapRelieve, then we help 4,950 people in expectation. Due to the SafeRelieve drugs we distribute we reduce the variance, that is we reduce the chance that extremely few people will be saved if we happen to be unlucky. The price we pay for this is that we also reduce the chance that a very large number of people will be saved, and that the expected number of people who will be saved is reduced. If spending all our money on CheapRelieve is better than spending all our money on SafeRelieve, then there seems to be no argument why we should invest any money in SafeRelieve at all. The more drugs you can buy, the clearer it becomes that SafeRelieve is the better choice.

Let’s consider another scenario: Again, we coordinate the rescue package, but this time we find out that we have received an additional donation of $51, with which we can buy more drugs. If we buy SafeRelieve we guarantee that 51/2.04 = 25 additional people will be saved. If we buy CheapRelieve, we save 25.5 people in expectation. This time, there is a 44%(!) chance that CheapRelieve will save less people than SafeRelieve. Shall we, unlike in the case of the large donation, decide to rely on the safer option? The answer is no, because we should not only be concerned about risks, but also about opportunities. Even more often than 44%, CheapRelief will cure more than 25 people.

The additional $51 is not isolated; it is part of the overall budget. If we had started with a budget of $10,051, there would not have been any reason to change the strategy we chose previously. As we have seen above, it is better not to distribute the money between the two drugs.

The law of large numbers

The expected value of an outcome essentially measures the average outcome that would occur if we went through the same scenario a huge number of times. CheapRelieve is the drug with the highest expected utility. In addition, the larger the number of people that we are treating, the less likely it is the actual outcome ends up being much lower than the expected one.

An argument for always maximizing expected value is the “Law of Large Numbers”. In the long term, if we had to go through the same scenario again and again, the total amount of utility we would achieve is virtually guaranteed to be highest when we maximize the expected value for each decision.

This argument may be persuasive, but one could still argue that we really make the decision only once. Why should we maximize expected utility in unique decision-situations?


To respond to this, it’s useful to consider the following: If we are not going to maximize expected value, what else would we do? Once we begin to value outcomes less as their risk increases, the question becomes: How much less do we want to weight them? Here it seems as though, in principle, any form of weighting would be possible, without any of these infinite possible weightings standing out. Maximizing expected utility, however, is unique and makes sense to use.

Axiomatic approach

Another argument for expected utility maximization is the axiomatic approach, based on the Von Neumann-Morgenstern utility theorem. If a person states her preferences over a set of lotteries/bets, and if these preferences are in accordance with four intuitively very plausible axioms, then it can be proven that this person is acting as if she wanted to maximize her expected utility. Or in other words, this means that the rejection of expected value maximization results in a violation of at least one intuitively very obvious axiom.

The altruistic perspective

For altruistic utility functions there is, interestingly, an additional argument for maximizing expected value even in isolated decision-making situations. We can make decisions as if we were behind the veil of ignorance. We can ask ourselves which decision we would prefer as a randomly selected person in the group of affected beings. We would choose the course of action in which our chance of being saved is greatest. In the example involving the island, we would always use all our money to buy CheapRelieve, even if only $51 worth of drugs can be purchased.

Attitude to risk

Always choosing the option that maximizes expected utility is called being risk neutral. Preferring outcomes where there is higher safety (even if the expected value of another option would be higher) is considered risk averse. Because people often perceive losses to be greater than gains, they may often make irrational, risk-averse decisions.

People who argue that risk aversion can be rational often do not understand the difference between value and utility. If someone offers me a bet in which I would put my entire fortune on a coin toss, giving me one extra dollar if I win, this bet would give me $0.50 profit in expected monetary value. However, in terms of utility, it would obviously be much worse for me to lose all my belongings than it would be good for me to double my wealth. The fact that with money one is sometimes (but not always!) “risk-averse” makes sense because, ultimately, it is utility that we want to maximize.


This piece is based on Brian Tomasik’s Why maximize expected value?

Martin Peterson (2009). An Introduction to Decision Theory. Cambridge University Press