Statistical Mechanics Lecture 1
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Susskind, Leonard
Leonard Susskind introduces statistical mechanics as one of the most universal disciplines in modern physics. He begins with a brief review of probability theory, and then presents the concepts of entropy and conservation of information.
 Okay, if there are no more questions, tonight we begin the study of statistical mechanics. Now statistical mechanics is not really modern physics. It's pre-modern physics. It's modern physics. And I assure you it will be post-modern physics. It's probably, the second law of thermodynamics will probably outlast anything that comes up time and time again. The second law of thermodynamics has been sort of our guidepost, our guiding light, if you like, to know what we're talking about and to make sure we're making sense. Statistical mechanics and thermodynamics may not be as sexy as the Higgs boson, but I assure you it is at least as deep. And it's a lot deeper. My particle physics friends shouldn't disown me. It's a lot deeper. It's a lot more general. And it covers a lot more ground than explaining the world as we know it. And in fact, without statistical mechanics, we probably would not know about the Higgs boson. All right, so with that little starting point, what are statistical mechanics about? Well, let's go back a step. The laws of physics, the basic laws of physics, Newton's laws, principles of classical physics, quantum mechanics, the things that were in the classical mechanics course, quantum mechanics, and so forth, those things are all about predictability, perfect predictability. Now you say, well, in quantum mechanics, you can't predict perfectly. And that's true, but there are some things you can predict perfectly. And those things are the predictables of quantum mechanics. Again, as in classical mechanics, you can make your predictions with maximal, let's call it maximal precision or maximal, whatever it is, predictability, if you know two things. If you know the starting point, which is what we call initial conditions, and if you know the laws of evolution of a system. If you can measure the, or if you know for whatever reason, the initial starting point of a closed system, a closed system means one which is either everything or it is sufficiently isolated from everything else that the other things in the system don't influence it. If you have a closed system, if you know the initial conditions exactly, or at least with whatever precision is necessary, and you know the laws of evolution of the system, you have complete predictability and that's all there is to say. Now of course, in many cases that complete predictability would be totally useless, having a list of the position and velocities of every particle in this room would not be very useful to us. The list would be too long and subject to rather quick change as a matter of fact. So you can see while the basic laws of physics are very, very powerful in their predictability, they also in many cases can be totally useless for actually analyzing what's really going on. Statistical mechanics is what you use, basically probability theory. Statistical mechanics, let me say first of all, it is just basic probability theory. Statistical as applied to physical systems. When is it applicable? It's applicable when you don't know the initial conditions with complete perfection. It's applicable when you, it may even be applicable if you don't know the laws of motion with infinite precision. And it's applicable when the system you're investigating is not a closed system, whether it's interacting with other things on the outside. In other words, in just those situations where ideal predictability is impossible, then what do you resort to? You resort to probabilities. But because the number of molecules in this room is so large, and probabilities tend to become very, very precise predictors when the laws of large numbers are applicable, statistical mechanics itself can be highly predictable, but not for everything. As an illustration, you have a box of gas. The box of gas might even be an isolated closed box of gas. It has some energy in it. The particles rattle around. If you know some things about that box of gas, you can predict other things with great precision. If you know the temperature, you can predict the energy in the box of gas. You can predict the pressure. These things are highly predictable, but there are some things you can't predict. You can't predict the position of every molecule. You can't predict when there might be a fluctuation. A fluctuation, which, you know, fluctuations are things which happen which don't really violate probability theory. They're the sort of tails of the probability distribution, things which are unlikely but not impossible. Fluctions happen from time to time in the sealed room every so often. An extra large group, an extra large density of molecules will appear in some small region bigger than the average someplace else. Molecules will be less dense. And fluctuations like that are hard to predict. You can predict the probability for a fluctuation, but you can't predict when a fluctuation is going to happen. It's exactly the same sort of thing, flipping coins. Flipping coins is a good example, probably our favorite example for thinking about probabilities. If I flip a coin a billion times, you can bet that approximately half of them will come up heads and half will come up tails within some margin of error. But there will also be fluctuations. Every now and then, if you do it enough times, a thousand heads in a row will come up. Can you predict when a thousand heads will come up? No. But can you predict how often a thousand heads will come up? Yes. Not very often. So that's what statistical mechanics is for. It's for making statistical probabilistic predictions about systems which are either too small, contain elements which are too small to see, too numerous to keep track of, usually both, too small to see by sea. I mean, you know, it's true. You can see some pretty small things, molecules, but so they may not be too small to see. But there are too many of them. There are too many of them to keep track of. And that's when we use probability theory or statistical mechanics. We're going to go through some of the basic statistical mechanics applications, not just applications, the theory, the laws of thermodynamics, the laws of statistical mechanics, and then how they apply to gases, liquids, solids, whether we will get the quantum mechanical systems or not. I don't know. But just the basic ideas. Okay. And, incidentally, another thing which is very striking is that generally speaking over the history of, certainly over my history in physics, and I'm sure this goes back to the middle of the 19th century sometime, all great physicists, all of them were masters of statistical mechanics. It may not have been the sexiest thing in the world, but they were all masters of it. Why? First of all, because it was useful, but second of all, because it is truly beautiful. It is a truly beautiful subject of physics and mathematics. And it's hard not to get caught up in it. Not to fall in love with it. The reason I teach it is not for you. It's for me. I love teaching it. I love teaching it. I teach it over and over and over again. And in a sense, my life is consisted of learning and forgetting and learning and forgetting and learning and forgetting statistical mechanics. So here's my opportunity to learn it again. Okay. Let's begin with what I usually call a mathematical interlude. In this case, it's not an interlude. It's a starting point. And I'm just going to make some extremely brief remarks which you all know. At least I think you all know them about probability. Just to have, you know, just to level the ground, what are we talking about? And what I am not going to explain, because I don't think anybody can explain it, is why probability works. Why does it work? If you ask why it works, the first answer will be it doesn't always work. You may have a probability for something and you test it out. And sometimes it doesn't work. Those are called the exceptions. So the answer to the question is why does it work? Well, it doesn't always work. It mostly works, except when it doesn't. When doesn't it? Rarely. How rarely? Every so often. But there is a calculus of probability, a mathematical theory of probabilities. And we'll talk about it a little bit. Okay. So we'll take probability to be a primitive concept, basically primitive concept. And we'll suppose that there is a space of some sort, a space of possibilities. The space of possibilities could be the space of outcomes of experiments, or it could actually be the space of states of a system. The state of a system could be the outcome of an experiment. If the experiment consists of determining the state of the system, then the state of the system is the outcome. So we have a space, and let's call that space, let's label the elements of that space with a little i. For example, if we were flipping coins, i would be either heads or tails. If we were flipping dice, you know, dice, dice of dice, i would run from one to six. If there were two dyes, then we would have enough indices to keep track of two dice and so forth. So i is the space of possibilities of outcomes, or the space of possible states of a system. And if we are ignorant, statistics always has to do with ignorance. You don't know everything, and so you assign probabilities to outcomes. And so we assign a probability p of i to the ith outcome to the answer to our question. Okay? What are the rules for p of i? What does p of i have to satisfy? And let's, for the beginning, at least in the beginning, let's imagine that i enumerates some discrete finite collection of possibilities. Later on, we can have an infinite number of possibilities, or even a continuously infinite number of possibilities. But for the time being, i might run from one to n, n possibilities. And the rules are, first of all, p sub i has to be greater than or equal to zero. Negative probabilities, we don't like them. Don't know what they mean. Okay? Next, the summation over i of p sub i, p of i, should be one. That means that the total probability, when you add everything up, all possibilities should be one. You certainly should get some result. Okay? Next, now this is a kind of hypothesis. This is the law of large numbers that if you either make many replicas of the same system or do the same experiment over and over a very, very large number of times, and take all of the outcomes which gave you all of the experiments which gave you the i-th outcome, that's some number, let's call it n of i, that's the number of times that the experiment turned up the i-th possibility, and you divide it by the total number of trials. Total number of trials means the sum of all i, or just the total number of trials, that the limit of this, this is a physical hypothesis. It's a physical hypothesis, it can go wrong if n is not large enough, but in the limit of large n, n goes to infinity, and the limit of very, very, of course n never goes to infinity, you never get to do an infinite number of experiments. But nevertheless we're kind of idealizing, we're assuming we can do so many experiments that the limit n goes to infinity is effectively been reached, then that is p of i. So p of i controls by assumption the ratio of the n of i's. Okay, everybody happy with that? You use this all the time I think. Well, sometimes we use it. Okay, now let's suppose that there is a quantity, let's call it f of i. It's some quantity that's associated with the i-th state. We can assign it, we can make it up. For example, if our system is heads and tails and nothing but heads and tails, we could assign f of heads and call it plus one, and f of tails and call it minus one. If our system has many, many more states, we may want to assign a much larger number of possible f's, but f is some function of the state. It's also a thing that we imagine measuring. It could be the energy of a state, or it could be the momentum of a state. Given a state of some system, it has an energy. It would be called in that case perhaps e of i, or it could be the momentum, or it could be something else. It could be whatever you happen to like to think about. Then an important quantity is the average of f of i. The average of f of i, I will use the quantum mechanical notation for it, even though we're not doing quantum mechanics. It's a nice notation. Physicists tend to use it all over the place. Mathematicians hate it. Just put a pair of brackets around it. It means the average. The average value of the quantity averaged over the probability distribution. It has a definition. Its definition is that it's the sum of i of f of i weighted with the probability. For example, and incidentally, the average of f of i does not have to be any of the possible values that f can take on. For example, in this case, where f of heads is plus one and f of tails is minus one, and you flip a million times, and the probability is a half of heads and a half of tails, the average of f will be zero. So it's not a possible outcome to the experiment. There's no rule why the average should be one of the possible experimental outputs, but it is the average. This is its definition. Each value of f is weighted with the probability for that value of f. You can write it another way. You can write it as a sum over i of f of i times the number of times that you measure i divided by the total number of measurements. That's what p of i is in the limit that there are a large number of measurements. That's defined to be the average. That's our mathematical preliminary for today. That's all I wanted to level the playing field by making sure everybody knows what the probability is and what an average is. We'll use it over and over. Okay, let's start with coin flips. I always start with coin. I start every single class with coin flips, even when I'm teaching about the Higgs boson. Okay. If I flip a coin a lot of times, or whether I flip a coin a lot of times or not, the probability for heads is usually deemed to be one-half and the probability for tails is usually also deemed to be one-half. Why do we do that? Why is it a half and a half? What's the logic there? What logic tells us that? In this case, it's symmetry. It's the symmetry of the coin. Of course, no coin is perfectly symmetric and even making a little mark on it to distinguish the heads and tails, bias is it a little bit, but apart from that tiny, tiny bias of marking the coin with maybe just a tiny little scratch, the coin is symmetric. Higgs and tails are symmetric with respect to each other and therefore there is no reason, no rationale for when you flip a coin for it to turn up heads more often than tails. It's symmetry quite often. I might even say always in some deeper sense, but at least in many cases, symmetry is the thing which dictates probabilities. Probabilities are usually taken to be equal for configurations which are related to each other by some symmetry. Symmetry means if you act with a symmetry, you reflect everything, you turn everything over that the system behaves the same way. Okay, another example besides coin flipping would be dice flipping. Dice flipping instead of having two states has six states, one die, and we can imagine coloring them. We color the faces, red, yellow, blue, and then on the back green, purple, and orange. Okay, that's our die and it's been colored. We don't have to keep track of numbers, we can keep track of colors. And what is the probability that when we flip the die, flip it into the air, hits the ground, what's the probability that it turns up red? No, it's one-sixth, right? There are six possibilities. They're all symmetric with respect to each other. We use the principle of symmetry to tell us that the P of each i, they're all equal and they're all equal to one-sixth. But what if there is no symmetry? What if really the die is not symmetric? For example, what if it's weighted in some unfair way? Or what if it's been cut with faces that are not nice and parallel cubes? Then what's the answer? The answer is symmetry won't tell you. You may be able to use some deeper underlying theory and to use some concept of symmetry from the deeper underlying theory, but in the absence of something else, there is no answer. The answer is experiment. Do this experiment a billion times, keep track of the numbers, assume that things have converged and that way you measure the probabilities. You measure the probabilities and thereafter you can use them. You can use them if you keep a table of them and then you can use them in the next round of experiments. Or you may have some theory, some deep underlying theory which tells you well, like quantum mechanics or statistical mechanics. Statistical mechanics tends to rely mostly on symmetry, as we'll see. So if there's no symmetry to guide you or to guide your implementation of probabilities, then it's experiment. Now there's another answer. There's another possible answer. This answer is often frequently invoked and it's a correct answer under other circumstances. It can have to do with the evolution of a system, the way a system changes with time. So let me give you some examples of what it might have to do. Let's take our six sided cube and assume that our six sided cube is not symmetric. It's not symmetric but we know a rule. We know that if we put that cube down on the table, it's not a cube, when we put that die down on the table and we stand back, this thing has this habit of jumping to another state and jumping to another state and jumping to another state. It's called the law of motion of the system. The law of motion of the system is that whatever it is at one instant, at some next instant, it will be something else according to a definite rule. The instance could be seconds, it could be microseconds or whatever, but imagine a discrete sequence and let's suppose there's a law, a genuine law that tells us how this cube moves around. For example, if it's red, now we've done this over and over many times in different contexts but it is so important that I feel a need to emphasize it again. This is what a law of motion is. It's a rule telling you what the next configuration will be given, it's a rule of updating, of updating configurations. Red goes to blue, blue goes to yellow, yellow goes to green, green goes to orange, orange goes to purple and purple goes back to red. Given the configuration at any time, you know what it will be next and you know what it will continue to do. Of course, you may not know the law. Maybe all you know is that there is a law of this type. You know what I'm going to do next, what am I going to do next? I'm going to draw this law as a diagram. You've all seen me do this in other contexts. Let's do it. We have red, too hard to draw squares. Red, blue, green, orange, what happened? Yellow, yellow, orange, purple. A law like this can be just represented by a set of lines connecting a set of arrows. Red goes to blue, blue goes to green, green goes to yellow, yellow goes to orange, orange goes to purple, purple goes back to red. Given the assumption now that there's a discrete time interval between such events, I am not assuming that the cube has any symmetry to it anymore. The cube may not be symmetric at all. It may have points, you know, one edge, one face, maybe tiny, another face, but if this is the rule to go from one configuration to another and each step takes, let's say, a microsecond, I might have no idea where I begin, but I can still tell you if I, let's say it's a microsecond, a microsecond, and my job is to catch it at a particular instant and ask what the color is. I don't know where it started, okay? But I can still tell you the probability for each one of these is one-sixth. It doesn't have to do with symmetry. Well, maybe it does have to do with some symmetry, but in this case, it wouldn't be the symmetry of the structure of a die. It would just be the fact that as it passes through these sequence of states, it spends one-sixth of its time red, one-sixth of its time blue, one-sixth of a time green, and if I don't know where it starts and I just take a flash, you know, a flash shot of it, my probability will be one-sixth. Now that one-sixth did not really depend on knowing the detailed law. For example, the law could have been different. Let's make up a new law. Red goes to green, green goes to orange, orange goes to yellow, yellow goes to purple, purple goes to blue, and blue goes back to red. This shares with the previous law that there's a closed cycle of events in which you pass through each color once before you cycle around. You may not know which law of nature is for this system, but you can tell me again that the probability will be one-sixth for each one of them. So this prediction of one-sixth doesn't depend on knowing the starting point and doesn't depend on knowing the law of physics. It's just important to know that there is a particular kind of law. Are there possible laws for the system which will not give you one-sixth? Yes. Let's write another law. Red, blue, green, yellow, orange, purple. This rule says that if you start with red, you go to blue. If you start with blue, you go to green, and if you get to green, you go back to red. Or if you start with purple, you go to yellow, yellow to orange, orange back to purple. Notice in this case, if you're on one of these two cycles, you stay there forever. If you knew you were on the upper cycle, if you knew you would start it, it doesn't matter where you start, but if you knew that you started on the upper cycle somewhere, then you would know that there was a one-third probability to be red, a one-third probability to be blue, and a one-third probability to be green, and zero probability to be red, yellow, or orange. On the other hand, you could have started with the second cycle. You could have started with purple. Might not have known where you started, but you knew that you started in the lower triangle and the lower cycle here. Then you would know the probabilities of one-third for each of these and zero for each of these. Now, what about a more general case? The more general case might be that you know with some probability that you start on the upper triangle here and with some other probability on the lower triangle. In fact, let's give these triangles names. Let's call this triangle the plus one triangle, and this one the minus one triangle. Just giving them names, attaching to them a number, a numerical value. If you hear something or other is called plus one, if you hear something or other is called minus one. All right, now you'll have to append, you'll have to start with something you've got to get from someplace else. It doesn't follow from symmetry, and it doesn't follow from cycling through the system some probability that you're either on cycle plus one or cycle minus one. Where might that come from? Flipping somebody else's coin over here, flipping a coin over here might decide which of these two. It might be a biased coin, so you will have a probability to be plus one and a probability to be minus one. These two probabilities are not probabilities for individual colors, they're probabilities for individual cycles. Okay, now what's the probability for blue? The probability for blue begins with the probability that you're on the first cycle, times the probability that if you're on the first cycle, you get blue. That's one third. So the probability for blue, red, or green is one third the probability that you're on the first cycle, and likewise the probability that you're at yellow will be, this is the probability for red, blue, or green in this case, and this times one third will be the probability for purple, yellow, or orange. Okay, so in this case you need to supply another probability that you've got to get from somebody's else. This case here is what we call having a conservation law. In this case, the conservation law would be just the conservation of this number. For red, blue, and green, we've assigned the value plus one. That plus one could be the energy, or it could be something else. I tend to call it the zilch for some reason. I call everything a zilch if there is no name for it. So anyway, let's think of it as the energy to keep things familiar. The energy of these three configurations might all be plus one, and the energy of these three configurations might all be minus one. And the point is that because the rule keeps you always on the same cycle, that quantity, energy, zilch, whatever we call it, is conserved. It doesn't change. That's what a conservation law is. A conservation law is that the configuration space, the space of possibilities, divides up into cycles like this. Now, the cycles don't have to have equal size. Here's another case. One, two, three, four. You go around this way, and then the two guys over here go into each other. So red goes to blue, goes to green, goes to purple. That's the upper cycle here, and the lower cycle is yellow goes to orange, goes to yellow goes to orange. Still, we have a conservation law here. It's just the number of states with one value of the conserved quantities, not the same as the number of states or the other value, but still, it's a conservation law. And again, somebody would have to supply for you some idea of the relative probabilities of these two. Where that comes from is part of the study of statistical mechanics. And the other part of the study has to do with saying, if I know I'm one of these tracks, how much time do I spend with each particular configuration? That's what determines probabilities of statistical mechanics. Some a priori probability from somewheres that tells you the probabilities for different conserved quantities and cycling through the system. Yeah, question? No, okay. So, so far, you're assuming that within any conservation arena, you will, the probabilities of all the states are the same? The time spent in each state is the same. Right. So, it's completely deterministic. Laws are completely deterministic. This would be classical physics. Laws completely deterministic. No real ambiguity of what the state is, except you're kind of lazy. You didn't determine the initial condition. Your timing wasn't very good. Each state only lasts for a microsecond. You're a lazy guy and you only have a resolution of a millisecond. But nevertheless, you're able to take a very quick flash picture and pick out one of the states. That's the circumstance that we're talking about. Yeah. If we take two pictures, is it reasonable to then assume that if the first picture indicated that we are in one cycle, the later one should indicate the same cycle since it couldn't get out of it? Yes, that's a good assumption. Yes. Right. So, once you determine the value of some conserved quantities, then you know it. And then you can reset the probabilities for it. Unless, all right, so let's talk about honest energy for a minute. Yes, if we have a closed system, to represent the closed system, I will just draw a box. Closed now, closed means that it's not in interaction with anything else and therefore can be thought of as a whole universe unto itself. Okay. It has an energy. The energy is some function of the state of the system, whatever determines the state of the system. Now let's suppose we have another closed system which is built out of two identical or not the identical versions of the same thing. Now, if they're both closed systems, there will be two conserved quantities. The energy of this system and the energy of this system and they'll both be separately conserved. Why? Because they don't talk to each other. They don't interact with each other. The two energies are conserved and you could have probabilities for each of those individuals. But now supposing they're connected. They're connected by a little tiny tube which allows energy to flow back and forth. Then there's only one conserved quantity, the total energy, and it's sort of split between the two of them. You can then ask, what is the probability given a total amount of energy? You could ask, what's the probability that the energy of one subsystem is one thing and the energy of the other subsystem is the other? If the two boxes are equal, you would expect on the average they have equal energy, but you can still ask, what's the probability for a given energy in this box given some overall piece of information? That's a circumstance where it may be that giving the probability for which cycle you're on, now which cycle you're on, I'm talking about the cycle of one of these systems here, may be determined by thinking about the system as part of a bigger system. And we're going to do that. That's important. But in general, you need some other ingredient besides just cycling around through the system here to tell you the relative probabilities of conserved quantities. Okay. So we're often flying with statistical mechanics. There are bad laws. By bad laws, I don't, not in the sense of DOMA or any of those kind of laws, but in the sense that the rules of physics don't allow them. You all know what they are. The laws that violate the conservation of information. The most primitive and basic rule of physics, the conservation of information. Conservation of information is not a standard conservation law like this. It's the rule that you keep, that you can keep track. You can keep track both going forward and backward. So let's just mention that again. It's all work, it's all described in the classical mechanics book. I'm just reviewing it now. But let's take a bad law. It's a possible law. By bad, I mean one, two, three, four, one, two, three, four, five, six. So these are the faces of a die again. But the rule is wherever you are, this is red, wherever you are, you go to red. Even if you are red, you go to red. Okay. We'll discuss in a moment what's wrong with this law. But this law has one of the features that it has, is it's not reversible. It's not reversible in the sense that you can go from blue to red, but you cannot go from red back to blue. So in that sense, it's not reversible. You can predict the future wherever you are. The future is very simple for this particular law, wherever you are, you'll next be at red. You can make it more complicated. You could make a few, you can make it more complicated. But this law always winds up with red. It's a bad law because it loses track of where you started. Whereas these laws don't lose track. If you know that you've gone through 56, 56 and a half cycles, then you know that if you started at red, you'll come back and you can tell exactly where you'll be. And you can also tell where you came from. You can tell not only where you'll be, but exactly where you came from. Well, this law, you can't say where you came from. This is a law that loses information. And it's exactly the kind of thing that classical physics does not allow. Classical physics also doesn't allow the quantum mechanical version of it. So the rule that this type of rule, that this type of law is unallowed, I give a name to. It's as I said many times, there is no name for it because it's just so basically primitive that everybody always forgets about it. It's so basic. I call it the minus first law of physics. And I wish it would catch on. People should start using it. I mean, it is really the most basic law of physics that information is never lost, that distinctions or differences between states propagate with time and you never lose track. In principle, if you have the capacity to follow the system, because you may be too lazy to follow the system, that's your problem. But nature doesn't have that problem. Nature allows, in principle, that you can reconstruct where you came from. All right, so that's a bad law. How do you tell the good laws from the bad laws? Just by diagrammatics here, it's very simple. Good laws, every state has one incoming arrow and one outgoing arrow. An arrow to tell you where you came from and an arrow to tell you where you're going. So those are good laws. In classical mechanics, continuum classical mechanics, there is a version of this same law. Anybody know the name of that version? Of the theorem that goes with the conservation of information? It's called Leaville's theorem. We studied it in classical mechanics. But let me give you a counter example to Leaville's theorem. Friction is an apparent contradiction. Wherever you start, you come to rest. It's sort of like saying wherever you start, you come to red. Wherever you start, you come to rest. Well, you may not know exactly where you are, but you always come to rest. That seems like a violation of the laws that tell you that distinctions have to be preserved. But of course, it's not really true. What's really going on is that when you run the eraser through here, it's heating up the surface here. And if you could keep track of every molecule, you would find out that the distinctions between starting points is recorded. But let's imagine now that there was a fundamental law of physics. By fundamental law, I mean a rock bottom fundamental law for a series of particles, for a collection of particles, and the equations of motion for the particles were this. d second x, that's the position of the particle, somewhere by dt squared, that's called acceleration. We could put a mass in, but the mass is not doing anything. There's a lot of particles, so I'll label them i. Oh, we've used i to label states. I should not do that. Let's call it n, little n. The nth particle, and what is that equal to? It's equal to minus some number gamma, we've seen that number before in another context, times dxn by dt. Anybody remember what this formula represents? Friction. Viscous drag. Again, it has the property that if you start with a moving particle, it will very quickly come almost to rest. It'll exponentially come to rest pretty quickly. And so if all particles in a gas, for example, satisfied this law of physics, it's perfectly deterministic. It tells you what happens next, but it has the unfortunate consequence that every particle just comes to rest. That sounds odd. It sounds like no matter what temperature you start the room, it will quickly come to zero temperature. That doesn't happen. This is a perfectly good differential equation, but there's something wrong with it from the point of view of conservation of energy. There's something wrong with it from the point of view of thermodynamics. If you start a closed system and you start it running, you start with a lot of kinetic energy, temperature we usually call it. It doesn't run to zero temperature. That's not what happens. In fact, this is not only a violation of energy conservation. It looks like a violation of the second law of thermodynamics. It says things get simpler. You start with a random bunch of particles moving in random directions, and you let it run and they all come to rest. What you end up with is simpler and requires less information to describe than what you started with. That's very, very much like everything going to red. Among other things, it violates the second law of thermodynamics, which generally says things get worse. Things get more complicated, not less complicated. Okay. But there's another way to say, another important way to say this rule, that every state has to have one arrow in and one arrow out. The thing that I called either the minus first law or the conservation of information. Supposing we have a collection of states and we assign to them probabilities. P of state one, P of state two, P of state three, and so forth. For some subset of the states, not all of them, some subset of them. All the others, we say have probability zero. Okay. So, for example, we can take our die and assign red, yellow, and blue probability a third, and green, orange, and pink, or whatever it was, probability zero. Where we got that from, doesn't matter. We got it from somewhere. Somebody secretly told us in our ear, it's either red, yellow, or blue, and I'm not going to tell you which. All right. And now you follow the system. You follow it as it evolves. Whatever kind of law of physics, as long as it's an allowable law of physics, after a while, and you're following it in detail, you're not constrained by your laziness in this case. You are capable of following in detail. And what is the probability, what are the probabilities at a later time? Well, if you don't know which the laws of physics are, you can't say, of course. But you can say one thing. You can say there are three states with probability one-third and three states with probability zero. They may get reshuffled, which ones were probable and which ones were improbable, but after a certain time, there will be those same three, not the same three states, but there will continue to be three states which have probability and the rest don't. So in general, you could characterize these information-conserving theories by saying, supposing you assign some subset of the states, let's say, N out of N states, let's say there are N states altogether, that's the total number of states, and now we look at some M where M is less than N, and we say for those M states, the probability for those M states is one over M for these states and zero for all the others. You understand why I say one over M? If there are M states equally probable, then each one has probability one over M and all the remaining have probability zero. Then the number of states which have non-zero probability will remain constant and the probabilities will remain equal to one over M. Is that clear? Is that obvious? That should be obvious. The states may reshuffle, but the number with non-zero probability will remain fixed. That's a characterization, a different characterization of the information-conserving laws. For the information-non-conserving laws, everybody goes to red. You may start with a probability distribution that's one over five for red, green, purple, orange, and yellow, and then a little bit later, there's only one state that has a probability and that's red. This is another way to describe information conservation. We can quantify that. We can quantify that by saying let M be the number of states which all have equal, under the assumption that they all have equal probability, let M be the number, let's give it a name, occupied states, states which have non-zero probability with equal probability, and then M, what is M characterizing? M is characterizing your ignorance. The bigger M is, if M is equal to N, that means equal probability for everything. Maximum ignorance. If M is equal to one-half N, that means you know that the system is in one out of half the states. You're still pretty ignorant, but you're not that ignorant. You're less ignorant. What's the maximum, what's the minimum amount of ignorance you can have? That you know precisely what state it's in, in which case M is what? M is one. You know that it's in one particular state. All right, so M is a measure of your ignorance. Really M in relation to N is a measure of your ignorance. And associated with it is the concept of entropy. Now we come to the concept of entropy, notice entropy is coming before anything else. Entropy is coming before temperature, it's even coming before energy. Entropy is more fundamental in a certain sense than any of them, although in a certain sense it's, we'll discuss entropy in a minute, but S is a logarithm of M. Logarithm of the number of states that have an appreciable probability more or less all equal for the specific circumstance that I talked about. That entropy is conserved. All that happens is the states which are occupied reshuffle, but there will always be M of them with probability one over M. Okay, so that's where we are. And that's the conservation of entropy if we can follow the system in detail. Now, of course in reality we may be again lazy, lose track of the system, and we might have after a point lost track of the equations, and lost track of our timing device, and so forth and so on. Now we may wind up, we may have started with some, a lot of knowledge, and wound up with very little knowledge. That's because, again, not because the equations cause information to be lost, but because we just weren't careful. Perhaps we can't be careful, perhaps there are too many degrees of freedom to keep track of. So when that happens, the entropy increases, but it simply increases because our ignorance has gone up, not because anything has really happened in the system which has, if we could follow it, we would find that the entropy is conserved. Okay, that's the concept of entropy in a nutshell. We're going to expand on it. We're going to expand on it a lot. We're going to redefine it with a more careful definition. But what does it measure? It measures approximately the number of states that have a non-zero probability. Okay. The bigger it is, the less you know. What's the maximum value of s? Log in, log in. Now of course, n could be infinite. You might have an infinite number of states, and if you do, then there's no upper bound to the amount of ignorance you can have. But you know, in a world with only n states, your ignorance is bounded. So the notion of maximum entropy is a measure of how many states there are altogether. Now I said that entropy is deep and fundamental, and so it is, but there's also an aspect to it which makes it in a certain sense less fundamental. It's not just a property of a system. It's a property of a system and your state of knowledge of the system. It depends on two things. It depends on characteristics of the system, and it also depends on your state of knowledge of the system or the state of knowledge of the system. So keep that in mind. Okay. Now let's talk about continuous mechanics. Mechanics of particles moving around with continuous positions, continuous velocities. How do we describe that? How do we describe the space of states of a mechanical system, you know, a real mechanical system, particles and so forth? We describe it as points in phase space. We learned about phase space. Phase space consists of positions and momenta. Momenta in simple context, momentum is mass times velocity, so roughly speaking, it's the space of positions and velocities. Let's draw it. P is momentum. It goes that way. And this axis is a stand-in for all of the momentum degrees of freedom. If there are 10 to the 23rd particles, there are 10 to the 23rd P's, but I can't draw more than one of them. Well, I could draw two of them, but then I wouldn't have any room for the Q's, for the X's. And horizontally, the positions of the particles, which we can call X. X or P, doesn't matter. All right. A point here is a possible state of the system. If you know a point here, you know a position and a velocity, and you can predict from that through if you know the forces. Okay, let's start with the analog of a probability distribution, which is zero for some set of states and constant or the same for some other set, for some smaller set. Well, some fraction of the states all have the same probability, and the other states have zero probability. We can represent that by drawing a patch in here, a subregion in the phase space, and say in that subregion, there's equal probability that the system is at any point in here and zero probability outside. This is sort of a situation where you may know something, where you may know something about the particles that they're in some subregion here. For example, you know that all the particles in this room are in the room, right? So that puts some boundaries on what X are. You may know that all the particles have momentum which are within some range that confines them to this way. So a typical bit of knowledge about the room might be represented at least approximately by saying that there's zero probability to be outside this region and a probability equal, I won't say one, but equal probability to be in there. Okay, now what happens as the system evolves? As the system evolves, X and P change. The equations of motion say that X and P change, if you start over here, you might go to here. If you start nearby, you'll go to some nearby point and so forth. And the motion of the system with time is almost like a fluid flowing in the phase space. If you think of the points of the phase space as fluid points and let time go, the phase space moves like a fluid. In particular, this patch over here, let's call it the occupied patch, the occupied patch becomes some other patch. That other patch, after a certain amount of time, the system now is known to be in here. After a certain amount of time, we now know that the system is in here, not in here anymore, and that it has equal, in some sense, equal probability to be anywhere in there. Okay? There's a theorem that goes with this. The theorem is called Leaville's theorem. And what it says is that the volume in phase space, the amount of volume of this region in the XP space, and keep in mind, the XP space may be high dimensional. Not just two, if it were two dimensional, we would think of it as the area. Phase space is never three dimensional. It's always even dimensional. It has a P for every X. So the next more complicated system would be four dimensional. When I speak of the volume in phase space, I mean the volume in whatever dimensionality the phase space is. If you follow the phase space in this manner here, Leaville's theorem, you can go back to Leaville's theorem. It's in the classical mechanics lecture notes. It occupies, I think, a whole lecture, I think. All right, you follow? And it tells you whatever this evolves into, it evolves into something of the same volume. In other words, roughly speaking, the same number of states. It's the immediate analog of the discrete situation where if you start with M states and you follow the system according to the equations of motion, you will occupy the same number of states afterwards as you started with. There'll be different states, but you will preserve the number of them, and the probabilities will remain equal. So the rule is, not the rule, the theorem says that the volume of this occupied region will stay the same. And a little bit better, it says that if you start with a uniform probability distribution in here, it will be uniform in here. So there's a very, very close analog between the discrete case and the continuous case. And this is what prevents this kind of fundamental equation from this kind of equation, an equation where everything comes to rest, that can't happen. Okay, why not? Let's see why it can't happen. Let's just look on this blackboard and see why. Imagine that no matter where you started, you ended up with P equals zero. That would mean every point on here got mapped to the x-axis. It would mean that this entire region here would get mapped to a one-dimensional region, and one-dimensional region has zero area. So Leeville's theorem prevents that. What it says, in fact, is if the blob squeezes in one direction, it must expand in the other direction. The situation for the moving eraser is that if the phase space of the eraser gets shrunk, it means somebody else's, some other components in the phase space, the probability distribution is spread out. What are the other components in this case? It's the P's and X's of all the molecules that are in the table. So for the case of the eraser, there's really a very high-dimensional phase space, and as the eraser may come to rest, almost rest, so that the phase space squeezes this way, it spreads out in the other directions, the other directions having to do with the other hidden microscopic degrees of freedom. Okay, so there we are with information conservation, minus first law of physics, and let's pass... Let's not go to the zeroth law. Let's jump the zeroth law. We'll come back to the zeroth law. You know what the zeroth law says? Well, I'll tell you what it says. We haven't defined what thermal equilibrium is, okay? But it says whatever the hell thermal equilibrium is, if you have several systems, and system A is in thermal equilibrium with B, and B is in thermal equilibrium with C, then A is in thermal equilibrium with C. We will come back to that. Just put it out of your mind for the time being, because we haven't described what thermal equilibrium means. But we can now jump to the first law, minus one, zero, and first law. And the first law is simply energy conservation. It is simply energy conservation, nothing more. It's really simple to write down. That simplicity belies its power. It is the statement that, first of all, there is a conserved quantity, and the fact that we call that conserved quantity energy will play for the moment not such a big role right now, but let's just say there's energy conservation. What does that say? That simply says DE. Simply energy is DE by DT is equal to zero. Now this is the law of energy conservation for a closed system. If a system consists of more than one part in interaction with each other, then of course any one of the parts can have a changing energy, but the sum total of all of the parts will conserve energy. So if a system is composed, as I drew before, of two parts with a link between them, and this is called one and this is two, then this reads that DE1 by DT is equal to minus DE2 by DT. I've really written DE1 by DT plus DE2 by DT is equal to zero, but then I transposed one of them to the right hand side just to indicate, just to make graphic, that if you lose energy on one side, you gain it on the other. So that's the first law of thermodynamics, and that's all the first law of thermodynamics says, it says energy conservation. Now in this context here, there's a slightly hidden assumption. We've assumed that if a system is composed of two parts, that the energy is the sum of the two parts. That's really not generally true. If you have two systems and they interact with each other, there may be, for example, forces between the two parts, so there might be a potential energy that's a function of both of the coordinates. For example, the energy of the solar system, being very naive, I'm thinking of the solar system as two orbiting Newtonian particles. The energy consists of the kinetic energy of one particle plus the kinetic energy of the other particle plus a term which doesn't belong to either particle. It belongs to both of them in a sense, and it's the potential energy of interaction between them. In that context, you really can't say that the energy is the sum of the energy of one thing plus the energy of the other thing. Energy conservation is still true, but you can't divide the system into two parts this way. On the other hand, there are many, many contexts where the interaction energies between systems is negligible compared to the energy that the systems themselves have. If we were to divide this table top up into blocks, let's think about it, divide the table top up into blocks, how much energy is in each block? Well, the amount of energy that's in each block is more or less proportional to the volume of each block. How much energy of interaction is there between the blocks? The energy of interaction is a surface effect. They interact with each other because their surface is touch, and typically, surface area is small by comparison with volume. So many, many, we'll come back to that. We'll come back to that. In many, many contexts, the energy of interaction between two systems is negligible compared to the energy of either of them. When that happens, you can say to a good approximation, the energy can just be represented as the sum of two energies of the two parts of the system plus a teeny little thing which has to do with their interactions. Under those circumstances, the first law of thermodynamics, the top is always true. The second has that little caveat that we're talking about systems where energy is strictly additive, where you add energies. Does everybody understand why I say you don't always add energies that sometimes energies are not additive? Yeah, actually, I was thinking that we have the same possible problem with the probabilities. We assume that the outcomes were mutually exclusive. Otherwise, the sum law doesn't work. Yeah. Yeah. Okay. So in all the contexts which we've talked about, the dye, if it's yellow, it can't be red. You say orange. You say orange, orange is both yellow and red, well, we don't count that way. Yeah. So that's correct. No, that's absolutely correct. We made the assumption that what I called states, what I called states are mutually exclusive. Absolutely. Okay, let's come back to entropy. We're not finished with entropy. We've done entropy. We've done energy. We haven't gotten the temperature yet. Just the temperature comes in behind entropy, and even energy comes in behind entropy. But temperature is a highly derived quantity. By highly derived, I mean it's a, despite the fact that it's the thing you feel with your body, so it makes it really feel like it's something intuitive, it is a mathematically derived concept, less primitive and less fundamental in either energy or entropy, but we'll come to it. Let's come back to entropy. We define entropy, but only for certain special probability distributions. Let's lay out on the horizontal axis, just to be schematic. On the horizontal axis, we will put down all the different states. Here's i equals one, here's i equals two, here's i equals three. This axis just labels the various states, and of course, vertically, let me plot probability. Okay. Well, the probability, of course, is only defined on the integers here. That's not very good. It's some probability. But let's, I don't want to have to draw such a complicated thing every time I want to draw a probability distribution, let's just draw a graph. Some probability distribution, or some probability for each position. Now what we did was we defined entropy for a very special case, the special case being where some subset have equal probabilities and the rest have zero probability. For example, if our subset consists of this group over here of m, the whole group being n, then all of these have the same probability, and their probabilities have to add up to one, so the probability is one over n. We just draw that by drawing a box like that. Then we define the entropy to be the logarithm of the number of states in here. Generally speaking, we don't have probability distributions like this. Generally speaking, we have probability distributions which are more complicated. In fact, they can be anything as long as they're positive and all add up to one. So the question is how do we define entropy in a more general context where the probability distribution looks like this? I'm going to write down the formula, and then we're going to check that it really gives us this answer when it should give us this answer. For today, I'm just going to write it down and tell you this is the definition. You'll get familiar with it, and you'll start to see why it's a good definition. It's representing something about the probability distribution, and what it's representing is in some average sense, the average number of states which are importantly contained inside the probability distribution. The narrower the probability distribution, the smaller the entropy will be. The broader the probability distribution, the bigger the entropy will be. I'll write it down for you now. We'll write it down and then explore it just a little bit tonight. S for a general probability distribution is, first of all, minus. That's funny because this is positive, but nevertheless the formula begins with minus, a sum, and it's a sum over all of the states, all of the possibilities. So it's a sum over i. There's a contribution for each place here. The probability of i times the logarithm of the probability of i. Do you remember? All right, let's write something else then. Remember that the average of f is equal to the summation over i, f of i times p of i. This is actually the average of log p sub i. It's the average of log p sub i. All right, let's work this out. Let's see what this gives. In the special case where the probability distribution is 1 over m for m states altogether. It has width m, and because it has width m, it must have height 1 over m because all the probabilities have to add up to 1. All right, so let's work this out. Let's take the contribution for all the unoccupied states. All the unoccupied states, p sub i is 0. You get nothing. But log of p over i is minus infinity. Yeah, that's right. Now what's the, all right, good. So let's consider the limit of log p over p. Or let's just say the limit of, no, not that's not right, the limit of p log p as p goes to 0. You know how to calculate that? Okay, I'm going to leave it to you. It's a little calculus exercise to calculate the limit as p goes to 0 of p log p. It's 0. The point is that p goes to 0 a lot faster than log p goes to infinity. Log p as p goes to 0 goes to, is very slow. p goes to 0 fast. So this goes to 0. You're absolutely right though. That has to be, that has to be shown. But p log p in the limit that p goes to 0 is 0. So with that piece of knowledge, the contribution from states with very, very small probabilities, probability will be very, very small. And as the probability for those states goes to 0, this quantity, the contribution will go to 0. But what about the ones here which have significant probability? They all have the same piece of body. And they all have the same log piece of body. The log piece of body is logarithm of 1 over m for all of them. The piece of body is also 1 over m. Not log 1 over m, but 1 over m. So each contribution is 1 over m times log 1 over m. How many contributions are there like that? All right, so we multiply by m and get rid of the 1 over m there. There's a minus sign here, I'll carry it along. All times m because there are m such terms. So the 1 over m cancels and we just left with log 1 over m. What's log 1 over m? Minus log m, right? So that's why the minus sign was put there in the first place. There's no miracle. The minus sign was put there because probabilities are less than 1. And so the logarithms of them are always negative. So you soak up that negative with an overall negative sign and entropy is positive. But this is exactly the same answer that we are, the original definition just s equals log m. Logarithm of the number of states. But this is a definition now that makes sense even when you have a more complicated probability distribution. And it is a good and effective. It's the average of log p. For the special case where the probability distribution is constant like this, then all of the probabilities in here are 1 over m and calculating the average of log 1 over m just gives you this. All right. So this is the general definition of the entropy that's associated with a probability distribution. And notice, entropy is associated with a probability distribution. It's not a thing like energy which is a property of a system. It's not a thing like momentum. It's a thing which has to do with a specific probability distribution, probability distribution on the space of possible states. So that's why it's a little bit of a more obscure quantity from the point of view of, you know, intuitive definition. As I said, its definition has to do with both the system and your state of knowledge of the system. Let's do some examples. Let's calculate some entropy for a couple of simple systems. The first system is just going to be not a single coin but a lot of coins. So we have capital N coins, N of them, and each one can be heads or tails, et cetera, and so on. As a matter of fact, we have no idea what the state of the system is. We know nothing. The probability distribution, in other words, is the same for all states. Good ignorance, absolute ignorance. What is the entropy associated with such a configuration? There are n of these. All the probabilities are equal. Under the circumstance where all of the probabilities are equal, we just get to use logarithm of the number of possible states. The answer here is the logarithm of the number of total states. How many states are there altogether? Two to the N. Right. Two to the N. Oh, let's, I'm sorry, I'm going to change definitions for a minute for a reason that you'll see in a minute. I'm going to call this little N. Number of coins is little N. Over here, big N stood for the total number of states. So if I match terminologies, big N, the total number of states, is e to the 2n, no, 2 to the N. Sorry, I'm getting tired. 2 to the N. 2 to the N states altogether. Two states for the first coin, two states for the second coin, and so forth. 2 to the N altogether. And the total number of states is capital N. What's the entropy given that we know nothing? Two log N. Two log N. N log 2. S is equal to N log 2. That's the logarithm of 2 to the N. All right, so here we see an example of the fact that entropy is kind of additive over the system. It's proportional to the number of degrees of freedom in this case. N times log 2. And we also discover a unit of entropy. The unit of entropy is called a bit. That's what a bit is, an information theory. It's the basic unit of entropy for a system which has only two states, up or down, heads or tails or whatever. The entropy is proportional to the number of bits, or in this case, the number of coins, times the logarithm of 2. So log 2 plays a fundamental role in information theory as the unit of entropy. It does not mean that in general that entropy is an integer multiple of log 2. We'll see in a second, it doesn't mean that. Okay, so that's this case over here. Let's try another case. Let's try another state of knowledge. Here our state of knowledge was zilch. We knew nothing. This is the maximum entropy. Logarithm of capital N, or logarithm of 2 to the little n, which is n log 2. One bit of entropy for each coin, if you like. Okay, let's try something else. Let's try a state in which what we know is, oh, something else. Supposing we know the state completely. In other words, that's the case where m is equal to 1. That would be the case where we know that the probability is only nonzero for one state. Then m is 1 and s is 0, logarithm of 0. So absolute knowledge, perfect knowledge, complete knowledge corresponds to zero entropy. The more you know, the smaller the entropy, excuse me. Okay, let's take an interesting case. Let's take our heads and tails again. And here's what we know. We know that all of them are heads. Let's begin with that, all of them are heads, what's the entropy then? Zero. Except for one of them, which is tails. Now, supposing we know which one is tails, what's the entropy? Zero. But suppose we don't know which one is tails. Equal probability for all states which contain one tail and n minus one heads. What's the entropy? Indeed. Okay, so why is it log n? How many states are there with nonzero probability? The answer is little n. It could be this one, it could be this one, it could be this one. In other words, for this particular situation, capital M, the number of states that have nonzero probability is just little n. All the states have the same probability. So we're in exactly this situation except that M is just equal to little n. N possible states, this one could be tails, this one could be tails, this one could be tails. They all have equal probability. So capital M is n and the entropy now is, for this situation, the entropy is equal to logarithm of little n. Notice that that's not an integer multiple of log two in general. So in general, entropy is not an integer multiple of log two. Nevertheless, log two is a good unit, it's a basic unit of entropy, it's called a bit. Yes in this case is equal to log n. Yes? The two in the log two come from the fact that you only have a probability of heads or tail, right? Yeah. So if you had three, it'd be log three. Absolutely. Absolutely. Now, computer scientists of course like to think in terms of two for a variety of reasons. First of all, the mathematics of it is nice, but two is the smallest number which is not one. It's a small signature. Yeah, it's a small signature, not equal to one. But it's also true that the physics that goes on inside your computer is connected with switches which are either on or off. So counting in units of log two is very useful. Excuse me. Yeah? It's probably not important, but what is the base of the logarithm? Yeah, the standard, this is a definition, of course. This is definition. The definition, okay, good. The, good. It depends on who you are. If you're an information theorist, a computer scientist often, or a disciple of Shannon, then you like log to the base two. In which case, this is just one, n, and the entropy here is just n, measured in units of bits. If you're a physicist, then you usually work in base E. Okay? But the relationship is just a multiplicative, you know, log to the base E and log to the base two are just related by a numerical factor that's always the same. Okay. So, when I write log log, I mean log to the base E, but very little would change if we use some other base for the logs. Okay. So there we are. Let me just tell you what the definition is of the entropy in phase space. If we're not talking about, if we're not talking about these finite discrete systems like this, we're talking about continuously infinite systems, phase space. Now let's begin supposing the probability distribution is just some blob where the probabilities are equal inside the blob and zero outside the blob. In other words, the simple situation. Then the definition of the entropy is simply, well, you could say the logarithm of the number of states, but how many states are there in here? Clearly a continuous infinity of them. So instead, you just say it's the log of the volume of the probability distribution in phase space. S for a continuous system is just defined to be the logarithm of the volume in phase space. Now if I wrote V, you might get the sense that I'm talking about volume in space. No, I'm talking about the logarithm in phase space, the volume in the phase space. The phase space is high dimensional. Another dimensionality of the phase space is the volume is measured in those kind of in units of momentum times velocity to the power of the number of coordinates in the system. All right, but S is equal to the logarithm of the phase space volume. Let's call it P phase, V phase space. That's under the volume of the region which is occupied and has a non-zero probability distribution. This is the closest analog that we can think of to log M, where M represents the number of equally probable states of the discrete system. More generally, if we want, if we have some arbitrary probability distribution, the arbitrary probability distribution, P, what would P probability? What would it be a function of? It would be a function of all of the coordinates and all of the momenta. All of the momenta and all of the coordinates. P's and Q's or P's and X's, whichever you like, all of them. Probability for the system to be located at point of phase space P and X. Incidentally, when you have continuous variables like that, do you write that the sum is all equal to one? That wouldn't make sense. You can't sum over a continuously infinite set of variables. It becomes integral. We'll come back to this, but let's just spell it out right now. That if you have a probability distribution on a phase space, the rule is that the integral of it is equal one. P is really a probability density on phase space. It's a probability per unit cell in phase space. What would you expect the entropy to be? I'll give you a hint that starts out with minus. Where's the formula that we had here before? Let's rewrite the formula that we had before. S is equal to minus summation over I of P sub I log of P sub I. To go to the continuum, you simply replace sum by integral. There's an integral over the phase space. That's like the sum over I. Then probability of P and Q times the logarithm of the probability. In first approximation, I don't mean first approximation numerically. I mean first conceptual approximation. It's measuring the logarithm of the volume of the probability blob in phase space. Okay, oh, I could know mistake. In classical mechanics, X's and Q's are coordinates and P's are momenta. You know that. Okay, so we've now defined entropy, which, as we've seen, depends on a probability distribution. I'm going to go one more step tonight and define temperature. We could stop here. I think that's probably enough for one night. Next time we'll do temperature and then discuss the Boltzmann distribution, which is the probability distribution for thermal equilibrium. We haven't quite defined thermal equilibrium yet, but we will. You can ask some questions. I don't mind some questions now. I just, I just, I have the feeling that I've probably done enough for one night. Oh. Well, I haven't made any such assumption. I haven't made any such assumption. My only assumption in calculating the various entropies was either that you know nothing or I told you what you know. How you got to know that and what the reasoning was and whether it had to do with knowing something about the dynamics and the independence and so forth may come into your calculation of what P is. But in saying you know nothing, the implication was that all states are equally probable. Without asking how you knew that. To say that all states are equally probable is closely related to saying that there are no correlations. It does say that if you, all right, let's, good. Let's talk about correlations for a moment. To say that you know nothing means you know nothing. So in particular, if you, you, you begin knowing nothing and you measure one of the coins, what do you know about the other ones? Nothing. Nothing. You started knowing nothing about them. You measured one of them. You still know nothing about the other ones. Of course, you know about the one that you've measured. Now let's take the other, the other case that we studied. Anything. We know that all coins are heads except for one which is tails. And we now measure one of them and we find that it's tails. What can we say about some other one? It's surely heads. What if we measure that that one is heads? What do we know about the other ones? It changes the probability. It changes the probability. Just a new probability that one of the other ones is tails. Right. It's one over n, one over n minus one instead of one over, sorry, one over, yeah, one over n minus one. So that's correlation. That's correlation where when you measure something, you learn something new or the probability distribution for the other things is modified by measuring something. That's called correlation. For the complete ignorance, there is no correlation. For any other kind of configuration in general, there's very likely to be some, not necessarily, but there's very likely to be some correlation. Correlation as I said means you learn something about the probability distribution of other things by measuring the first one. Or you modify the probability distribution. Good. Okay. I don't know if that's what you asked about or not, but yeah. Okay. All right. Well, is that original system there? We had two parts. Once we measured one thing, we had to conserve quantity. What's that again? We had to conserve quantity there that once we did one measurement, we knew which side we were. Yeah. Yeah. Incidentally, entropy is additive. It's additive. It's the sum of the entropies of all the individuals. It's proportional to a number of things. It's additive whenever there is no correlation. When there's no correlation, it's additive. Now, so uncorrelated systems have additive entropies. We'll come back. That's a theme that we'll come back to. Is an interesting question. Here's what you know, you know that if you measure a coin up, then with three-quarter probability, it's too, they're laid out in a row. If you measure one of them and it's up, then the probability for its neighbors is three-quarters to be down. That's what you're given. That's all you know. All you know is that if one of them is up, if any one of them is up, its neighbors are three-quarters likely to be down. It's an interesting thing to try to calculate the entropy of such a distribution. That is correlated, of course. That is correlated because when you measure one, you immediately know something about its neighbor. Well, make up your own example like that. Make up your own example like that and compute the entropy. You can learn something from it. Any other questions before we go home? Not particularly. The formula here is due to Boltzmann. I think it's just the one that's on Boltzmann's tomb, I'm not sure. Now what's on Boltzmann's tomb? He meant this. Well he did write this. This is Boltzmann's final formula for entropy. The only difference between the Shannon entropy and the Boltzmann entropy is that Shannon used log two. Now, of course, Shannon discovered this entirely by himself. He didn't know Boltzmann's work from an entirely different direction from information theory rather than thermodynamics, but none of it would have surprised Boltzmann. Nor do I think Boltzmann's definition would have surprised Shannon. So they're really the same thing. There's no real point in comparing them because they're comparing them, they are the same. There's no real difference. Shannon may have, I don't know whether he did or not put the minus sign in here. If you don't put the minus sign in there, it's called information. If you put the minus sign, it's called lack of information or entropy. So I don't know which Shannon wrote down. Anyway. Shannon wrote down entropy. He did. Yeah. Is there a simple way to relate to Heisenberg's uncertainty principle? No, no, this is a separate issue. This doesn't have to do with quantum mechanical uncertainty. It has to do with the uncertainty implicit in mixed states, not the uncertainty implicit in pure states. Okay. Oh, oh boy. Yeah, there's a conversion factor. You know, C equals H bar equals G equals Boltzmann's constant equals one. Very natural. Boltzmann's constant was a conversion factor from temperature to energy. The natural unit for temperature is really energy. But the energy of a molecule, for example, is approximately equal to its temperature in certain units. Those units contain a conversion factor K Boltzmann. I'll remind me to talk about K Boltzmann. For more, please visit us at stanford.edu.
 Okay, if there are no more questions, tonight we begin the study of statistical mechanics.  Now statistical mechanics is not really modern physics.  It's pre-modern physics.  It's modern physics.  And I assure you it will be post-modern physics.  It's probably, the second law of thermodynamics will probably outlast anything that comes  up time and time again.  The second law of thermodynamics has been sort of our guidepost, our guiding light,  if you like, to know what we're talking about and to make sure we're making sense.  Statistical mechanics and thermodynamics may not be as sexy as the Higgs boson, but I  assure you it is at least as deep.  And it's a lot deeper.  My particle physics friends shouldn't disown me.  It's a lot deeper.  It's a lot more general.  And it covers a lot more ground than explaining the world as we know it.  And in fact, without statistical mechanics, we probably would not know about the Higgs  boson.  All right, so with that little starting point, what are statistical mechanics about?  Well, let's go back a step.  The laws of physics, the basic laws of physics, Newton's laws, principles of classical physics,  quantum mechanics, the things that were in the classical mechanics course, quantum mechanics,  and so forth, those things are all about predictability, perfect predictability.  Now you say, well, in quantum mechanics, you can't predict perfectly.  And that's true, but there are some things you can predict perfectly.  And those things are the predictables of quantum mechanics.  Again, as in classical mechanics, you can make your predictions with maximal, let's  call it maximal precision or maximal, whatever it is, predictability, if you know two things.  If you know the starting point, which is what we call initial conditions, and if you know  the laws of evolution of a system.  If you can measure the, or if you know for whatever reason, the initial starting point  of a closed system, a closed system means one which is either everything or it is sufficiently  isolated from everything else that the other things in the system don't influence it.  If you have a closed system, if you know the initial conditions exactly, or at least with  whatever precision is necessary, and you know the laws of evolution of the system, you have  complete predictability and that's all there is to say.  Now of course, in many cases that complete predictability would be totally useless,  having a list of the position and velocities of every particle in this room would not be  very useful to us.  The list would be too long and subject to rather quick change as a matter of fact.  So you can see while the basic laws of physics are very, very powerful in their predictability,  they also in many cases can be totally useless for actually analyzing what's really going  on.  Statistical mechanics is what you use, basically probability theory.  Statistical mechanics, let me say first of all, it is just basic probability theory.  Statistical as applied to physical systems.  When is it applicable?  It's applicable when you don't know the initial conditions with complete perfection.  It's applicable when you, it may even be applicable if you don't know the laws of motion with  infinite precision.  And it's applicable when the system you're investigating is not a closed system, whether  it's interacting with other things on the outside.  In other words, in just those situations where ideal predictability is impossible, then what  do you resort to?  You resort to probabilities.  But because the number of molecules in this room is so large, and probabilities tend to  become very, very precise predictors when the laws of large numbers are applicable, statistical  mechanics itself can be highly predictable, but not for everything.  As an illustration, you have a box of gas.  The box of gas might even be an isolated closed box of gas.  It has some energy in it.  The particles rattle around.  If you know some things about that box of gas, you can predict other things with great  precision.  If you know the temperature, you can predict the energy in the box of gas.  You can predict the pressure.  These things are highly predictable, but there are some things you can't predict.  You can't predict the position of every molecule.  You can't predict when there might be a fluctuation.  A fluctuation, which, you know, fluctuations are things which happen which don't really  violate probability theory.  They're the sort of tails of the probability distribution, things which are unlikely but  not impossible.  Fluctions happen from time to time in the sealed room every so often.  An extra large group, an extra large density of molecules will appear in some small region  bigger than the average someplace else.  Molecules will be less dense.  And fluctuations like that are hard to predict.  You can predict the probability for a fluctuation, but you can't predict when a fluctuation is  going to happen.  It's exactly the same sort of thing, flipping coins.  Flipping coins is a good example, probably our favorite example for thinking about probabilities.  If I flip a coin a billion times, you can bet that approximately half of them will come  up heads and half will come up tails within some margin of error.  But there will also be fluctuations.  Every now and then, if you do it enough times, a thousand heads in a row will come up.  Can you predict when a thousand heads will come up?  No.  But can you predict how often a thousand heads will come up?  Yes.  Not very often.  So that's what statistical mechanics is for.  It's for making statistical probabilistic predictions about systems which are either  too small, contain elements which are too small to see, too numerous to keep track of,  usually both, too small to see by sea.  I mean, you know, it's true.  You can see some pretty small things, molecules, but so they may not be too small to see.  But there are too many of them.  There are too many of them to keep track of.  And that's when we use probability theory or statistical mechanics.  We're going to go through some of the basic statistical mechanics applications, not just  applications, the theory, the laws of thermodynamics, the laws of statistical mechanics, and then  how they apply to gases, liquids, solids, whether we will get the quantum mechanical  systems or not.  I don't know.  But just the basic ideas.  Okay.  And, incidentally, another thing which is very striking is that generally speaking over  the history of, certainly over my history in physics, and I'm sure this goes back to  the middle of the 19th century sometime, all great physicists, all of them were masters  of statistical mechanics.  It may not have been the sexiest thing in the world, but they were all masters of it.  Why?  First of all, because it was useful, but second of all, because it is truly beautiful.  It is a truly beautiful subject of physics and mathematics.  And it's hard not to get caught up in it.  Not to fall in love with it.  The reason I teach it is not for you.  It's for me.  I love teaching it.  I love teaching it.  I teach it over and over and over again.  And in a sense, my life is consisted of learning and forgetting and learning and forgetting  and learning and forgetting statistical mechanics.  So here's my opportunity to learn it again.  Okay.  Let's begin with what I usually call a mathematical interlude.  In this case, it's not an interlude.  It's a starting point.  And I'm just going to make some extremely brief remarks which you all know.  At least I think you all know them about probability.  Just to have, you know, just to level the ground, what are we talking about?  And what I am not going to explain, because I don't think anybody can explain it, is  why probability works.  Why does it work?  If you ask why it works, the first answer will be it doesn't always work.  You may have a probability for something and you test it out.  And sometimes it doesn't work.  Those are called the exceptions.  So the answer to the question is why does it work?  Well, it doesn't always work.  It mostly works, except when it doesn't.  When doesn't it?  Rarely.  How rarely?  Every so often.  But there is a calculus of probability, a mathematical theory of probabilities.  And we'll talk about it a little bit.  Okay.  So we'll take probability to be a primitive concept, basically primitive concept.  And we'll suppose that there is a space of some sort, a space of possibilities.  The space of possibilities could be the space of outcomes of experiments, or it could actually  be the space of states of a system.  The state of a system could be the outcome of an experiment.  If the experiment consists of determining the state of the system, then the state of  the system is the outcome.  So we have a space, and let's call that space, let's label the elements of that space with  a little i.  For example, if we were flipping coins, i would be either heads or tails.  If we were flipping dice, you know, dice, dice of dice, i would run from one to six.  If there were two dyes, then we would have enough indices to keep track of two dice and  so forth.  So i is the space of possibilities of outcomes, or the space of possible states of a system.  And if we are ignorant, statistics always has to do with ignorance.  You don't know everything, and so you assign probabilities to outcomes.  And so we assign a probability p of i to the ith outcome to the answer to our question.  Okay?  What are the rules for p of i?  What does p of i have to satisfy?  And let's, for the beginning, at least in the beginning, let's imagine that i enumerates  some discrete finite collection of possibilities.  Later on, we can have an infinite number of possibilities, or even a continuously infinite  number of possibilities.  But for the time being, i might run from one to n, n possibilities.  And the rules are, first of all, p sub i has to be greater than or equal to zero.  Negative probabilities, we don't like them.  Don't know what they mean.  Okay?  Next, the summation over i of p sub i, p of i, should be one.  That means that the total probability, when you add everything up, all possibilities should  be one.  You certainly should get some result.  Okay?  Next, now this is a kind of hypothesis.  This is the law of large numbers that if you either make many replicas of the same system  or do the same experiment over and over a very, very large number of times, and take all of  the outcomes which gave you all of the experiments which gave you the i-th outcome, that's some  number, let's call it n of i, that's the number of times that the experiment turned up the  i-th possibility, and you divide it by the total number of trials.  Total number of trials means the sum of all i, or just the total number of trials, that  the limit of this, this is a physical hypothesis.  It's a physical hypothesis, it can go wrong if n is not large enough, but in the limit  of large n, n goes to infinity, and the limit of very, very, of course n never goes to infinity,  you never get to do an infinite number of experiments.  But nevertheless we're kind of idealizing, we're assuming we can do so many experiments  that the limit n goes to infinity is effectively been reached, then that is p of i.  So p of i controls by assumption the ratio of the n of i's.  Okay, everybody happy with that?  You use this all the time I think.  Well, sometimes we use it.  Okay, now let's suppose that there is a quantity, let's call it f of i.  It's some quantity that's associated with the i-th state.  We can assign it, we can make it up.  For example, if our system is heads and tails and nothing but heads and tails, we could assign  f of heads and call it plus one, and f of tails and call it minus one.  If our system has many, many more states, we may want to assign a much larger number  of possible f's, but f is some function of the state.  It's also a thing that we imagine measuring.  It could be the energy of a state, or it could be the momentum of a state.  Given a state of some system, it has an energy.  It would be called in that case perhaps e of i, or it could be the momentum, or it  could be something else.  It could be whatever you happen to like to think about.  Then an important quantity is the average of f of i.  The average of f of i, I will use the quantum mechanical notation for it, even though we're  not doing quantum mechanics.  It's a nice notation.  Physicists tend to use it all over the place.  Mathematicians hate it.  Just put a pair of brackets around it.  It means the average.  The average value of the quantity averaged over the probability distribution.  It has a definition.  Its definition is that it's the sum of i of f of i weighted with the probability.  For example, and incidentally, the average of f of i does not have to be any of the possible  values that f can take on.  For example, in this case, where f of heads is plus one and f of tails is minus one, and  you flip a million times, and the probability is a half of heads and a half of tails, the  average of f will be zero.  So it's not a possible outcome to the experiment.  There's no rule why the average should be one of the possible experimental outputs, but  it is the average.  This is its definition.  Each value of f is weighted with the probability for that value of f.  You can write it another way.  You can write it as a sum over i of f of i times the number of times that you measure  i divided by the total number of measurements.  That's what p of i is in the limit that there are a large number of measurements.  That's defined to be the average.  That's our mathematical preliminary for today.  That's all I wanted to level the playing field by making sure everybody knows what the probability  is and what an average is.  We'll use it over and over.  Okay, let's start with coin flips.  I always start with coin.  I start every single class with coin flips, even when I'm teaching about the Higgs boson.  Okay.  If I flip a coin a lot of times, or whether I flip a coin a lot of times or not, the probability  for heads is usually deemed to be one-half and the probability for tails is usually also  deemed to be one-half.  Why do we do that?  Why is it a half and a half?  What's the logic there?  What logic tells us that?  In this case, it's symmetry.  It's the symmetry of the coin.  Of course, no coin is perfectly symmetric and even making a little mark on it to distinguish  the heads and tails, bias is it a little bit, but apart from that tiny, tiny bias of marking  the coin with maybe just a tiny little scratch, the coin is symmetric.  Higgs and tails are symmetric with respect to each other and therefore there is no reason,  no rationale for when you flip a coin for it to turn up heads more often than tails.  It's symmetry quite often.  I might even say always in some deeper sense, but at least in many cases, symmetry is the  thing which dictates probabilities.  Probabilities are usually taken to be equal for configurations which are related to each  other by some symmetry.  Symmetry means if you act with a symmetry, you reflect everything, you turn everything  over that the system behaves the same way.  Okay, another example besides coin flipping would be dice flipping.  Dice flipping instead of having two states has six states, one die, and we can imagine  coloring them.  We color the faces, red, yellow, blue, and then on the back green, purple, and orange.  Okay, that's our die and it's been colored.  We don't have to keep track of numbers, we can keep track of colors.  And what is the probability that when we flip the die, flip it into the air, hits the ground,  what's the probability that it turns up red?  No, it's one-sixth, right?  There are six possibilities.  They're all symmetric with respect to each other.  We use the principle of symmetry to tell us that the P of each i, they're all equal and  they're all equal to one-sixth.  But what if there is no symmetry?  What if really the die is not symmetric?  For example, what if it's weighted in some unfair way?  Or what if it's been cut with faces that are not nice and parallel cubes?  Then what's the answer?  The answer is symmetry won't tell you.  You may be able to use some deeper underlying theory and to use some concept of symmetry  from the deeper underlying theory, but in the absence of something else, there is no  answer.  The answer is experiment.  Do this experiment a billion times, keep track of the numbers, assume that things have converged  and that way you measure the probabilities.  You measure the probabilities and thereafter you can use them.  You can use them if you keep a table of them and then you can use them in the next round  of experiments.  Or you may have some theory, some deep underlying theory which tells you well, like quantum  mechanics or statistical mechanics.  Statistical mechanics tends to rely mostly on symmetry, as we'll see.  So if there's no symmetry to guide you or to guide your implementation of probabilities,  then it's experiment.  Now there's another answer.  There's another possible answer.  This answer is often frequently invoked and it's a correct answer under other circumstances.  It can have to do with the evolution of a system, the way a system changes with time.  So let me give you some examples of what it might have to do.  Let's take our six sided cube and assume that our six sided cube is not symmetric.  It's not symmetric but we know a rule.  We know that if we put that cube down on the table, it's not a cube, when we put that die  down on the table and we stand back, this thing has this habit of jumping to another  state and jumping to another state and jumping to another state.  It's called the law of motion of the system.  The law of motion of the system is that whatever it is at one instant, at some next instant,  it will be something else according to a definite rule.  The instance could be seconds, it could be microseconds or whatever, but imagine a discrete  sequence and let's suppose there's a law, a genuine law that tells us how this cube moves  around.  For example, if it's red, now we've done this over and over many times in different contexts  but it is so important that I feel a need to emphasize it again.  This is what a law of motion is.  It's a rule telling you what the next configuration will be given, it's a rule of updating, of  updating configurations.  Red goes to blue, blue goes to yellow, yellow goes to green, green goes to orange, orange  goes to purple and purple goes back to red.  Given the configuration at any time, you know what it will be next and you know what it  will continue to do.  Of course, you may not know the law.  Maybe all you know is that there is a law of this type.  You know what I'm going to do next, what am I going to do next?  I'm going to draw this law as a diagram.  You've all seen me do this in other contexts.  Let's do it.  We have red, too hard to draw squares.  Red, blue, green, orange, what happened?  Yellow, yellow, orange, purple.  A law like this can be just represented by a set of lines connecting a set of arrows.  Red goes to blue, blue goes to green, green goes to yellow, yellow goes to orange, orange  goes to purple, purple goes back to red.  Given the assumption now that there's a discrete time interval between such events, I am not  assuming that the cube has any symmetry to it anymore.  The cube may not be symmetric at all.  It may have points, you know, one edge, one face, maybe tiny, another face, but if this  is the rule to go from one configuration to another and each step takes, let's say, a  microsecond, I might have no idea where I begin, but I can still tell you if I, let's  say it's a microsecond, a microsecond, and my job is to catch it at a particular instant  and ask what the color is.  I don't know where it started, okay?  But I can still tell you the probability for each one of these is one-sixth.  It doesn't have to do with symmetry.  Well, maybe it does have to do with some symmetry, but in this case, it wouldn't be the symmetry  of the structure of a die.  It would just be the fact that as it passes through these sequence of states, it spends  one-sixth of its time red, one-sixth of its time blue, one-sixth of a time green, and if  I don't know where it starts and I just take a flash, you know, a flash shot of it, my  probability will be one-sixth.  Now that one-sixth did not really depend on knowing the detailed law.  For example, the law could have been different.  Let's make up a new law.  Red goes to green, green goes to orange, orange goes to yellow, yellow goes to purple,  purple goes to blue, and blue goes back to red.  This shares with the previous law that there's a closed cycle of events in which you pass  through each color once before you cycle around.  You may not know which law of nature is for this system, but you can tell me again that  the probability will be one-sixth for each one of them.  So this prediction of one-sixth doesn't depend on knowing the starting point and doesn't  depend on knowing the law of physics.  It's just important to know that there is a particular kind of law.  Are there possible laws for the system which will not give you one-sixth?  Yes.  Let's write another law.  Red, blue, green, yellow, orange, purple.  This rule says that if you start with red, you go to blue.  If you start with blue, you go to green, and if you get to green, you go back to red.  Or if you start with purple, you go to yellow, yellow to orange, orange back to purple.  Notice in this case, if you're on one of these two cycles, you stay there forever.  If you knew you were on the upper cycle, if you knew you would start it, it doesn't matter  where you start, but if you knew that you started on the upper cycle somewhere, then  you would know that there was a one-third probability to be red, a one-third probability  to be blue, and a one-third probability to be green, and zero probability to be red,  yellow, or orange.  On the other hand, you could have started with the second cycle.  You could have started with purple.  Might not have known where you started, but you knew that you started in the lower triangle  and the lower cycle here.  Then you would know the probabilities of one-third for each of these and zero for each of these.  Now, what about a more general case?  The more general case might be that you know with some probability that you start on the  upper triangle here and with some other probability on the lower triangle.  In fact, let's give these triangles names.  Let's call this triangle the plus one triangle, and this one the minus one triangle.  Just giving them names, attaching to them a number, a numerical value.  If you hear something or other is called plus one, if you hear something or other is called  minus one.  All right, now you'll have to append, you'll have to start with something you've got to  get from someplace else.  It doesn't follow from symmetry, and it doesn't follow from cycling through the system some  probability that you're either on cycle plus one or cycle minus one.  Where might that come from?  Flipping somebody else's coin over here, flipping a coin over here might decide which of these  two.  It might be a biased coin, so you will have a probability to be plus one and a probability  to be minus one.  These two probabilities are not probabilities for individual colors, they're probabilities  for individual cycles.  Okay, now what's the probability for blue?  The probability for blue begins with the probability that you're on the first cycle, times the  probability that if you're on the first cycle, you get blue.  That's one third.  So the probability for blue, red, or green is one third the probability that you're on  the first cycle, and likewise the probability that you're at yellow will be, this is the  probability for red, blue, or green in this case, and this times one third will be the  probability for purple, yellow, or orange.  Okay, so in this case you need to supply another probability that you've got to get from somebody's  else.  This case here is what we call having a conservation law.  In this case, the conservation law would be just the conservation of this number.  For red, blue, and green, we've assigned the value plus one.  That plus one could be the energy, or it could be something else.  I tend to call it the zilch for some reason.  I call everything a zilch if there is no name for it.  So anyway, let's think of it as the energy to keep things familiar.  The energy of these three configurations might all be plus one, and the energy of these three  configurations might all be minus one.  And the point is that because the rule keeps you always on the same cycle, that quantity,  energy, zilch, whatever we call it, is conserved.  It doesn't change.  That's what a conservation law is.  A conservation law is that the configuration space, the space of possibilities, divides  up into cycles like this.  Now, the cycles don't have to have equal size.  Here's another case.  One, two, three, four.  You go around this way, and then the two guys over here go into each other.  So red goes to blue, goes to green, goes to purple.  That's the upper cycle here, and the lower cycle is yellow goes to orange, goes to yellow  goes to orange.  Still, we have a conservation law here.  It's just the number of states with one value of the conserved quantities, not the same  as the number of states or the other value, but still, it's a conservation law.  And again, somebody would have to supply for you some idea of the relative probabilities  of these two.  Where that comes from is part of the study of statistical mechanics.  And the other part of the study has to do with saying, if I know I'm one of these tracks,  how much time do I spend with each particular configuration?  That's what determines probabilities of statistical mechanics.  Some a priori probability from somewheres that tells you the probabilities for different  conserved quantities and cycling through the system.  Yeah, question?  No, okay.  So, so far, you're assuming that within any conservation arena, you will, the probabilities  of all the states are the same?  The time spent in each state is the same.  Right.  So, it's completely deterministic.  Laws are completely deterministic.  This would be classical physics.  Laws completely deterministic.  No real ambiguity of what the state is, except you're kind of lazy.  You didn't determine the initial condition.  Your timing wasn't very good.  Each state only lasts for a microsecond.  You're a lazy guy and you only have a resolution of a millisecond.  But nevertheless, you're able to take a very quick flash picture and pick out one of  the states.  That's the circumstance that we're talking about.  Yeah.  If we take two pictures, is it reasonable to then assume that if the first picture indicated  that we are in one cycle, the later one should indicate the same cycle since it couldn't  get out of it?  Yes, that's a good assumption.  Yes.  Right.  So, once you determine the value of some conserved quantities, then you know it.  And then you can reset the probabilities for it.  Unless, all right, so let's talk about honest energy for a minute.  Yes, if we have a closed system, to represent the closed system, I will just draw a box.  Closed now, closed means that it's not in interaction with anything else and therefore  can be thought of as a whole universe unto itself.  Okay.  It has an energy.  The energy is some function of the state of the system, whatever determines the state  of the system.  Now let's suppose we have another closed system which is built out of two identical or not  the identical versions of the same thing.  Now, if they're both closed systems, there will be two conserved quantities.  The energy of this system and the energy of this system and they'll both be separately  conserved.  Why?  Because they don't talk to each other.  They don't interact with each other.  The two energies are conserved and you could have probabilities for each of those individuals.  But now supposing they're connected.  They're connected by a little tiny tube which allows energy to flow back and forth.  Then there's only one conserved quantity, the total energy, and it's sort of split between  the two of them.  You can then ask, what is the probability given a total amount of energy?  You could ask, what's the probability that the energy of one subsystem is one thing and  the energy of the other subsystem is the other?  If the two boxes are equal, you would expect on the average they have equal energy, but  you can still ask, what's the probability for a given energy in this box given some  overall piece of information?  That's a circumstance where it may be that giving the probability for which cycle you're  on, now which cycle you're on, I'm talking about the cycle of one of these systems here,  may be determined by thinking about the system as part of a bigger system.  And we're going to do that.  That's important.  But in general, you need some other ingredient besides just cycling around through the system  here to tell you the relative probabilities of conserved quantities.  Okay.  So we're often flying with statistical mechanics.  There are bad laws.  By bad laws, I don't, not in the sense of DOMA or any of those kind of laws, but in  the sense that the rules of physics don't allow them.  You all know what they are.  The laws that violate the conservation of information.  The most primitive and basic rule of physics, the conservation of information.  Conservation of information is not a standard conservation law like this.  It's the rule that you keep, that you can keep track.  You can keep track both going forward and backward.  So let's just mention that again.  It's all work, it's all described in the classical mechanics book.  I'm just reviewing it now.  But let's take a bad law.  It's a possible law.  By bad, I mean one, two, three, four, one, two, three, four, five, six.  So these are the faces of a die again.  But the rule is wherever you are, this is red, wherever you are, you go to red.  Even if you are red, you go to red.  Okay.  We'll discuss in a moment what's wrong with this law.  But this law has one of the features that it has, is it's not reversible.  It's not reversible in the sense that you can go from blue to red, but you cannot go  from red back to blue.  So in that sense, it's not reversible.  You can predict the future wherever you are.  The future is very simple for this particular law, wherever you are, you'll next be at red.  You can make it more complicated.  You could make a few, you can make it more complicated.  But this law always winds up with red.  It's a bad law because it loses track of where you started.  Whereas these laws don't lose track.  If you know that you've gone through 56, 56 and a half cycles, then you know that if  you started at red, you'll come back and you can tell exactly where you'll be.  And you can also tell where you came from.  You can tell not only where you'll be, but exactly where you came from.  Well, this law, you can't say where you came from.  This is a law that loses information.  And it's exactly the kind of thing that classical physics does not allow.  Classical physics also doesn't allow the quantum mechanical version of it.  So the rule that this type of rule, that this type of law is unallowed, I give a name to.  It's as I said many times, there is no name for it because it's just so basically primitive  that everybody always forgets about it.  It's so basic.  I call it the minus first law of physics.  And I wish it would catch on.  People should start using it.  I mean, it is really the most basic law of physics that information is never lost, that  distinctions or differences between states propagate with time and you never lose track.  In principle, if you have the capacity to follow the system, because you may be too lazy  to follow the system, that's your problem.  But nature doesn't have that problem.  Nature allows, in principle, that you can reconstruct where you came from.  All right, so that's a bad law.  How do you tell the good laws from the bad laws?  Just by diagrammatics here, it's very simple.  Good laws, every state has one incoming arrow and one outgoing arrow.  An arrow to tell you where you came from and an arrow to tell you where you're going.  So those are good laws.  In classical mechanics, continuum classical mechanics, there is a version of this same  law.  Anybody know the name of that version?  Of the theorem that goes with the conservation of information?  It's called Leaville's theorem.  We studied it in classical mechanics.  But let me give you a counter example to Leaville's theorem.  Friction is an apparent contradiction.  Wherever you start, you come to rest.  It's sort of like saying wherever you start, you come to red.  Wherever you start, you come to rest.  Well, you may not know exactly where you are, but you always come to rest.  That seems like a violation of the laws that tell you that distinctions have to be preserved.  But of course, it's not really true.  What's really going on is that when you run the eraser through here, it's heating up the  surface here.  And if you could keep track of every molecule, you would find out that the distinctions between  starting points is recorded.  But let's imagine now that there was a fundamental law of physics.  By fundamental law, I mean a rock bottom fundamental law for a series of particles, for a collection  of particles, and the equations of motion for the particles were this.  d second x, that's the position of the particle, somewhere by dt squared, that's called acceleration.  We could put a mass in, but the mass is not doing anything.  There's a lot of particles, so I'll label them i.  Oh, we've used i to label states.  I should not do that.  Let's call it n, little n.  The nth particle, and what is that equal to?  It's equal to minus some number gamma, we've seen that number before in another context,  times dxn by dt.  Anybody remember what this formula represents?  Friction.  Viscous drag.  Again, it has the property that if you start with a moving particle, it will very quickly  come almost to rest.  It'll exponentially come to rest pretty quickly.  And so if all particles in a gas, for example, satisfied this law of physics, it's perfectly  deterministic.  It tells you what happens next, but it has the unfortunate consequence that every particle  just comes to rest.  That sounds odd.  It sounds like no matter what temperature you start the room, it will quickly come to  zero temperature.  That doesn't happen.  This is a perfectly good differential equation, but there's something wrong with it from the  point of view of conservation of energy.  There's something wrong with it from the point of view of thermodynamics.  If you start a closed system and you start it running, you start with a lot of kinetic  energy, temperature we usually call it.  It doesn't run to zero temperature.  That's not what happens.  In fact, this is not only a violation of energy conservation.  It looks like a violation of the second law of thermodynamics.  It says things get simpler.  You start with a random bunch of particles moving in random directions, and you let it  run and they all come to rest.  What you end up with is simpler and requires less information to describe than what you  started with.  That's very, very much like everything going to red.  Among other things, it violates the second law of thermodynamics, which generally says  things get worse.  Things get more complicated, not less complicated.  Okay.  But there's another way to say, another important way to say this rule, that every state has  to have one arrow in and one arrow out.  The thing that I called either the minus first law or the conservation of information.  Supposing we have a collection of states and we assign to them probabilities.  P of state one, P of state two, P of state three, and so forth.  For some subset of the states, not all of them, some subset of them.  All the others, we say have probability zero.  Okay.  So, for example, we can take our die and assign red, yellow, and blue probability a third,  and green, orange, and pink, or whatever it was, probability zero.  Where we got that from, doesn't matter.  We got it from somewhere.  Somebody secretly told us in our ear, it's either red, yellow, or blue, and I'm not going  to tell you which.  All right.  And now you follow the system.  You follow it as it evolves.  Whatever kind of law of physics, as long as it's an allowable law of physics, after a  while, and you're following it in detail, you're not constrained by your laziness in  this case.  You are capable of following in detail.  And what is the probability, what are the probabilities at a later time?  Well, if you don't know which the laws of physics are, you can't say, of course.  But you can say one thing.  You can say there are three states with probability one-third and three states with probability  zero.  They may get reshuffled, which ones were probable and which ones were improbable, but after  a certain time, there will be those same three, not the same three states, but there will  continue to be three states which have probability and the rest don't.  So in general, you could characterize these information-conserving theories by saying,  supposing you assign some subset of the states, let's say, N out of N states, let's say there  are N states altogether, that's the total number of states, and now we look at some  M where M is less than N, and we say for those M states, the probability for those M states  is one over M for these states and zero for all the others.  You understand why I say one over M?  If there are M states equally probable, then each one has probability one over M and all  the remaining have probability zero.  Then the number of states which have non-zero probability will remain constant and the probabilities  will remain equal to one over M. Is that clear?  Is that obvious?  That should be obvious.  The states may reshuffle, but the number with non-zero probability will remain fixed.  That's a characterization, a different characterization of the information-conserving laws.  For the information-non-conserving laws, everybody goes to red.  You may start with a probability distribution that's one over five for red, green, purple,  orange, and yellow, and then a little bit later, there's only one state that has a probability  and that's red.  This is another way to describe information conservation.  We can quantify that.  We can quantify that by saying let M be the number of states which all have equal, under  the assumption that they all have equal probability, let M be the number, let's give it a name,  occupied states, states which have non-zero probability with equal probability, and then  M, what is M characterizing?  M is characterizing your ignorance.  The bigger M is, if M is equal to N, that means equal probability for everything.  Maximum ignorance.  If M is equal to one-half N, that means you know that the system is in one out of half  the states.  You're still pretty ignorant, but you're not that ignorant.  You're less ignorant.  What's the maximum, what's the minimum amount of ignorance you can have?  That you know precisely what state it's in, in which case M is what?  M is one.  You know that it's in one particular state.  All right, so M is a measure of your ignorance.  Really M in relation to N is a measure of your ignorance.  And associated with it is the concept of entropy.  Now we come to the concept of entropy, notice entropy is coming before anything else.  Entropy is coming before temperature, it's even coming before energy.  Entropy is more fundamental in a certain sense than any of them, although in a certain  sense it's, we'll discuss entropy in a minute, but S is a logarithm of M.  Logarithm of the number of states that have an appreciable probability more or less all  equal for the specific circumstance that I talked about.  That entropy is conserved.  All that happens is the states which are occupied reshuffle, but there will always be M of them  with probability one over M.  Okay, so that's where we are.  And that's the conservation of entropy if we can follow the system in detail.  Now, of course in reality we may be again lazy, lose track of the system, and we might  have after a point lost track of the equations, and lost track of our timing device, and so  forth and so on.  Now we may wind up, we may have started with some, a lot of knowledge, and wound up with  very little knowledge.  That's because, again, not because the equations cause information to be lost, but because  we just weren't careful.  Perhaps we can't be careful, perhaps there are too many degrees of freedom to keep track  of.  So when that happens, the entropy increases, but it simply increases because our ignorance  has gone up, not because anything has really happened in the system which has, if we could  follow it, we would find that the entropy is conserved.  Okay, that's the concept of entropy in a nutshell.  We're going to expand on it.  We're going to expand on it a lot.  We're going to redefine it with a more careful definition.  But what does it measure?  It measures approximately the number of states that have a non-zero probability.  Okay.  The bigger it is, the less you know.  What's the maximum value of s?  Log in, log in.  Now of course, n could be infinite.  You might have an infinite number of states, and if you do, then there's no upper bound  to the amount of ignorance you can have.  But you know, in a world with only n states, your ignorance is bounded.  So the notion of maximum entropy is a measure of how many states there are altogether.  Now I said that entropy is deep and fundamental, and so it is, but there's also an aspect  to it which makes it in a certain sense less fundamental.  It's not just a property of a system.  It's a property of a system and your state of knowledge of the system.  It depends on two things.  It depends on characteristics of the system, and it also depends on your state of knowledge  of the system or the state of knowledge of the system.  So keep that in mind.  Okay.  Now let's talk about continuous mechanics.  Mechanics of particles moving around with continuous positions, continuous velocities.  How do we describe that?  How do we describe the space of states of a mechanical system, you know, a real mechanical  system, particles and so forth?  We describe it as points in phase space.  We learned about phase space.  Phase space consists of positions and momenta.  Momenta in simple context, momentum is mass times velocity, so roughly speaking, it's  the space of positions and velocities.  Let's draw it.  P is momentum.  It goes that way.  And this axis is a stand-in for all of the momentum degrees of freedom.  If there are 10 to the 23rd particles, there are 10 to the 23rd P's, but I can't draw more  than one of them.  Well, I could draw two of them, but then I wouldn't have any room for the Q's, for the  X's.  And horizontally, the positions of the particles, which we can call X.  X or P, doesn't matter.  All right.  A point here is a possible state of the system.  If you know a point here, you know a position and a velocity, and you can predict from that  through if you know the forces.  Okay, let's start with the analog of a probability distribution, which is zero for some set of  states and constant or the same for some other set, for some smaller set.  Well, some fraction of the states all have the same probability, and the other states  have zero probability.  We can represent that by drawing a patch in here, a subregion in the phase space, and  say in that subregion, there's equal probability that the system is at any point in here and  zero probability outside.  This is sort of a situation where you may know something, where you may know something  about the particles that they're in some subregion here.  For example, you know that all the particles in this room are in the room, right?  So that puts some boundaries on what X are.  You may know that all the particles have momentum which are within some range that confines  them to this way.  So a typical bit of knowledge about the room might be represented at least approximately  by saying that there's zero probability to be outside this region and a probability equal,  I won't say one, but equal probability to be in there.  Okay, now what happens as the system evolves?  As the system evolves, X and P change.  The equations of motion say that X and P change, if you start over here, you might go to here.  If you start nearby, you'll go to some nearby point and so forth.  And the motion of the system with time is almost like a fluid flowing in the phase space.  If you think of the points of the phase space as fluid points and let time go, the phase  space moves like a fluid.  In particular, this patch over here, let's call it the occupied patch, the occupied patch  becomes some other patch.  That other patch, after a certain amount of time, the system now is known to be in here.  After a certain amount of time, we now know that the system is in here, not in here anymore,  and that it has equal, in some sense, equal probability to be anywhere in there.  Okay?  There's a theorem that goes with this.  The theorem is called Leaville's theorem.  And what it says is that the volume in phase space, the amount of volume of this region  in the XP space, and keep in mind, the XP space may be high dimensional.  Not just two, if it were two dimensional, we would think of it as the area.  Phase space is never three dimensional.  It's always even dimensional.  It has a P for every X.  So the next more complicated system would be four dimensional.  When I speak of the volume in phase space, I mean the volume in whatever dimensionality  the phase space is.  If you follow the phase space in this manner here, Leaville's theorem, you can go back  to Leaville's theorem.  It's in the classical mechanics lecture notes.  It occupies, I think, a whole lecture, I think.  All right, you follow?  And it tells you whatever this evolves into, it evolves into something of the same volume.  In other words, roughly speaking, the same number of states.  It's the immediate analog of the discrete situation where if you start with M states  and you follow the system according to the equations of motion, you will occupy the same  number of states afterwards as you started with.  There'll be different states, but you will preserve the number of them, and the probabilities  will remain equal.  So the rule is, not the rule, the theorem says that the volume of this occupied region  will stay the same.  And a little bit better, it says that if you start with a uniform probability distribution  in here, it will be uniform in here.  So there's a very, very close analog between the discrete case and the continuous case.  And this is what prevents this kind of fundamental equation from this kind of equation, an equation  where everything comes to rest, that can't happen.  Okay, why not?  Let's see why it can't happen.  Let's just look on this blackboard and see why.  Imagine that no matter where you started, you ended up with P equals zero.  That would mean every point on here got mapped to the x-axis.  It would mean that this entire region here would get mapped to a one-dimensional region,  and one-dimensional region has zero area.  So Leeville's theorem prevents that.  What it says, in fact, is if the blob squeezes in one direction, it must expand in the other  direction.  The situation for the moving eraser is that if the phase space of the eraser gets shrunk,  it means somebody else's, some other components in the phase space, the probability distribution  is spread out.  What are the other components in this case?  It's the P's and X's of all the molecules that are in the table.  So for the case of the eraser, there's really a very high-dimensional phase space, and as  the eraser may come to rest, almost rest, so that the phase space squeezes this way,  it spreads out in the other directions, the other directions having to do with the other  hidden microscopic degrees of freedom.  Okay, so there we are with information conservation, minus first law of physics, and let's pass...  Let's not go to the zeroth law.  Let's jump the zeroth law.  We'll come back to the zeroth law.  You know what the zeroth law says?  Well, I'll tell you what it says.  We haven't defined what thermal equilibrium is, okay?  But it says whatever the hell thermal equilibrium is, if you have several systems, and system  A is in thermal equilibrium with B, and B is in thermal equilibrium with C, then A is  in thermal equilibrium with C.  We will come back to that.  Just put it out of your mind for the time being, because we haven't described what thermal  equilibrium means.  But we can now jump to the first law, minus one, zero, and first law.  And the first law is simply energy conservation.  It is simply energy conservation, nothing more.  It's really simple to write down.  That simplicity belies its power.  It is the statement that, first of all, there is a conserved quantity, and the fact that  we call that conserved quantity energy will play for the moment not such a big role right  now, but let's just say there's energy conservation.  What does that say?  That simply says DE.  Simply energy is DE by DT is equal to zero.  Now this is the law of energy conservation for a closed system.  If a system consists of more than one part in interaction with each other, then of course  any one of the parts can have a changing energy, but the sum total of all of the parts will  conserve energy.  So if a system is composed, as I drew before, of two parts with a link between them, and  this is called one and this is two, then this reads that DE1 by DT is equal to minus DE2  by DT.  I've really written DE1 by DT plus DE2 by DT is equal to zero, but then I transposed  one of them to the right hand side just to indicate, just to make graphic, that if you  lose energy on one side, you gain it on the other.  So that's the first law of thermodynamics, and that's all the first law of thermodynamics  says, it says energy conservation.  Now in this context here, there's a slightly hidden assumption.  We've assumed that if a system is composed of two parts, that the energy is the sum of  the two parts.  That's really not generally true.  If you have two systems and they interact with each other, there may be, for example,  forces between the two parts, so there might be a potential energy that's a function of  both of the coordinates.  For example, the energy of the solar system, being very naive, I'm thinking of the solar  system as two orbiting Newtonian particles.  The energy consists of the kinetic energy of one particle plus the kinetic energy of the  other particle plus a term which doesn't belong to either particle.  It belongs to both of them in a sense, and it's the potential energy of interaction between  them.  In that context, you really can't say that the energy is the sum of the energy of one  thing plus the energy of the other thing.  Energy conservation is still true, but you can't divide the system into two parts this  way.  On the other hand, there are many, many contexts where the interaction energies between systems  is negligible compared to the energy that the systems themselves have.  If we were to divide this table top up into blocks, let's think about it, divide the table  top up into blocks, how much energy is in each block?  Well, the amount of energy that's in each block is more or less proportional to the volume  of each block.  How much energy of interaction is there between the blocks?  The energy of interaction is a surface effect.  They interact with each other because their surface is touch, and typically, surface area  is small by comparison with volume.  So many, many, we'll come back to that.  We'll come back to that.  In many, many contexts, the energy of interaction between two systems is negligible compared  to the energy of either of them.  When that happens, you can say to a good approximation, the energy can just be represented as the  sum of two energies of the two parts of the system plus a teeny little thing which has  to do with their interactions.  Under those circumstances, the first law of thermodynamics, the top is always true.  The second has that little caveat that we're talking about systems where energy is strictly  additive, where you add energies.  Does everybody understand why I say you don't always add energies that sometimes energies  are not additive?  Yeah, actually, I was thinking that we have the same possible problem with the probabilities.  We assume that the outcomes were mutually exclusive.  Otherwise, the sum law doesn't work.  Yeah.  Yeah.  Okay.  So in all the contexts which we've talked about, the dye, if it's yellow, it can't be  red.  You say orange.  You say orange, orange is both yellow and red, well, we don't count that way.  Yeah.  So that's correct.  No, that's absolutely correct.  We made the assumption that what I called states, what I called states are mutually exclusive.  Absolutely.  Okay, let's come back to entropy.  We're not finished with entropy.  We've done entropy.  We've done energy.  We haven't gotten the temperature yet.  Just the temperature comes in behind entropy, and even energy comes in behind entropy.  But temperature is a highly derived quantity.  By highly derived, I mean it's a, despite the fact that it's the thing you feel with  your body, so it makes it really feel like it's something intuitive, it is a mathematically  derived concept, less primitive and less fundamental in either energy or entropy, but we'll come  to it.  Let's come back to entropy.  We define entropy, but only for certain special probability distributions.  Let's lay out on the horizontal axis, just to be schematic.  On the horizontal axis, we will put down all the different states.  Here's i equals one, here's i equals two, here's i equals three.  This axis just labels the various states, and of course, vertically, let me plot probability.  Okay.  Well, the probability, of course, is only defined on the integers here.  That's not very good.  It's some probability.  But let's, I don't want to have to draw such a complicated thing every time I want to draw  a probability distribution, let's just draw a graph.  Some probability distribution, or some probability for each position.  Now what we did was we defined entropy for a very special case, the special case being  where some subset have equal probabilities and the rest have zero probability.  For example, if our subset consists of this group over here of m, the whole group being  n, then all of these have the same probability, and their probabilities have to add up to  one, so the probability is one over n.  We just draw that by drawing a box like that.  Then we define the entropy to be the logarithm of the number of states in here.  Generally speaking, we don't have probability distributions like this.  Generally speaking, we have probability distributions which are more complicated.  In fact, they can be anything as long as they're positive and all add up to one.  So the question is how do we define entropy in a more general context where the probability  distribution looks like this?  I'm going to write down the formula, and then we're going to check that it really gives  us this answer when it should give us this answer.  For today, I'm just going to write it down and tell you this is the definition.  You'll get familiar with it, and you'll start to see why it's a good definition.  It's representing something about the probability distribution, and what it's representing is  in some average sense, the average number of states which are importantly contained  inside the probability distribution.  The narrower the probability distribution, the smaller the entropy will be.  The broader the probability distribution, the bigger the entropy will be.  I'll write it down for you now.  We'll write it down and then explore it just a little bit tonight.  S for a general probability distribution is, first of all, minus.  That's funny because this is positive, but nevertheless the formula begins with minus,  a sum, and it's a sum over all of the states, all of the possibilities.  So it's a sum over i.  There's a contribution for each place here.  The probability of i times the logarithm of the probability of i.  Do you remember?  All right, let's write something else then.  Remember that the average of f is equal to the summation over i, f of i times p of i.  This is actually the average of log p sub i.  It's the average of log p sub i.  All right, let's work this out.  Let's see what this gives.  In the special case where the probability distribution is 1 over m for m states altogether.  It has width m, and because it has width m, it must have height 1 over m because all the  probabilities have to add up to 1.  All right, so let's work this out.  Let's take the contribution for all the unoccupied states.  All the unoccupied states, p sub i is 0.  You get nothing.  But log of p over i is minus infinity.  Yeah, that's right.  Now what's the, all right, good.  So let's consider the limit of log p over p.  Or let's just say the limit of, no, not that's not right, the limit of p log p as p goes  to 0.  You know how to calculate that?  Okay, I'm going to leave it to you.  It's a little calculus exercise to calculate the limit as p goes to 0 of p log p.  It's 0.  The point is that p goes to 0 a lot faster than log p goes to infinity.  Log p as p goes to 0 goes to, is very slow.  p goes to 0 fast.  So this goes to 0.  You're absolutely right though.  That has to be, that has to be shown.  But p log p in the limit that p goes to 0 is 0.  So with that piece of knowledge, the contribution from states with very, very small probabilities,  probability will be very, very small.  And as the probability for those states goes to 0, this quantity, the contribution will  go to 0.  But what about the ones here which have significant probability?  They all have the same piece of body.  And they all have the same log piece of body.  The log piece of body is logarithm of 1 over m for all of them.  The piece of body is also 1 over m.  Not log 1 over m, but 1 over m.  So each contribution is 1 over m times log 1 over m.  How many contributions are there like that?  All right, so we multiply by m and get rid of the 1 over m there.  There's a minus sign here, I'll carry it along.  All times m because there are m such terms.  So the 1 over m cancels and we just left with log 1 over m.  What's log 1 over m?  Minus log m, right?  So that's why the minus sign was put there in the first place.  There's no miracle.  The minus sign was put there because probabilities are less than 1.  And so the logarithms of them are always negative.  So you soak up that negative with an overall negative sign and entropy is positive.  But this is exactly the same answer that we are, the original definition just s equals  log m.  Logarithm of the number of states.  But this is a definition now that makes sense even when you have a more complicated probability  distribution.  And it is a good and effective.  It's the average of log p.  For the special case where the probability distribution is constant like this, then all  of the probabilities in here are 1 over m and calculating the average of log 1 over m just  gives you this.  All right.  So this is the general definition of the entropy that's associated with a probability distribution.  And notice, entropy is associated with a probability distribution.  It's not a thing like energy which is a property of a system.  It's not a thing like momentum.  It's a thing which has to do with a specific probability distribution, probability distribution  on the space of possible states.  So that's why it's a little bit of a more obscure quantity from the point of view of,  you know, intuitive definition.  As I said, its definition has to do with both the system and your state of knowledge of  the system.  Let's do some examples.  Let's calculate some entropy for a couple of simple systems.  The first system is just going to be not a single coin but a lot of coins.  So we have capital N coins, N of them, and each one can be heads or tails, et cetera,  and so on.  As a matter of fact, we have no idea what the state of the system is.  We know nothing.  The probability distribution, in other words, is the same for all states.  Good ignorance, absolute ignorance.  What is the entropy associated with such a configuration?  There are n of these.  All the probabilities are equal.  Under the circumstance where all of the probabilities are equal, we just get to use logarithm of  the number of possible states.  The answer here is the logarithm of the number of total states.  How many states are there altogether?  Two to the N.  Right.  Two to the N.  Oh, let's, I'm sorry, I'm going to change definitions for a minute for a reason that  you'll see in a minute.  I'm going to call this little N.  Number of coins is little N.  Over here, big N stood for the total number of states.  So if I match terminologies, big N, the total number of states, is e to the 2n, no, 2 to  the N.  Sorry, I'm getting tired.  2 to the N.  2 to the N states altogether.  Two states for the first coin, two states for the second coin, and so forth.  2 to the N altogether.  And the total number of states is capital N.  What's the entropy given that we know nothing?  Two log N.  Two log N.  N log 2.  S is equal to N log 2.  That's the logarithm of 2 to the N.  All right, so here we see an example of the fact that entropy is kind of additive over  the system.  It's proportional to the number of degrees of freedom in this case.  N times log 2.  And we also discover a unit of entropy.  The unit of entropy is called a bit.  That's what a bit is, an information theory.  It's the basic unit of entropy for a system which has only two states, up or down, heads  or tails or whatever.  The entropy is proportional to the number of bits, or in this case, the number of coins,  times the logarithm of 2.  So log 2 plays a fundamental role in information theory as the unit of entropy.  It does not mean that in general that entropy is an integer multiple of log 2.  We'll see in a second, it doesn't mean that.  Okay, so that's this case over here.  Let's try another case.  Let's try another state of knowledge.  Here our state of knowledge was zilch.  We knew nothing.  This is the maximum entropy.  Logarithm of capital N, or logarithm of 2 to the little n, which is n log 2.  One bit of entropy for each coin, if you like.  Okay, let's try something else.  Let's try a state in which what we know is, oh, something else.  Supposing we know the state completely.  In other words, that's the case where m is equal to 1.  That would be the case where we know that the probability is only nonzero for one state.  Then m is 1 and s is 0, logarithm of 0.  So absolute knowledge, perfect knowledge, complete knowledge corresponds to zero entropy.  The more you know, the smaller the entropy, excuse me.  Okay, let's take an interesting case.  Let's take our heads and tails again.  And here's what we know.  We know that all of them are heads.  Let's begin with that, all of them are heads, what's the entropy then?  Zero.  Except for one of them, which is tails.  Now, supposing we know which one is tails, what's the entropy?  Zero.  But suppose we don't know which one is tails.  Equal probability for all states which contain one tail and n minus one heads.  What's the entropy?  Indeed.  Okay, so why is it log n?  How many states are there with nonzero probability?  The answer is little n.  It could be this one, it could be this one, it could be this one.  In other words, for this particular situation, capital M, the number of states that have  nonzero probability is just little n.  All the states have the same probability.  So we're in exactly this situation except that M is just equal to little n.  N possible states, this one could be tails, this one could be tails, this one could be  tails.  They all have equal probability.  So capital M is n and the entropy now is, for this situation, the entropy is equal  to logarithm of little n.  Notice that that's not an integer multiple of log two in general.  So in general, entropy is not an integer multiple of log two.  Nevertheless, log two is a good unit, it's a basic unit of entropy, it's called a bit.  Yes in this case is equal to log n.  Yes?  The two in the log two come from the fact that you only have a probability of heads or tail,  right?  Yeah.  So if you had three, it'd be log three.  Absolutely.  Absolutely.  Now, computer scientists of course like to think in terms of two for a variety of reasons.  First of all, the mathematics of it is nice, but two is the smallest number which is not  one.  It's a small signature.  Yeah, it's a small signature, not equal to one.  But it's also true that the physics that goes on inside your computer is connected with  switches which are either on or off.  So counting in units of log two is very useful.  Excuse me.  Yeah?  It's probably not important, but what is the base of the logarithm?  Yeah, the standard, this is a definition, of course.  This is definition.  The definition, okay, good.  The, good.  It depends on who you are.  If you're an information theorist, a computer scientist often, or a disciple of Shannon,  then you like log to the base two.  In which case, this is just one, n, and the entropy here is just n, measured in units  of bits.  If you're a physicist, then you usually work in base E. Okay?  But the relationship is just a multiplicative, you know, log to the base E and log to the  base two are just related by a numerical factor that's always the same.  Okay.  So, when I write log log, I mean log to the base E, but very little would change if we  use some other base for the logs.  Okay.  So there we are.  Let me just tell you what the definition is of the entropy in phase space.  If we're not talking about, if we're not talking about these finite discrete systems like this,  we're talking about continuously infinite systems, phase space.  Now let's begin supposing the probability distribution is just some blob where the probabilities  are equal inside the blob and zero outside the blob.  In other words, the simple situation.  Then the definition of the entropy is simply, well, you could say the logarithm of the  number of states, but how many states are there in here?  Clearly a continuous infinity of them.  So instead, you just say it's the log of the volume of the probability distribution in  phase space.  S for a continuous system is just defined to be the logarithm of the volume in phase  space.  Now if I wrote V, you might get the sense that I'm talking about volume in space.  No, I'm talking about the logarithm in phase space, the volume in the phase space.  The phase space is high dimensional.  Another dimensionality of the phase space is the volume is measured in those kind of  in units of momentum times velocity to the power of the number of coordinates in the  system.  All right, but S is equal to the logarithm of the phase space volume.  Let's call it P phase, V phase space.  That's under the volume of the region which is occupied and has a non-zero probability  distribution.  This is the closest analog that we can think of to log M, where M represents the number  of equally probable states of the discrete system.  More generally, if we want, if we have some arbitrary probability distribution, the arbitrary  probability distribution, P, what would P probability?  What would it be a function of?  It would be a function of all of the coordinates and all of the momenta.  All of the momenta and all of the coordinates.  P's and Q's or P's and X's, whichever you like, all of them.  Probability for the system to be located at point of phase space P and X.  Incidentally, when you have continuous variables like that, do you write that the sum is all  equal to one?  That wouldn't make sense.  You can't sum over a continuously infinite set of variables.  It becomes integral.  We'll come back to this, but let's just spell it out right now.  That if you have a probability distribution on a phase space, the rule is that the integral  of it is equal one.  P is really a probability density on phase space.  It's a probability per unit cell in phase space.  What would you expect the entropy to be?  I'll give you a hint that starts out with minus.  Where's the formula that we had here before?  Let's rewrite the formula that we had before.  S is equal to minus summation over I of P sub I log of P sub I.  To go to the continuum, you simply replace sum by integral.  There's an integral over the phase space.  That's like the sum over I.  Then probability of P and Q times the logarithm of the probability.  In first approximation, I don't mean first approximation numerically.  I mean first conceptual approximation.  It's measuring the logarithm of the volume of the probability blob in phase space.  Okay, oh, I could know mistake.  In classical mechanics, X's and Q's are coordinates and P's are momenta.  You know that.  Okay, so we've now defined entropy, which, as we've seen, depends on a probability distribution.  I'm going to go one more step tonight and define temperature.  We could stop here.  I think that's probably enough for one night.  Next time we'll do temperature and then discuss the Boltzmann distribution, which is the probability  distribution for thermal equilibrium.  We haven't quite defined thermal equilibrium yet, but we will.  You can ask some questions.  I don't mind some questions now.  I just, I just, I have the feeling that I've probably done enough for one night.  Oh.  Well, I haven't made any such assumption.  I haven't made any such assumption.  My only assumption in calculating the various entropies was either that you know nothing  or I told you what you know.  How you got to know that and what the reasoning was and whether it had to do with knowing  something about the dynamics and the independence and so forth may come into your calculation  of what P is.  But in saying you know nothing, the implication was that all states are equally probable.  Without asking how you knew that.  To say that all states are equally probable is closely related to saying that there are  no correlations.  It does say that if you, all right, let's, good.  Let's talk about correlations for a moment.  To say that you know nothing means you know nothing.  So in particular, if you, you, you begin knowing nothing and you measure one of the coins, what  do you know about the other ones?  Nothing.  Nothing.  You started knowing nothing about them.  You measured one of them.  You still know nothing about the other ones.  Of course, you know about the one that you've measured.  Now let's take the other, the other case that we studied.  Anything.  We know that all coins are heads except for one which is tails.  And we now measure one of them and we find that it's tails.  What can we say about some other one?  It's surely heads.  What if we measure that that one is heads?  What do we know about the other ones?  It changes the probability.  It changes the probability.  Just a new probability that one of the other ones is tails.  Right.  It's one over n, one over n minus one instead of one over, sorry, one over, yeah, one over  n minus one.  So that's correlation.  That's correlation where when you measure something, you learn something new or the  probability distribution for the other things is modified by measuring something.  That's called correlation.  For the complete ignorance, there is no correlation.  For any other kind of configuration in general, there's very likely to be some, not necessarily,  but there's very likely to be some correlation.  Correlation as I said means you learn something about the probability distribution of other  things by measuring the first one.  Or you modify the probability distribution.  Good.  Okay.  I don't know if that's what you asked about or not, but yeah.  Okay.  All right.  Well, is that original system there?  We had two parts.  Once we measured one thing, we had to conserve quantity.  What's that again?  We had to conserve quantity there that once we did one measurement, we knew which side  we were.  Yeah.  Yeah.  Incidentally, entropy is additive.  It's additive.  It's the sum of the entropies of all the individuals.  It's proportional to a number of things.  It's additive whenever there is no correlation.  When there's no correlation, it's additive.  Now, so uncorrelated systems have additive entropies.  We'll come back.  That's a theme that we'll come back to.  Is an interesting question.  Here's what you know, you know that if you measure a coin up, then with three-quarter  probability, it's too, they're laid out in a row.  If you measure one of them and it's up, then the probability for its neighbors is three-quarters  to be down.  That's what you're given.  That's all you know.  All you know is that if one of them is up, if any one of them is up, its neighbors are  three-quarters likely to be down.  It's an interesting thing to try to calculate the entropy of such a distribution.  That is correlated, of course.  That is correlated because when you measure one, you immediately know something about  its neighbor.  Well, make up your own example like that.  Make up your own example like that and compute the entropy.  You can learn something from it.  Any other questions before we go home?  Not particularly.  The formula here is due to Boltzmann.  I think it's just the one that's on Boltzmann's tomb, I'm not sure.  Now what's on Boltzmann's tomb?  He meant this.  Well he did write this.  This is Boltzmann's final formula for entropy.  The only difference between the Shannon entropy and the Boltzmann entropy is that Shannon  used log two.  Now, of course, Shannon discovered this entirely by himself.  He didn't know Boltzmann's work from an entirely different direction from information theory  rather than thermodynamics, but none of it would have surprised Boltzmann.  Nor do I think Boltzmann's definition would have surprised Shannon.  So they're really the same thing.  There's no real point in comparing them because they're comparing them, they are the same.  There's no real difference.  Shannon may have, I don't know whether he did or not put the minus sign in here.  If you don't put the minus sign in there, it's called information.  If you put the minus sign, it's called lack of information or entropy.  So I don't know which Shannon wrote down.  Anyway.  Shannon wrote down entropy.  He did.  Yeah.  Is there a simple way to relate to Heisenberg's uncertainty principle?  No, no, this is a separate issue.  This doesn't have to do with quantum mechanical uncertainty.  It has to do with the uncertainty implicit in mixed states, not the uncertainty implicit  in pure states.  Okay.  Oh, oh boy.  Yeah, there's a conversion factor.  You know, C equals H bar equals G equals Boltzmann's constant equals one.  Very natural.  Boltzmann's constant was a conversion factor from temperature to energy.  The natural unit for temperature is really energy.  But the energy of a molecule, for example, is approximately equal to its temperature  in certain units.  Those units contain a conversion factor K Boltzmann.  I'll remind me to talk about K Boltzmann.  For more, please visit us at stanford.edu.
StatisticalMechanics_Lecture1_720p__flash9_1_1_0_299.png StatisticalMechanics_Lecture1_720p__flash9_1_1_300_671.png StatisticalMechanics_Lecture1_720p__flash9_1_1_672_719.png StatisticalMechanics_Lecture1_720p__flash9_1_1_720_1799.png StatisticalMechanics_Lecture1_720p__flash9_1_1_1800_2099.png StatisticalMechanics_Lecture1_720p__flash9_1_1_2100_2243.png StatisticalMechanics_Lecture1_720p__flash9_1_1_2244_2315.png StatisticalMechanics_Lecture1_720p__flash9_1_1_2316_2675.png StatisticalMechanics_Lecture1_720p__flash9_1_1_2676_3239.png StatisticalMechanics_Lecture1_720p__flash9_1_1_3240_3335.png StatisticalMechanics_Lecture1_720p__flash9_1_1_3336_4331.png StatisticalMechanics_Lecture1_720p__flash9_1_1_4332_5231.png StatisticalMechanics_Lecture1_720p__flash9_1_1_5232_5279.png StatisticalMechanics_Lecture1_720p__flash9_1_1_5280_5819.png StatisticalMechanics_Lecture1_720p__flash9_1_1_5820_6155.png StatisticalMechanics_Lecture1_720p__flash9_1_1_6156_6767.png StatisticalMechanics_Lecture1_720p__flash9_1_1_6768_7511.png StatisticalMechanics_Lecture1_720p__flash9_1_1_7512_8243.png StatisticalMechanics_Lecture1_720p__flash9_1_1_8244_8651.png StatisticalMechanics_Lecture1_720p__flash9_1_1_8652_8783.png StatisticalMechanics_Lecture1_720p__flash9_1_1_8784_9887.png StatisticalMechanics_Lecture1_720p__flash9_1_1_9888_10619.png StatisticalMechanics_Lecture1_720p__flash9_1_1_10620_10907.png StatisticalMechanics_Lecture1_720p__flash9_1_1_10908_10979.png StatisticalMechanics_Lecture1_720p__flash9_1_1_10980_11303.png StatisticalMechanics_Lecture1_720p__flash9_1_1_11304_11555.png StatisticalMechanics_Lecture1_720p__flash9_1_1_11556_11819.png StatisticalMechanics_Lecture1_720p__flash9_1_1_11820_13199.png StatisticalMechanics_Lecture1_720p__flash9_1_1_13200_13571.png StatisticalMechanics_Lecture1_720p__flash9_1_1_13572_13727.png StatisticalMechanics_Lecture1_720p__flash9_1_1_13728_13907.png StatisticalMechanics_Lecture1_720p__flash9_1_1_13908_15971.png StatisticalMechanics_Lecture1_720p__flash9_1_1_15972_16955.png StatisticalMechanics_Lecture1_720p__flash9_1_1_16956_17735.png StatisticalMechanics_Lecture1_720p__flash9_1_1_17736_18755.png StatisticalMechanics_Lecture1_720p__flash9_1_1_18756_18947.png StatisticalMechanics_Lecture1_720p__flash9_1_1_18948_20171.png StatisticalMechanics_Lecture1_720p__flash9_1_1_20172_22307.png StatisticalMechanics_Lecture1_720p__flash9_1_1_22308_22667.png StatisticalMechanics_Lecture1_720p__flash9_1_1_22668_22859.png StatisticalMechanics_Lecture1_720p__flash9_1_1_22860_23447.png StatisticalMechanics_Lecture1_720p__flash9_1_1_23448_25127.png StatisticalMechanics_Lecture1_720p__flash9_1_1_25128_25775.png StatisticalMechanics_Lecture1_720p__flash9_1_1_25776_26351.png StatisticalMechanics_Lecture1_720p__flash9_1_1_26352_26723.png StatisticalMechanics_Lecture1_720p__flash9_1_1_26724_28103.png StatisticalMechanics_Lecture1_720p__flash9_1_1_28104_29375.png StatisticalMechanics_Lecture1_720p__flash9_1_1_29376_29927.png StatisticalMechanics_Lecture1_720p__flash9_1_1_29928_32807.png StatisticalMechanics_Lecture1_720p__flash9_1_1_32808_33563.png StatisticalMechanics_Lecture1_720p__flash9_1_1_33564_34595.png StatisticalMechanics_Lecture1_720p__flash9_1_1_34596_36743.png StatisticalMechanics_Lecture1_720p__flash9_1_1_36744_36827.png StatisticalMechanics_Lecture1_720p__flash9_1_1_36828_37451.png StatisticalMechanics_Lecture1_720p__flash9_1_1_37452_38471.png StatisticalMechanics_Lecture1_720p__flash9_1_1_38472_39203.png StatisticalMechanics_Lecture1_720p__flash9_1_1_39204_39827.png StatisticalMechanics_Lecture1_720p__flash9_1_1_39828_41675.png StatisticalMechanics_Lecture1_720p__flash9_1_1_41676_41951.png StatisticalMechanics_Lecture1_720p__flash9_1_1_41952_44183.png StatisticalMechanics_Lecture1_720p__flash9_1_1_44184_44495.png StatisticalMechanics_Lecture1_720p__flash9_1_1_44496_44639.png StatisticalMechanics_Lecture1_720p__flash9_1_1_44640_44915.png StatisticalMechanics_Lecture1_720p__flash9_1_1_44916_46595.png StatisticalMechanics_Lecture1_720p__flash9_1_1_46596_48827.png StatisticalMechanics_Lecture1_720p__flash9_1_1_48828_53843.png StatisticalMechanics_Lecture1_720p__flash9_1_1_53844_54623.png StatisticalMechanics_Lecture1_720p__flash9_1_1_54624_55307.png StatisticalMechanics_Lecture1_720p__flash9_1_1_55308_56675.png StatisticalMechanics_Lecture1_720p__flash9_1_1_56676_57083.png StatisticalMechanics_Lecture1_720p__flash9_1_1_57084_57251.png StatisticalMechanics_Lecture1_720p__flash9_1_1_57252_57947.png StatisticalMechanics_Lecture1_720p__flash9_1_1_57948_59219.png StatisticalMechanics_Lecture1_720p__flash9_1_1_59220_60299.png StatisticalMechanics_Lecture1_720p__flash9_1_1_60300_62699.png StatisticalMechanics_Lecture1_720p__flash9_1_1_62700_63095.png StatisticalMechanics_Lecture1_720p__flash9_1_1_63096_64043.png StatisticalMechanics_Lecture1_720p__flash9_1_1_64044_64691.png StatisticalMechanics_Lecture1_720p__flash9_1_1_64692_65579.png StatisticalMechanics_Lecture1_720p__flash9_1_1_65580_66167.png StatisticalMechanics_Lecture1_720p__flash9_1_1_66168_66659.png StatisticalMechanics_Lecture1_720p__flash9_1_1_66660_67703.png StatisticalMechanics_Lecture1_720p__flash9_1_1_67704_68075.png StatisticalMechanics_Lecture1_720p__flash9_1_1_68076_69035.png StatisticalMechanics_Lecture1_720p__flash9_1_1_69036_71579.png StatisticalMechanics_Lecture1_720p__flash9_1_1_71580_74015.png StatisticalMechanics_Lecture1_720p__flash9_1_1_74016_74723.png StatisticalMechanics_Lecture1_720p__flash9_1_1_74724_76367.png StatisticalMechanics_Lecture1_720p__flash9_1_1_76368_76871.png StatisticalMechanics_Lecture1_720p__flash9_1_1_76872_79151.png StatisticalMechanics_Lecture1_720p__flash9_1_1_79152_79331.png StatisticalMechanics_Lecture1_720p__flash9_1_1_79332_80339.png StatisticalMechanics_Lecture1_720p__flash9_1_1_80340_81263.png StatisticalMechanics_Lecture1_720p__flash9_1_1_81264_82955.png StatisticalMechanics_Lecture1_720p__flash9_1_1_82956_83159.png StatisticalMechanics_Lecture1_720p__flash9_1_1_83160_85643.png StatisticalMechanics_Lecture1_720p__flash9_1_1_85644_91019.png StatisticalMechanics_Lecture1_720p__flash9_1_1_91020_91259.png StatisticalMechanics_Lecture1_720p__flash9_1_1_91260_94139.png StatisticalMechanics_Lecture1_720p__flash9_1_1_94140_94535.png StatisticalMechanics_Lecture1_720p__flash9_1_1_94536_94775.png StatisticalMechanics_Lecture1_720p__flash9_1_1_94776_94895.png StatisticalMechanics_Lecture1_720p__flash9_1_1_94896_95255.png StatisticalMechanics_Lecture1_720p__flash9_1_1_95256_95783.png StatisticalMechanics_Lecture1_720p__flash9_1_1_95784_96755.png StatisticalMechanics_Lecture1_720p__flash9_1_1_96756_96875.png StatisticalMechanics_Lecture1_720p__flash9_1_1_96876_97055.png StatisticalMechanics_Lecture1_720p__flash9_1_1_97056_97643.png StatisticalMechanics_Lecture1_720p__flash9_1_1_97644_101675.png StatisticalMechanics_Lecture1_720p__flash9_1_1_101676_102599.png StatisticalMechanics_Lecture1_720p__flash9_1_1_102600_103271.png StatisticalMechanics_Lecture1_720p__flash9_1_1_103272_104183.png StatisticalMechanics_Lecture1_720p__flash9_1_1_104184_104879.png StatisticalMechanics_Lecture1_720p__flash9_1_1_104880_105875.png StatisticalMechanics_Lecture1_720p__flash9_1_1_105876_111011.png StatisticalMechanics_Lecture1_720p__flash9_1_1_111012_115787.png StatisticalMechanics_Lecture1_720p__flash9_1_1_115788_116555.png StatisticalMechanics_Lecture1_720p__flash9_1_1_116556_118583.png StatisticalMechanics_Lecture1_720p__flash9_1_1_118584_121295.png StatisticalMechanics_Lecture1_720p__flash9_1_1_121296_121631.png StatisticalMechanics_Lecture1_720p__flash9_1_1_121632_121847.png StatisticalMechanics_Lecture1_720p__flash9_1_1_121848_123491.png StatisticalMechanics_Lecture1_720p__flash9_1_1_123492_126755.png StatisticalMechanics_Lecture1_720p__flash9_1_1_126756_127271.png StatisticalMechanics_Lecture1_720p__flash9_1_1_127272_128351.png StatisticalMechanics_Lecture1_720p__flash9_1_1_128352_129299.png StatisticalMechanics_Lecture1_720p__flash9_1_1_129300_130655.png StatisticalMechanics_Lecture1_720p__flash9_1_1_130656_131027.png StatisticalMechanics_Lecture1_720p__flash9_1_1_131028_132371.png StatisticalMechanics_Lecture1_720p__flash9_1_1_132372_133343.png StatisticalMechanics_Lecture1_720p__flash9_1_1_133344_134987.png StatisticalMechanics_Lecture1_720p__flash9_1_1_134988_137279.png StatisticalMechanics_Lecture1_720p__flash9_1_1_137280_138683.png StatisticalMechanics_Lecture1_720p__flash9_1_1_138684_138731.png StatisticalMechanics_Lecture1_720p__flash9_1_1_138732_138923.png StatisticalMechanics_Lecture1_720p__flash9_1_1_138924_140075.png StatisticalMechanics_Lecture1_720p__flash9_1_1_140076_141071.png StatisticalMechanics_Lecture1_720p__flash9_1_1_141072_149063.png StatisticalMechanics_Lecture1_720p__flash9_1_1_149064_157967.png StatisticalMechanics_Lecture1_720p__flash9_1_1_157968_160163.png StatisticalMechanics_Lecture1_720p__flash9_1_1_160164_161327.png StatisticalMechanics_Lecture1_720p__flash9_1_1_161328_161453.png