Lecture 4: Entropy and Asymptotic Equipartition Property

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK. Just to review where we are, we've been talking about source coding as one of the two major parts of digital communication. Remember, you take sources, you turn them into bits. Then you take bits and you transmit them over channels. And that sums up the whole course. This is the part where you transmit over channels. This is the part where you process the sources. We're concentrating now on the source side of things.

Partly because by concentrating on the source side of things, we will build up the machinery we need to look at the channel side of things. The channel side is really more interesting, I think. Although there's been a great deal of work on both of them. They're both important.

And we said that we could separate source coding into three pieces. If you start out with a waveform source, the typical thing to do, and almost the only thing to do, is to turn those waveforms into sequences of numbers. Those sequences might be samples. They might be numbers in an expansion. They might be whatever. But the first thing you almost always do is turn waveforms into a sequence of numbers. Because waveforms are just too complicated to deal with.

The next thing we do with a sequence of numbers is quantize them. After we quantize them we wind up with a finite set of symbols. And the next thing we do is, we take that sequence of symbols. And -- and what we do at that point is to do data compression. So that we try to represent those symbols with as small as possible a number of binary digits per source symbol. We want to do that in such a way that we can receive it the other end.

So let's review a little bit about what we've done in the last couple of lectures. We talked about the Kraft inequality. And the Kraft inequality, you remember, says that the lengths of the codewords in any prefix-free code have to satisfy this funny inequality here. And this funny inequality, in some sense, says if you want to make one codeword short, by making that one codeword short, it eats up a large part of this fraction. Since this sum has to be less than or equal to 1. If, for example, you make l sub 1 equal to 1, then that uses up a half of this sum right there. And all the other codewords have to be much longer. So what this is saying, essentially, and we proved it, and we did a bunch of things with it, and your homework you worked with it, we have shown that that inequality has to hold.

The next thing we did is, given a set of probabilities on a source, for example, p1 up to p sub m, by this time you should feel very comfortable in realizing that what you call these symbols doesn't make any difference whatsoever as far as any matter of encoding sources is concerned. The first thing you can do, if you like to, is take whatever name somebody has given to a set of symbols, replace them with your own symbols, and the easiest set of symbols to use is the integers 1 to m. And that's what we will usually do.

So, given this set of probabilities, and they have to add up to 1, the Huffman algorithm is this ingenious algorithm, very, very, clever, which constructs a prefix-free code of minimum expected length. And that minimum expected length is just defined as the sum over i, of p sub i times l sub i. And the trick in the algorithm is to find that set of l sub i's that satisfy this inequality but minimize this expected value.

Next thing we started to talk about was a discrete memoryless source. A discrete memoryless source is really a toy source. It's a toy source where you assume that each letter in the sequence is independent, and equally independent, and identically distributed. In other words, every letter is the same. Every letter is independent of every other letter. That's more appropriate for a gambling game than it is for real sources. But, on the other hand, by understanding this problem, we're starting to see that we understand the whole problem of source coding. So we'll get on with that as we go.

But, anyway, when we have a discrete memoryless source, what we found -- first we defined the entropy of such a source as h of x, which is the sum over i of minus p sub i, of logarithm of p sub i. And that was just something that came out of trying to do this optimization not the way that Huffman did it but the way that Shannon did it. Namely, Shannon looked at this optimization in terms of dealing with entropy and things like that. Huffman dealt with it in terms of finding the optimal code.

One of the very surprising things is that the way Huffman looked at it, in terms of entropy, is the way this really valuable, even though it doesn't come up with an optimal code. I mean, here was poor Huffman, who generated this beautiful algorithm, which is extraordinarily simple, which solved what looked like a hard problem. But yet, as far as information theory is concerned, he used that algorithm, sure. But as far as all the generalizations are concerned, it has almost nothing to do with anything.

But, anyway, when you look at this entropy, what comes out of it is the fact that the entropy of the source is less than or equal to the minimum number of bits per source symbol that you can come up with in a prefix-free code, which is less than h of x plus 1. And the way we did that was just to try to look at this minimization. And by looking at the minimization, we usually showed it had to be greater than or equal to H of x. And by looking at any code which satisfied this inequality with any set of length -- well, after we looked at this, this said that what we really wanted to do was to make the length of each codeword minus log of p sub i. That's not an integer. So the thing we did to get this inequality is said, OK, if it's not an integer, we'll raise it up the next value. Make it an integer. And as soon as we do that, the Kraft inequality is satisfied. And you can generate a code with that set of lengths. So that's where you get these two bounds.

This bound here is usually very, very weak. Can anybody suggest the almost unique example where this is almost tight? It's a simplistic sample you can think of. It's a binary source. And what binary source has the property that this is almost equal to this? Anybody out there?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Make it almost probability 0 and probability 1. You can't quite do that because as soon as you make the probability of the 0, 0, then you don't have to represent it. And you just know that it's a sequence of all 1's. So you're all done. And you don't need any bits to represent it. But if there's just some very tiny epsilon probability of a 0 and a big probability of a 1, then the entropy is almost equal to 0. And this 1 here is needed because l bar min is 1. You can't make it any smaller than that. So, that's where that comes from.

Let's talk about entropy just a little bit. If we have an alphabet which has size m, which is the symbol we'll usually use for the alphabet, x. And the probability that x equals i, is p sub i. In other words, again, we're using this convention so we're going to call the symbols the integers 1 to capital M. Then the entropy is equal to this. And a nice way of representing this is that the entropy is equal to the expected value of minus the logarithm. Logarithms are always to the base 2, in this course. When we want natural logarithms we'll write ln, in other words it's log to the base 2. So it's log to the base 2 of the probability of the symbol x.

We call this the log pmf random variable. We started out with x being a chance variable. I mean, we happen to have turned it into a random variable because we've given it numbers. But that's irrelevant. We really want to think of it as a chance variable. But now this quantity is a numerical function of the symbol which comes out of the source. And, therefore, this quantity is a well-defined random variable. And we call it the log pmf random variable. Some people call it self-information. We'll find out why later. I don't particularly like that word. One, because what we're dealing with here has nothing to do with information. Probably the thought that this all has something to do with information, namely, that information theory has something to do with information, probably held up the field for about five years. Because everyone tried to figure out, what does this have to do with information. And, of course, it had nothing to do with information. It really only had to do with data. And with probabilities of various things in the data. So, anyway. Some people call it self-information and we'll see why later. But this is the quantity we're interested in.

And we call it log pmf, we sort of forget about the minus sign. It's not good to forget about the minus sign, but I always do it. So I sort of expect other people to do it, too. One of the properties of entropy is, it has to be greater than or equal to 0. Why is it greater than or equal to 0? Well, because these probabilities here have to be less than or equal to 1. And the logarithm of something less than or equal to 1 is negative. And therefore minus the logarithm has to be greater than or equal to 0.

This entropy is also less than or equal to log M, log capital M. I'm not going to prove that here. It's proven in the notes, it's a trivial thing to do. Or maybe it's proven in one of the problems, I forget. But, anyway, you can do it using the inequality log of x is less than or equal to x minus 1. Just like all inequalities can be proven with that inequality.

So there's a quality here if x is equiprobable. Which is pretty clear, because if all of these probabilities are equal to 1 over M, and you take the expected value of logarithm of M, you get logarithm of M. So there's nothing very surprising here.

Now, the next thing -- and here's where what we're going to do is, on one hand very simple, and on the other hand very confusing. After you get the picture of it, it becomes very simple. Before that, it all looks rather strange. If you have two independent chance variables, say x and y, then the choice where the sample value of the chance variable x, and the choice of the sample value y, together that's a pair of sample values which we can view as one sample value. In other words, we can view xy as a chance variable all in its own right. This isn't the sequence x, followed by y, where you can think of it that way. But we want to think of this here as a chance variable itself. Which takes on different values. And the values it takes on are pairs of sample values, 1 from x, 1 from ensemble y, and xy takes on the sample value of little xy with the probability p of x times p of y.

As we move one with this course, we'll become much less careful about putting these subscripts down, which talk about random variables. And the arguments which talk about sample values of those random variables. I want to keep doing it for a while because most courses in probability, even 6.041, which is the first course in probability, almost deliberately fudges the difference between sample values and random variables. And most people who work with probability do this all the time. And you never know when you're talking about a random variable and when you're talking about a sample value of a random variable. And that's convenient for getting insight about thing. And you do it for a while and then pretty soon you wonder, what the heck is going on. Because you have no idea what's a random variable any more, and what's a sample value of it.

So, this entropy, H, of the chance variable, xy, is then expected value of minus the logarithm of the probability of the chance variable, xy. Mainly, when you take the expected value, you're taking the expected value of a random variable. And for the random variable here, in the chance variables, xy and here. This is the expected value of minus the logarithm of p of x times the probability p of x. Which is the expected value. Since they're independent of each other it's the sum. And that gives you H of x y is equal to H of x plus H of y. This is really the reason why you're interested in these chance variables which are logarithms of probabilities. Because when you have independent chance variables then you have the situation that probabilities multiply and therefore log probabilities add.

All of the major theorems in probability theory, in particular the law of large numbers, which is the most important result in probability, simple though it might be, that's the key to everything in probability. That particular result talks about sums of random variables and not about products of random variables. So that's why Shannon did everything in terms of these log PMF. And we will soon be doing everything in terms of log PMF also.

So now let's get back to discrete memoryless sources. If you now have a block of n chance variables, x1 to xn, and they're all IID, again we can do this whole block as one big monster chance variable. If each one of these takes on m possible values, then this monster chance variable can take on m to the n possible values. Namely, each possible string of symbols, each possible string of n symbols where each one is a choice from the integers 1 to capital M. So we're talking about tuples of numbers now. And those are the values that this particular chance variable, x sub n, takes on.

So it takes on these probabilities, takes on these values with the probability p of x n is the product from i equals 1 to n, of the individual probabilities. In other words, again, when you have independent chance variables, the probabilities multiply. Which is all I'm saying here.

So the chance variable x sub n has the entropy H of x n, which is the expected value of minus the logarithm of that probability. Which is the expected value of minus the logarithm of the product of probabilities, which is the expected value of the sum of minus the log of the probabilities, which is n times H of x.

If you compare this with the previous slide, you'll see I haven't said anything new. This argument and this argument are really exactly the same. All I did was do it for two chance variables first. And then observe. But it generalizes, to an arbitrary number of chance variables. You can say that it generalizes to an infinite number of chance variables also. And in some sense it does. And I would advise you not to go there. Because you just get tangled up with a lot of mathematics that you don't need.

So the next thing is, how do we fix the variable prefix-free codes and what do we gain by it? So the thing we're going to do now, instead of trying to compress one symbol at a time from the source, we're going to segment the source into blocks of n symbols each. And after we segment it into blocks of n symbols each, we're going to encode the block of n symbols. Now, what's new there? Nothing whatsoever is new. A block of n symbols is just a chance variable. We know how to do optimal encoding of chance variables. Namely, we use the Huffman algorithm. You can do that here on these n blocks. We also have this nice theorem, which says that the entropy -- well, first the entropy of x n as n times the entropy of x. So, in other words, when you have independent identically distributed chance variables, this entropy is just n times this entropy. But the important thing is this result of doing the encoding. Which is the same result we had before. Namely, this is the result of what happens when you take a set -- when you take an alphabet, and the alphabet can be anything whatsoever. And you encode that alphabet in an optimal way, according to the probabilities of each symbol within the alphabet. And the result that you get is the entropy of this big chance variable x sub n is less than or equal to the minimum -- well, it's less than or equal to the expected value of the length of a code, whatever code you have. But when you put the minimum on here, this is less than the entropy of the chance variable x super n plus 1. That's the same theorem that we proved before. There's nothing new here.

Now, if you divide this by n, this by n, and this by n, you still have a valid inequality. When you divide this by n, what do you get? You get H of x. When you divide this by n, by definition L bar -- what we mean is the number of bits per source symbol. We have n source symbols here. So when we divide by n, we get the number of bits per source symbol in this monster symbol. So l min is equal to this. When we divide this by n, we get this. When we divide this by n, we get H of x. And now the whole reason for doing this is, this silly little 1 here, which we're trying very hard to think of as being negligible or unimportant, has suddenly become 1 over n. And by making n big enough, this truly is unimportant.

If you're thinking in terms of encoding a binary source, this 1 here is very important. In other words, when you're trying to encode a binary source, if you're encoding one letter at a time, there's nothing you can do. You're absolutely stuck. Because if both of those letters have non-zero probabilities, and you want a uniquely decodable code, and you want to find codewords for each of those two symbols, the best you can do is to have an expected length of 1. Namely, you need 1 to encode 1, and you need 0 to encode 0. And there's nothing else, there's no freedom at all that you have. So you say, OK, in that situation, I really have to go to longer blocks. And when I go to longer blocks, I can get this resolved. And I know how to do it. I use Huffman's algorithm or whatever. So, suddenly, I can start to get the expected number of bits per source symbol. Down as close to h of x as I want to make it. And I can't make it any smaller. Which says that H of x now has a very clear interpretation, at least for prefix-free codes, of being the number of bits you need for encoding prefix-free codes when you allow the possibility of encoding them a block at a time.

We're going to find later today that the significance of this is far greater than that, even. Because why use prefix-free codes, we could use any old kind of code. When we study the Lempel-Ziv codes tomorrow, we'll find out they aren't prefix-free codes at all. They're really variable length of variable length codes. So they aren't fixed to variable length. And they do some pretty fancy and tricky things. But they're still limited to this same inequality. You can never beat the entropy bound. If you want something to be uniquely decodable, you're stuck with this bound. And we'll see why in a very straightforward way, later. It's a very straightforward way which I can guarantee all of you are going to look at it and say, yes, that's obvious. And tomorrow you will look at it and say, I don't understand that at all. And the next day you'll look at it again and say, well, of course. And the day after that you'll say, I don't understand that. And you'll go back and forth like that for about two weeks. Don't be frustrated, because it is simple. But at the same time it's very sophisticated.

So, let's now review the weak law of large numbers. I usually just call it the law of large numbers. I bridle a little bit when people call it weak because, in fact it's the centerpiece of probability theory. But there is this other thing called the strong law of large numbers, which mathematicians love because it lets them look at all kinds of mathematical minutiae. It's also important, I shouldn't play it down too much. And there are places where you truly need it. For what we'll be doing, we don't need it at all. And the weak law of large numbers does in fact hold in many places where the strong law doesn't hold. So if you know what the strong law, is temporarily forget it for the for the rest of the term. And just focus on the weak law. And the weak law is not terribly complicated.

We have a sequence of random variables. And each of them has a mean y bar. And each of them has a variance sigma sub y squared. And let's assume that they're independent and identically distributed for the time being. Just to avoid worrying about anything. If we look at the sum of those random variables, namely a is equal to y1 up to y sub n. Then the expected value of a is the expected value of this plus the expected valuable of y2, and so forth. So the expected value of a is n times y bar. And I think in one of the homework problems, you found the variance of a. And the variance of a, well, the easiest thing to do is to reduce this to its fluctuation. Reduce all of these to their fluctuation. Then look at the variance of the fluctuation, which is just the expected value of this squared. Which is the expected value of this squared plus the expected value of this squared, and so forth.

So the variance of a is n times sigma sub y squared. I want all of you know that. That's sort of day two of a probability course. As soon as you start talking about random variables, that's one of the key things that you do. One of the most important things you do.

The thing that we're interested in here is more the sample average of y1 up to y sub n. And the sample average, by definition, is the sum divided by n. So in other words, the thing that you're interested in here is to add all of these random variables up. Take one over n times it. Which is a thing we do all the time. I mean, we sum up a lot of events, we divide by n, and we hope by doing that to get some sort of typical value. And, usually there is some sort of typical value that arises from that. What the law of large numbers says is that there in fact is a typical value that arises. So this sample value is a over n, which is the sum divided by n. And we call that the sample average.

The mean of the sample average is just the mean of a, which is n times y bar, divided by n. So the mean of the sample average is y bar itself. The variance of the sample variance, -- the variance of the sample average, OK, that's, -- I'm talking too fast.

The sample average here, you would like to think of it as something which is known and specific, like the expected value. It, in fact, is a random variable. It changes with different sample values. It can change from almost nothing to very large quantities. And what we're interested in saying is that most of the time, it's close to the expected value. And that's what we're aiming at here. And that's what the law of large numbers says.

The sample average here, the variance of this, is now equal to the variance of a divided by n squared. In other words, we're trying to take the expected value of this quantity squared. So there's a 1 over n squared that comes in here. When you take the 1 over n squared here, this variance then becomes sigma -- I don't know why I have the n there. Just take that n out, if you will. I don't have my red pen with me. And so it's the variance of the random variable y, divided by n. In other words, the limit as n goes to infinity of the of the variance of the sum is equal to infinity. And the variance of the sample average as n goes to infinity is equal to 0. And that's because of this 1 over n squared effect here. When you add up a lot of independent random variables, what you wind up with is the sample average has a variance, which is going to 0. And the sum has a variance which is going to infinity. That's important. Aside from all of the theorems you've ever heard, this is sort of the gross, simple-minded thing which you always ought to keep foremost in your mind. This is what's happening in probability theory. When you talk about sample averages, this variance is getting small.

Let's look at a picture of this. Let's look at the distribution function of this random variable. The sample average as a random variable. And what we're finding here is that this distribution function, if we look at it for some modest value of n, we get something which looks like this upper curve here. Which is then the lower curve here. It's spread out more, so it has a larger variance. Namely, the sample average has a larger variance.

When you make n bigger, what's happening to the variance? The variance is getting smaller. The variance is getting smaller by a factor of 1/2. So this quantity is supposed to have a variance which is equal to 1/2 of the variance of this. How you find a variance in a distribution function is your problem. But you know that if something has a small variance, it's very closely tucked in around this. In other words, as the variance goes to 0, and the mean is y bar, you have a distribution function which approaches a unit step.

And all that just comes from this very, very simple argument that says, when you have a sum of IID random variables and you take the sample average of it, namely, you divide by n, the variance goes to 0. Which says, no matter how you look at it, you wind up with something that looks like a unit step.

Now, the Chebyshev inequality, which is one of the simpler inequalities in probability theory, and I don't prove it because it's something you've all seen. I don't know of any course in probability which avoids the Chebyshev inequality. And what it says is, for any epsilon greater than 0, the probability that the difference between the sample average and the true mean, the probability that that quantity and magnitude is greater than or equal to epsilon, is less than or equal to sigma sub y squared divided by n epsilon squared.

Oh, incidentally that thing that was called sigma sub n before was really sigma squared. That's mainly the variance of y. I hope it's right in the notes. Might not be. It is? Good.

So, that's what this inequality says. There's an easy way to derive this on the fly. Namely, if you're wondering what all these constants are here, here's a way to do it. What we're looking at, in this curve here, is we're trying to say, how much probability is there outside of these plus and minus epsilon limits. And the Chebyshev inequality says there can't be too much probability out here. And there can't be too much probability out here.

So, one way to get at this is to say, OK, suppose I have some given probability out here. And some given probability out here. And suppose I want to minimize the variance of a random variable which has that much probability out here and that much probability out here. How do I do it? Well, the variance deals with the square of how far you were away from the mean. So if I want to have a certain amount of probability out here, I minimize my variance by making this come straight, come up here with a little step, then go across here. Go up here. And then, oops. Go up here. I wish I had my red pencil. Does anybody have a red pen? That will write on this stuff? Yes? No? Oh, great. Thank you. I will return it.

So what we want is something which goes over here. Comes up here. Goes across here. Goes up here. Goes across here, and goes up again. That's the smallest you can make the variance. It's squeezing everything in as far as it can be squeezed in. Namely, everything out here gets squeezed in to y minus epsilon. Everything in here gets squeezed into 0. And everything out here gets squeezed into epsilon. OK, calculate the variance there. And that satisfies the Chebyshev inequality with equality.

So that's all the Chebyshev inequality is. And it's a loose inequality usually, because usually these curves look very nice. Usually this looks like a Gaussian distribution function, and the central limit theorem says that we don't need the central limit theorem here, and we don't want the central limit theorem here, because this thing is an inequality that says, life can't be any worse than this. And all the central limit theorem is, is an approximation. And then we have to worry about when it's a good approximation and when it's not a good approximation.

So this says, when we carry it one piece further, it's for any epsilon and delta greater than 0. If we make n large enough -- in other words, substitute delta for this. And then, when you make n small enough, this quantity is smaller than delta. And that says that the probability that s and y differ by more than epsilon is less than or equal to delta when we make n big enough. So it says, you can make this as small as you want. You can make this as small as you want. And all you need to do is make a sequence which is long enough.

Now, the thing which is mystifying about the law of large numbers is, you need both the epsilon and the delta. You can't get rid of either of them. In other words, you can't say -- you can't reduce this to 0. Because it won't make any sense. In other words, this curve here tends to spread out on its tails. And therefore, there's always a probability of error there. You can't move epsilon into 0 because, for no finite end, do you really get a step function here. So you need some wiggle room on both end. You need wiggle room here, and you need wiggle room here. And once you recognize that you need those two pieces of wiggle room. Namely, you cannot talk about the probability that this is equal to y bar, because that's usually 0. And you cannot talk about reducing this to 0 either. So both of those are needed.

Why did I go through all of this? Well, partly because it's important. But partly because I want to talk about something which is called the asymptotic equipartition property. And because of those long words you believe this has to be very complicated. I hope to convince you that what the asymptotic equipartition property is, is simply the week law of large numbers applied to the log pmf. Because that, in fact, is all it is. But it says some unusual and fascinating things.

So let's suppose that x1, x2, and so forth is the output from a discrete memoryless source. Look at the log pmf of each of these. Namely, they each have the same distribution function. So w of f x is going to be equal to minus the logarithm of p sub x of x, for each of these chance variables. w of x maps source symbols into real numbers. So there's a random variable, capital W of x sub j, which is a random variable. We have a random variable for each one of these symbols to come out of the source. So, for each one of these symbols, there's this log pmf random variable, which takes on different values.

So the expected value of this log pmf, for the j'th symbol out of the source is the sum of p sub x of x, namely the probability that the source takes on the value x, times minus the logarithm of p sub x. And that's just the entropy. So, the strange feeling about this log pmf random variable is its expected value is entropy. And w of x1, w of x2, and so forth, are a sequence of IID random variables, each one of them which has a mean, which is the entropy.

So it's just exactly the situation we had before. Instead of y bar, we have the entropy of x. And instead of the random variable y sub j, we have this random variable w of x sub j, which is defined by the symbol in an alphabet.

And just to review this, but it's what we said before, if capital X1, this little x1, namely, if little x1 is the sample value for the chance variable x1 and if x2 is a sample value for the chance variable X2, then the outcome for w of x1 plus w of x2 -- very hard to keep all these little letters and capital letters straight. Is w of x1 plus w of x2 is minus the logarithm of the probability of x1 minus the logarithm of the probability of x2, which is minus the logarithm of the product, which is minus the logarithm of the joint probability of x1 and x2, which is the random variable w of x1 x2. So the sum here is the random variable corresponding to log pmf of the joint outputs x1 and x2.

So w of x1 x2 is the log pmf of the event, but this joint chance variable takes on the value x1 x2. And the random variable x1 x2 is the sum of x1 and x2. So, what have I done here? I said this at the end of the last slide, and you won't believe me. So, again this is one of these things where tomorrow you won't believe me. And you'll have to go back and look at that. But, anyway, x1 x2 is a chance variable. And probabilities multiply in log pmf's add, which is what we've been saying for a couple of days now.

So. If I look at the sum of n of these random variables, the sum of these log probabilities is the sum of the log of pmf's, which is minus the logarithm of the probability of the entire sequence. That's just saying the same thing we said before, for two random variables. The sample average of a log pmf's is the sum of the w's divided by n, which is minus the logarithm of the probability divided by n. The weak law of large numbers applies, and the probability that this sample average minus the expected value of w of x is greater than or equal to epsilon is less than or equal to this quantity here.

This quantity is minus the logarithm of the probability of x sub n, divided by n, minus H of x, greater than or equal to epsilon. So this is the thing that we really want.

I'm going to spend a few slides trying to say what this means. But let's try to just look at it now, and see what it must mean. It says that with high probability, this quantity is almost the same as this quantity. It says that with high probability, the thing which comes out of the source is going to have a probability, a log probability, divided by n, which is close to the entropy. It says in some sense that with high probability, the probability of what comes out of the source is almost a constant. And that's amazing. That's what you'll wake up in the morning and say, I don't believe that. But it's true. But you have to be careful to interpret it right.

So, we're going to define the typical set. Namely, this is the typical set of x's, which come out of the source. Namely, the typical set of blocks of n symbols out of the source. And when you talk about a typical set, you want something which includes most of the probability. So what I'm going to include in this typical set is all of these things that we talked about before. Namely, we showed that the probability that this quantity is greater than or equal to epsilon is very small. So, with high probability what comes out of the source satisfies this inequality here. So I can write down the distribution function of this random variable here. It's just this w -- this is a random variable w. I'm looking at the distribution of that random variable w. And this quantity in here is the probability of this typical set. In other words, when I draw this distribution function for this combined random variable, I've defined this typical set to be all the sequences which lie between this point and this point. Namely, this is H minus epsilon. And this is H plus epsilon, moving H out here. So these are all the sequences which satisfy this inequality here. So that's what I mean by the typical set. It's all things which are clustered around H in this distribution function.

And as n approaches infinity, this typical set approaches probability 1. In the same way that the law of large numbers behaves that way. The probability that x sub n is in this typical set is greater than or equal to 1 minus sigma squared divided by n epsilon squared. Let's try to express that in a bunch of other ways.

If you're getting lost, please ask questions. But I hope to come back to this in a little bit, after we finish a little more of the story. So, another way of expressing this typical set -- let me look at that as the typical set. If I take this inequality here and I rewrite this, namely, this is the set of x's for which this is less than epsilon plus H of x and greater than H of x minus epsilon. So that's what I've expressed here. It's the set of x's for which n times H of x minus epsilon is less than this logarithm of probability is great less than n times H of x plus epsilon. Namely, I'm looking at this range of epsilon around H, which is this and this.

If I write it again, if I exponentiate all of this, it's the set of x's for which 2 to the minus n, H of x, plus epsilon, that's this term exponentiated, is less than this is less than this term exponentiated. And what's going on here is, I've taken care of the minus sign also. And if you can follow that in your head, you're a better person than I am. But, anyway, it works. And if you fiddle around with that, you'll see that that's what it is.

So this typical set is a bound on the probabilities of all these typical sequences. The typical sequences all are enclosed in this range of probabilities. So the typical elements are approximately equiprobable, in this strange sense above. Why do I say this is a strange sense? Well, as n gets large, what happens here? This is 2 to the minus n times H of x. Which is the important part of this. This epsilon here is multiplied by n. And we're trying to say, as n gets very, very big, we can make epsilon very, very small. But we really can't make n times epsilon very negligible. But the point is, the important thing here is, 2 to the minus n times H of x. So, in some sense, this is close to 2 to the minus n H of x. In what sense is it true?

Well, it's true in that sense. Where that, in fact is, a valid inequality. Namely in terms of sample averages, these things are close. When I do the exponentiation and get rid of the n and all that stuff, they aren't very close. But saying this sort of thing is sort of like saying that 10 to the minus 23 is approximately equal to 10 to the minus 25. And they're approximately equal because they're both very, very small. And that's the kind of thing that's going on here. And you're trying to distinguish 10 to the minus 23 and 10 to the minus 25 from 10 to the minus 60th and from 10 to the minus three. So that's the kind of approximations we're using. Namely, we're using approximations on a log scale, instead of approximations of ordinary numbers.

But, still it's convenient to think of these typical x's, typical sequences, as being sequences which are constrained in probability in this way. And this is the thing which is easy to work with. The atypical set of strings, namely, the compliment to this set, the thing we know about that is the entire set doesn't have much probability. Namely, if you fix epsilon and you let n get bigger and bigger, this atypical set becomes totally negligible. And you can ignore it. So let's plow ahead. Stop for an example pretty soon, but --

If I have a sequence which is in the typical set, we then know that its probability is greater than 2 to the minus n times H of x plus epsilon. That's what we said before. And, therefore, when I use this inequality, the probability of x to the n, for something in the typical set, is greater than this quantity here. In other words, this is greater than that. For everything in a typical set. So now I'm heading over things in a typical set. So I need to include the number of things in a typical set. So what I have is this sum. And what is this sum?

This is the probability of the typical set. Because I'm adding overall elements in the typical set. And it's greater than or equal to the number of elements in a typical set times these small probabilities. If I turn this around, it says that the number of elements in a typical set is less than 2 to the n times H of x plus epsilon. For any epsilon, no matter how small I want to make it. Which says that the elements in a typical set have probabilities which are about 2 to the minus n times H of x. And the number of them is approximately 2 to the n times H of x. In other words, what it says is that this typical set is a bunch of essentially uniform probabilities.

So what I've done is to take this very complicated source. And when I look at these very humongous chance variables, which are very large sequences out of the source, what I find is that there's a bunch of things which collectively have zilch probability. There's a bunch of other things which all have equal probability. And a number of them is enough to add up to y. So I have turned this source, when I look at it over a long enough sequence, into a source of equiprobable events. And each of those events has this probability here.

Now, we know how to encode equiprobable events. And that's the whole point of this. So, this is less than or equal to that. On the other side, we know that 1 minus delta is less than or equal to this probability of a typical set. And this is less than the number of elements in a typical set times 2 to the minus n h of x minus epsilon. This is an upper bound on this. This is less than this.

So I just add all these things up and I get this bound. So it says, the size of the typical set is greater than 1 minus delta, times this quantity. In other words, this is a pretty exact sort of thing. If you don't mind dealing with this 2 to the n epsilon factor here. If you agree that that's negligible in some strange sense, the all of this makes good sense. And if it is negligible, let me start talking about source coding, which is why this all works out.

So the summary is that the probability of the complement of the typical set is essentially 0. The number of elements in a typical set is approximately 2 to the n times h of x. I'm getting rid of all the deltas and epsilons here, to get sort of the broad view of what's important here. Each of the elements in a typical set has probability 2 to the minus n times H of x. So I've turned a source into a source of equiprobable elements. And there are 2 to the n times h of x of them.

Let's do an example of this. It's an example that you'll work on more in the homework and do it a little more cleanly. Let's look at a binary discrete memoryless source, where the probability that x is equal to 1 is p, which is less than 1/2. And the probability of 0 is greater than 1/2. So, this is what you get when you have a biased coin. And the biased coin has a 1 on one side and a 0 on the other side. And it's more likely to come up 0's than 1's.

I always used to wonder how to make a biased coin. And I can give you a little experiment which shows you you can make a biased coin. I mean, a biased is a little round thing which is flat on the top and bottom. Suppose instead of that you make a triangular coin. And instead of making it flat on top and bottom, you turn it into a tetrahedron. So in fact, what this is now is a coin which is built up on one side into a very massive thing. And is flat on the other side. Since it's a tetrahedron and it's an equilateral tetrahedron, the probability of 1 is going to be 1/4, and the probability of 0 is going to be 3/4. So you can make biased coins. So when you get into coin-tossing games with people, watch the coin that they're using. It probably won't be a tetrahedron, but anyway.

So the entropy here, the log pmf random variable, takes on the value of minus log p with probability p. And it takes on the value minus log 1 minus p, with probability 1 minus p. This is a probability of a 1. This is a probability of a 0. So, the entropy is equal to this.

Used to be that in information theory courses, people would almost memorize what this curve looked like. And they'd draw pictures of it. There were famous curves of this function, which looks like this. 0, 1, 1.

Turns out, that's not all that important a distribution. It's a nice example to talk about. The typical set, t epsilon n, is the set of strings with about p n1's and about 1 minus p times n 0's. In other words, that's the typical thing to happen. And it's the typical thing in terms of this law of large numbers here. Because you get 1's with probability p. And therefore in a long sequence, you're going to get about pn 1's and 1 minus p 0's.

The probability of a typical string is, if you get a string with this many 1's and this many 0's, it's probability is p. Namely, the probability of a 1 times the number of 1's you get, which is pn. Times the probability of a 0, times the number of 0's you get. And if you look at what this is, if you take p up in the exponent and 1 minus the p up in the exponent, this becomes 2 to the minus n times h of x, just like what it should be. So these typical strings, with about pn 1's and 1 minus pn 0's, are in fact typical in the sense we've been talking about.

The number of n strings with pn 1's is n factorial divided by pn factorial divided by n times 1 minus p factorial. I mean I hope you learned that a long time ago, but you should learn it in probability anyway. It's just very simple combinatorics.

So you have that many different strings. So what I'm trying to get across here is, there are a bunch of different things going on here. We can talk about the random variable which is the number of 1's that occur in this long sequence. And with high probability, the number of 1's that occur is close to pn. But if pn 1's occur, there's still an awful lot of randomness left. Because we have to worry about where those pn 1's appear. And those are the sequences we're talking about. So, there are this many sequences, all of which have that many 1's in them. And there's a similar number of sequences for all similar numbers of 1's. Namely, if you take pn plus 1 and pn plus 2, pn minus 1, pn minus 2, you get similar numbers here. So those are the typical sequences.

Now, the important thing to observe here is that you really have 2 to the n binary strings altogether. And what this result is saying is that collectively those don't make any difference. The law of large numbers says, OK, there's just a humongous number of strings. You get the largest number strings which have about half 1's and half 0's. But their probability is zilch. So the thing which is probable is getting pn 1's and 1 minus pn 0's.

Now, we have this typical set. What is the most likely sequence of all, in this experiment? How do I maximize the probability of a particular sequence? The probability of the sequence is p to the i times 1 minus p to the n minus i. And 1 minus p is the probability of 0. And p is the probability of a 1. How do I choose i to maximize this? Yeah?

AUDIENCE: [UNINTELLIGIBLE] all 0's.

PROFESSOR: You make them all 0's. So the most likely sequence is all 0's. But that's not a typical sequence. Why isn't it a typical sequence? Because we chose to define typical sequence in a different way. Namely is only one of those, and there are only n of them with only a single one. So, in other words, what's going on is that we have an enormous number of sequences which have around half 1's and half 0's. But they don't have any probability. And collectively they don't have any probability. We have a very small number of sequences which have a very large number of 0's. But there aren't enough of those to make any difference. And, therefore, the things that make a difference are these typical things which have about np 1's and 1 minus pn 0's.

And that all sounds very strange. But if I phrase this a different way, you would all say that's exactly the way you ought to do things. Because, in fact, when we look at very, very long sequences, you know with extraordinarily high probability what's going to come out of the source is something with about pn 1's and about 1 minus p times n 0's. So that's the likely set of things to have happen. And it's just that there are an enormous number of those things. There are this many of them. So, here what we're dealing with is a balance between the number of elements of a particular type, and the probability of them. And it turns out that this number and its probability balance out to say that usually what you get is about pn 1's and 1 minus p times n 0's. Which is what the law of large numbers said to begin with. All we're doing is interpreting that here. But the thing that you see from this example is, all of these things with exactly pn 1's in them, assuming that pn is an integer, are all equiprobable. They're all exactly equiprobable. So what we're doing when we're talking about this typical set, is first throwing out all the things which have to many 1's are or too few 1's in them. We're keeping only the ones which are typical in the sense that they obey the law of large numbers. And in this case, they obey the law of large numbers for log pmf's also. And then all of those things are about equally probable.

So the idea in source coding is, one of the ways to deal with source coding is, you want to assign codewords to only these typical things. Now, maybe you might want to assign codewords to something like all 0's also. Because it hardly costs anything. And a Huffman code would certainly do that. But it's not very important whether you do or not. The important thing is, you assign codewords to all of these typical sequences.

So let's go back to fixed-to-fixed length source codes. We talked a little bit about fixed-to-fixed length source codes before. Do you remember what we did with fixed-to-fixed length source codes before? We said we have an alphabet of size m. We want something which is uniquely decodable. And since we want something which is uniquely decodable, we have to provide codewords for everything. And, therefore, if we want to choose a block length of n, we've got to generate m to the n codewords.

Here we say, wow, maybe we don't have to provide codewords for everything. Maybe we're willing to tolerate a certain small probability that the whole thing fails and falls on its face. Now, does that make any sense? Well, view things the following way. We said, when we started out all of this, that we were going to look at prefix-free codes. Where some codewords had a longer length and some codewords had a shorter length. And we were thinking of encoding either single letters at a time, or a small block of letters at a time. So think of encoding, say, 10 letters at a time. And think of doing this for 10 to the 20th letters.

So you have the source here which is pumping out letters at a regular rate. You're blocking them into n letters at a time. You're encoding in a prefix-free code. Out comes something. What comes is not coming out at a regular right. What is coming out, sometimes you get a lot of bits out. Sometimes a small number of bits out. So, in other words, if you want to send things over a channel, you need a buffer there to save things. If, in fact, we decide that the expected number of bits per source letter is, say, five bits per source letter, then we expect over a very long time to be producing five bits per source letter. And if we turn our channel on for one year, to transmit all of these things, what's going to happen is this very unlikely sequence occurs. Which in fact requires not one year to transmit, but two years to transmit.

In fact, what do we do if it takes one year and five minutes to transmit instead of one year? Well, we've got a failure. Somehow or other, the network is going to fail us. I mean we all know that networks fail all the time despite what engineers say. I mean, all of us who use networks know that they do crazy things. And one of those crazy things is that unusual things sometimes happen. So, we develop this very nice theory of prefix-free codes. But prefix-free codes, in fact, fail also. And they fail also because buffers overflow. In other words, we are counting on encoding things with a certain number of bits per source symbol. And if these unusual things occur, and we have too many bits per source symbol, then we fail.

So the idea that we're trying to get at now is that prefix-free codes and fixed-to-fixed length source codes which only encode typical things. In fact, are sort of the same if you look at them over a very, very large sequence length. In other words, if you look at a prefix-free code which is dealing with blocks of 10 letters, and you look at a fixed-to-fixed length code which is only dealing with typical things but is looking at a length of 10 to the 20th, then over that length of 10 to the 20th, your variable length code is going to have a bunch of things which are about the length they ought to be. And a bunch of other things which are extraordinarily long. The bunch of things which are extraordinarily long are extraordinarily unpopular, but there are an extraordinarily large number of them. Just like with a fixed-to-fixed length code, you are going to fail. And you're going to fail on an extraordinary number of different sequences. But, collectively, that set of sequences don't have any probability.

So the point that I'm trying to get across is that, really, these two situations come together when we look very long lengths. Namely, prefix-free codes are just a way of generating codes that work for typical sequences and over a very large, long period of time, will generate about the right number of symbols. And that's what I'm trying to get at here. Or what I'm trying to get at in the next slide.

So the fixed-to-fixed length source code, I'm going to pick some epsilon and some delta. Namely, that epsilon and delta which appeared in the law of large numbers. I'm going to make n as big as I have to make it for that epsilon and that delta. And calculate how large it has to be, but we won't. Then I'm going to assign fixed length codewords to each sequence in the typical set.

Now, am I going to really build something which does this? Of course not. I mean, I'm talking about truly humongous lengths. So, this is really a conceptual tool to understand what's going on. It's not something we're going to implement. So I'm going to assign codewords to all these typical elements. And then what I find is that since the typical set, since the number of elements in it is less than 2 to the n times H of x plus epsilon, if I choose L bar, namely, the number of bits I'm going to use for encoding these things, it's going to have to be H of x plus epsilon in length. Because I need to provide codewords for each of these things. And it needs to be an extra 1 over n because of this integer constraint that we've been dealing with all along, which doesn't make any difference.

So if I choose L bar, that big, in other words, if I make it just a little bit bigger than the entropy, the probability of failure is going to be less than or equal to delta. And I can make delta -- and I can make the probability of failure as small as I want. So I can make this epsilon here which is the extra bits per source symbol as small as I want. So it says I can come as close to the entropy bound in doing this, and come as close to unique decodability as I want in doing this. And I have a fixed-to-fixed length code, which after one year is going to stop. And I can turn my decoder off. I can turn my encoder off. I can go buy a new encoder and a new decoder, which presumably works a little bit better. And there isn't any problem about when to turn it off. Because I know I can turn it off. Because everything will have come in by then.

Here's a more interesting story. Suppose I choose the number of bits per source symbol that I'm going to use to be less than or equal to the entropy minus 2 epsilon. Why 2 epsilon? Well, just wait a second. I mean, 2 epsilon is small and epsilon is small. But I want to compare with this other epsilon and my law of large numbers. And I'm going to pick n large enough. The number of typical sequences, we said before, was greater than 1 minus delta times 2 to the n times h of x minus epsilon. I'm going to make this epsilon the same as that epsilon, which is why I wanted this to be 2 epsilon.

So my typical set is this big when I choose n large enough. And this says that most of the typical set can't be assigned codewords. In other words, this number here is humongously larger then 2 to the l bar, which is in the order of 2 to the nh of x minus 2 epsilon n.

So the fraction of typical elements that I can provide codewords for, between this and this, I can only provide codewords for a fraction 2 to the minus epsilon n of the codewords. We have this big sea of codewords, which are all essentially equally likely. And I can't provide codewords for even a small fraction of them. So the probability of failure is going to be 1 minus delta. The 1 minus delta's the probability that I get something atypical. Plus, well, minus in this case, 2 to the minus epsilon n, which is the probability that I can't encode a typical codeword that comes out. And this quantity goes to 1. So this says that if I'm willing to use a number of bits bigger than the entropy, I can succeed with probability very close to 1. And if I want to use a smaller number of bits, I fail with probability 1. Which is the same as saying that I'm using a prefix-free code, I'm going to run out of buffer space eventually if I run long enough. If I have something that I'm encoding -- well, just erase that. I'll say it more carefully later.

I do want to talk a little bit about this Kraft inequality for unique decodability. You remember we proved the Kraft inequality for prefix-free codes. I now want to talk about the Kraft inequality for uniquely decodable codes. And you might think that I've done all of this development of the AEP, the asymptotic equipartition property. Incidentally, you now know where those words come from. It's asymptotic because this result is valid asymptotically as n goes to infinity. It's equipartition because everything is equally likely. And its property, because it's a property. So it's the asymptotic equipartition property. And I didn't do it so I could prove the Kraft inequality. It's just that that's an extra bonus that we get. And by understanding why the Kraft inequality has to hold for uniquely decodable codes, if is one application for AEP which lets you see a little bit about how to use it.

OK, so the argument is an argument by contradiction. Suppose you generate a set of lengths for codewords. And you want this -- yeah? And the thing you would like to do is to assign codewords of these lengths. And what we want to do is to set this equal to some quantity b. In other words, suppose we beat the Kraft inequality. Suppose we can make the lengths even shorter than Kraft says we can make them. I mean, he was only a graduate student, so we've got to be able to beat his inequality somehow. So we're going to try to make this equal to b. We're going to assume that b is greater than 1. And then what we're going to do is to show that we get a contradiction here. And this same argument can work whether we have a discrete memoryless source or a source with memory, or anything else. It can work with blocks, it can work with variable length to variable length codes. It's all essentially the same argument.

So what I want to do is to get a contradiction. I'm going to choose a discrete memoryless source. And I'm going to make the probabilities equal to 1 over b times 2 to the minus li. In other words, I can generate a discrete memoryless source for talking about it with any probabilities I want to give it. So I'm going to generate one with these probabilities. So the lengths are going to be equal to minus log of b times p sub i. Which says that the expected length of the codewords is equal to the sum of p sub i l sub i, which is equal to the entropy minus the logarithm of b. Which means I can get an expected length which is a little bit less than the entropy.

So now what I'm going to do is to consider strings of n source letters. I'm going to make these string very, very long. When I concatenate all these codewords, I'm going to wind up with a length that's less than n times H of x minus b over 2, minus log b over 2 with high probability. And as a fixed-length code of this length it's going to have a low failure probability. And, therefore, what this says is I can, using this remarkable code with unique decodability, and generating very long strings from it, I can generate a fixed-length code which has a low failure probability. And I just showed you in the last slide that I can't do that. The probability of failure with such a code has to be essentially 1. So that's a contradiction that says you can't have these unique decodable codes.

If you didn't get that in what I said, don't be surprised. Because all I'm trying to do is to steer you towards how to look at the section in the notes that does that. It was a little too fast and a little too late. But, anyway, that is the Kraft inequality for unique decodability. OK, thanks.