Lecture 1: Introduction

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educatiohal resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: As this says, and all the handouts say that this is 6.450. It's a first year graduate course in the principles of digital communication. It's sort of the major first course that you take as a graduate student in the communication area. The information theory course uses this as a prerequisite, uses it in a rather strong way. 6.432, the stochastic process course uses it as a prerequisite. 6.451, which is the companion course which follows after this uses it as a prerequisite. It's sort of material that you have to know if you're going to go into the communication area. It deals with this huge industry, which is the communication industry.

You look at the faculty here in the department and you're a little surprised to see that there are so few faculty members in digital communication, so many faculty members in the computer area. You wonder why this is. It's sort of this historical accident, because somehow or other the department didn't realize as these two fields were growing, the one that looked more glamorous for a long time was the computer field. Nobody in retrospect understands why that is. But at the same time that's how it is. You will not see as much excitement here. You also will see that because of the big bust in the year 2000, there's much less commercial activity in the communications field now. As students, you ought to relish this. You ought to really be happy about this if you want to go into the communication field, because at some point in the future, all of the failure to start new investments, and all of the large communication companies means a very explosive growth is about to start. So I don't think this is a dead field by any means.

Looking back historically, every time the communication field has seemed dead, and I can think of three or four such times in history, those have been the exact times where it was great to get into the field. There was a time in the early '70s when all the theoreticians who worked in communication theory all were going around with long faces saying everything that we can do has been done, the field is dead, nothing more to do, might as well give up and go into another field. This was really exactly the time that a number of small communication companies were really starting to grow very fast because they were using some of the modern ideas of communication theory and they were using these modern ideas to do new things. You now see companies like Qualcomm that were started then, Motorola of whom major divisions were started back then. An enormous amount of activity was starting just then, partly because the field seemed to be moribund and the theoreticians were turning and deciding well, I guess we have to do something practical now. If you want to find a good practical engineer, if you wind up in industry, if you wind up as an entrepreneur or if you wind up climbing the ladder and being a chief executive, if you want to find somebody who will really do important things, find somebody who understands theory who has decided that they suddenly want to start to do practical things. Because if you point to the people who have really done exciting things in this field, those are the ones who have done most of it.

OK, so anyway, the big field, it's an important field. What we want to do is to study the aspects of communication systems that are unique to communication systems. In other words, we don't want to study hardware here, we don't want to study software here, because hardware is pretty much the same for any problem you want to look at. It's not that highly specialized, and software is not either. So we're not going to study either of those things, we're going to study much more the architectural principles of communication. We're going to study a lot of the theory of communication, and I want to explain why it is that we're going to do that as we move on. Because it's not at all clear at first why we want to study the silly things that we'll be studying in the course, and I want to give you some idea today of why we're pushed into doing that. As we start doing these things, I'll give you more of an idea.

I know when I was a student I couldn't figure out why I was learning the things that I was learning. I just found that some things were much more interesting than others, and the things that tended to be interesting were the more theoretical things, so I started looking at them harder and thought gee, this is neat stuff. But I had no idea that it was ever going to be important. So I'm going to try to explain to you why this is important, and something today about what the proper inter-relationship is of engineering and theory.

As another example of my own life, for a long time I thought -- this is rather embarrassing to say -- but I was a theoretician and not much of an engineer for a very long time. And I kept worrying about this and saying well the stuff I'm doing is fun and I enjoy doing it, but why should anybody pay me for it? Of course, I was being paid by MIT, I was being paid as a consultant, but I felt I was robbing all these people. And finally, I decided a rationale for all of this. Most people felt that theory was silly, and therefore, they would keep asking me well, if you're so smart why aren't you rich? I didn't have any very good answer to that because I felt I was smart because I understood all this theory, but I wasn't rich. So I thought gee, what I've gotta do is do enough engineering to get rich, and then when people ask me that question I can say I am. I would suggest to all you, if you have theoretical leadings, to focus on some engineering also so you can become rich, not because there's anything to do with the money, but just because it lets you hold your head up high in a society that values money a lot more than values intellect. So it's important.

OK. This is a part of this two-term sequence. One comment here is I'm not sure that whether 6.451, which is the second part of this sequence, is going to be taught in the spring or not. It might not be taught until spring of the following year. If a large number of you decide that you really want to do this and do it now, make yourselves heard a little bit because the idea of a lot of people want this course will have something to do on whether somebody manages to teach it or not.

As I was starting to say, theory has had a very large impact on almost every field we can think of. I mean you think of electromagnetism, you think of all these other fields and theory is very important in them. But in communication systems it is particularly important. It's partly important because back in 1948, Claude Shannon who was the inventor of information theory -- you usually don't invent theories, but he really did. This came whole cloth out of this one mind. He had these ideas turning around in his head for about eight years while the second World War was going on. He was working on other things, which were, in fact, very important things, and he was doing those things for the government. He was working on aircraft control and things like that. He was working on cryptography. But he was just fascinated by these communication problems and he couldn't get his mind off them. He took up cryptography because it happened to use all the ideas that he was really interested in. He wrote something about cryptography during the middle of the war, which made many people believe that he had done his cryptography work before he did his communication work. But in fact, it was the other way around. This was purely the fact that he could write about cryptography, and he never liked to write, so he didn't want to write about communication because it was a more complicated field and he didn't understand the whole thing.

So for 25 years it was a theory floating around. You will learn a good deal of what this theory is. Most graduate courses on communication don't spend much time on information theory. This was not a mistake many, many years ago, but it is a mistake now, because communication theory at its core is information theory. Most people realize this. Most kids recognize it, most people in the street recognize it. When they start to use a system to communicate with the internet, what do they talk about? Do they talk about the bandwidth? No. They might call it bandwidth but what they're talking about is the number of bits per second they can communicate. This is a distinctly information theoretic idea, because communication used to be involved with studying different kinds of wave forms. Suddenly, at this point, people are studying all these other things and are based on Claude Shannon's ideas, therefore, we will teach some of those ideas as we go.

As a matter of fact, I was talking a little bit about the fact that there's been a sort of a love-hate relationship by communication engineers for information theory. Back in 1960, a long, long time ago when I got my PhD degree, the people at MIT, the people who started this field, many of them told me that information theory was dead at that point. There'd been an enormous amount of research done on it in the ten years before 1960. They told me I should go into vacuum tubes, which was the coming field at that time. Two points to that story. If people tell you that information theory isn't important, please remember that story or else tell them to start to study vacuum tubes. The other point of it -- well, I forgot what the other point of it is so let's go on.

Anyway, information theory is very much based on a bunch of very abstract ideas. I will start to say what that means as we go on, but basically it means that it's based on very simple-minded models. It's based on ignoring all the complexities of the communication systems and starting to play with toy models and trying to understand those. That's part of what we have to understand to understand why we're doing what we're doing.

A very complex relationship between modeling, theory, exercises and engineering design. Maybe I shouldn't be talking about that now and maybe the notes shouldn't talk about it, but I want to open the subject because I want you to be thinking about that throughout the term. Because it's something that most people really don't understand. Something that most teachers don't understand. It's something that most students don't understand. When you're a student, you do the exercises you do because you know you have to do them. There was a sort of apocryphal story awhile ago that somebody told me about a graduate student, not at MIT, thank God, who went into see the person teaching a course on wireless and saying, I don't want to know all that theory, I don't want to have to think about these things, just tell me how to do the problems.

Now, we will explain the enormous stupidity of that as we move on. Part of the stupidity is that the problems that you work on as a graduate student, even in your thesis, the problems that you work on are not real problems. The problems that we work on everywhere in engineering are toy problems. We work on them because we want to understand some aspect of the real problem. We can never understand the whole problem, so we study one aspect, then we study another aspect. The reason for all these equations that we have, it's not a way to understand reality. It really isn't. It's a way to understand little pieces of reality. You take all of those and after you have all of these in your mind, you then start to study the real engineering problem and you say I put together this and this and this and this and this -- this is important, this is not so important, here, this is more important. And then finally, if you're a really good engineer, instead of writing down an equation, you take an envelope, as the story goes, and you scribble down a few numbers, and you say the way this system ought to be built is the following.

That's the way all important systems get built, I claim. Important systems are not designed by the kinds of things you do in homework exercises. The things you do in homework exercises are designed to help you understand small pieces of problems. You have to understand those small pieces of problems in order to understand the whole engineering problems, but you don't do simulations to build a communications system. You understand what the whole system is, you can see it in your mind after you think about it long enough, and that's where good system design comes from. Simulations are a big part of all of this, writing down equations are a big part of it. But when you're all done, if you design an engineering system and you don't see in your mind why the whole thing works, you will wind up with something like Microsoft Word. That's the truth.

So the exercises that we're going to be doing are really aimed at understanding these simple models. The point of the exercises is not to get the right answer. The answer is totally irrelevant to everything. What's important is you start to understand what assumptions in the problem lead to what conclusions. How if you change something it changes something else. That's when you're using system design, because in system design you can never understand the whole thing. You have to look at what parts of it are important, what parts of it aren't important, and the only way you can do that is having a catalog of the simple-minded things in the back of your mind. Practical engineers get that by dealing with real systems day-to-day all their lives. They see catastrophes occur. Those things tell them what to avoid and what not to avoid. Students can do this in a much more efficient way. Obviously, in the practical experience also, but the experience that you get from looking at these exercises and looking at this analytical material in the right way, mainly looking at it is how do you analyze these simple toy models, how do you make conclusions from them is what let's you start being a good engineer.

I'm stressing this because at some point this term, all of you, all of you at different times are going to start saying why the hell am I doing this? Why am I dealing with this stupid little problem? Stupid little problems can get incredibly frustrating at times, because you see something that's so simple and you can't understand it. Then you say well, this simple thing can't be important anyway. What I really want to do is understand what this overall system is, and you give up at that point. Don't do that. Sometimes there's a very difficult question about what you mean by something being simple. I will sometimes say that something is very simple, and I will offend many you to whom it's not simple. The point is something becomes simple after you understand it. Nothing is simple before you understand it. So there's that point at which the light goes off in your head at which it becomes simple. When I say that something is simple, what I mean is that if you think about it long enough a light will go off in your head and it will become simple. It's not simple to start with. Other things are just messy. You've all heard of mathematical problems which are just ugly. There are these things of extraordinary complexity. There are things where there just isn't any structure. You see a lot of these things in computer science. I talked about Microsoft Word. Part of the reason it's such a lousy language is because the problem is incredibly difficult. It's incredibly unstructured. So people come up with things that don't make any sense, because that inherently is not simple. OK, these other things that we'll be studying here are inherently simple and the only problem is how do you get to the point of seeing these simple ideas.

OK, so that's enough philosophy. No, not quite enough, a little more of that. All the everyday communication systems that we deal with are incredibly complex in terms of the amount of hardware, the amount of software in them. If you look at the whole system and you try to understand it as a whole, you don't have prayer. There's no way to do it. These things work and they can be understood because they're very highly structured. They have simple architectural principles. What do I mean by architectural principles? It's a word that engineers use more and more. It started off with computer systems because it became essential there to start thinking in terms of architecture. It's exactly the same thing that you mean when you're talking about a house or a building. Mainly the architecture is the overall design. It's the way the different pieces fit together. In engineering, architecture is the same thing. It's the thing that happens when your eyes fuzz over a little bit and you can't see any of the details anymore, and suddenly what you're doing is you're looking at the whole thing as an entity and saying how do the pieces fit together. Well, what we're going to be focusing on here.

One of the major keys to making these things simpler is to have standardized interfaces and to have layering. And we'll talk a little bit about each of these because our entire study of this subject is based on studying the different layers that we'll wind up with. OK, the most important and most critical interface in a communication system is between the source and the channel. Today, that standard interface is almost always a binary data stream. You all know this. When you talk about your modab, how do you talk about it? 48 kilobits per second if you're an old-fashioned kind of guy. 1.2 megabits if you're a medium technology person, or several gigabits if you're one of these persons who likes to gobble up huge amounts of stuff. That's the way you describe channels -- how many bits per second can you send? The way you describe sources is how many bits per second do you need as a way of viewing that source. When you store a picture, how do you talk about it? You don't talk about in terms of the colors or any of these other things, picture is a bunch of bits. That, at the fundamental level, is what we're doing here. When you deal with source coding, you're talking fundamentally about the problem of taking whatever the source is, and it can be any sort of thing at all -- it can be voice, it can be text, it can be emails, the emails with viruses in it, whatever -- and the problem is you want to turn it into a bit stream.

Then this bit stream gets presented to the channel. Channels and the channel encoding equipment don't have any idea of what those bits mean. They don't want to have any idea of what those bits mean. That's what an interface means. It means the problem of interpreting the bits, the problem of going from something which you view as having intelligence down to a sequence of bits is the problem of the source coder. The problem of taking those bits and moving them from one place to another is the function of the channel designer, the network designer and all of those things. When you talk about information theory, it's a total misnomer from the beginning. Information theory does not deal with information at all, it deals with data. We will understand what that distinction is as we go on. When you talk about a bit stream, what you're talking about is a sequence of data. It doesn't mean anything. It's just that a one is different from a zero, and you don't care how it's different, it just is.

So the channel input is a binary stream. It doesn't have any meaning as far as the channel is concerned. The bit stream does have a meaning as far as the source is concerned. So the problem is the source encoder takes these bits, takes the source, whatever it is, with its meaning and everything else, turns it into a stream of bits -- you would like to turn it into as few bits as possible. Then you take those bits, you transmit them on the channel, the channel designer understands what the physical medium is all about, understands what the network is all about, and doesn't have a clue as to what all those bits mean. So the picture is the following. That's the major layering of all communication systems. Here's the input, which is whatever it happens to be. Here's the source encoder. The source encoder has to know a great deal about what that input is. You have to know the structure of the input. You have a source encoder for voice and you use it to try to encode a video stream it's not going to work, obviously. If you try to use a source encoder for English and try to use it on Chinese, it probably won't work very well either. It won't work for anything other than what it's intended for. So the source encoder has to understand the source. When I say understand the source, what do I mean? It means to understand the probabilistic structure of the source. This is another of the things that Shannon recognized clearly that hadn't been recognized at all before this. You have to somehow understand how to view what's coming out of the source, this picture, say, as one of a possible set of pictures.

If you know ahead of time that a communication system is going to be sending the Gettysburg Address or one of 50 other highly-inspiring texts, what do you do? Do you try to encode these things, all these different texts? Of course not. You just assign number one to the Gettysburg Address, number two to the second thing that you might be interested in. You send a number and then at the output out comes the Gettysburg Address because you've stored that there. So that's the idea of source coding. What is important is not the complexity of the individual messages, it's the probabilistic structure that tells you what are the different possibilities. That's what happens when you try to turn things into binary digits. You have one sequence of binary digits for each of the possible things that you want to represent.

Then in the channel you have the same sort of thing. You have noise, you have all sorts of other crazy things on the channel. You have a sequence of bits coming in, you have a sequence of bits coming out. The fundamental problem of the channel is very, very simple. You want to spit out the same bits that came in. You can take a certain amount of delay doing that, but that's what you have to do. In order to do that, you have to understand something about the probabilistic structure of the noise. So both ways you have probabilistic structure. It's why you have to understand probability at the level of 6.041, which is the undergraduate course here studying probability. If you don't have that background, please don't take the course because you're going to be crucified. If you think you can learn it on the fly, don't. You can't learn it on the fly. As a matter of fact, if you've taken the course, you still don't understand it, and you will need to scramble a little bit to understand it at a deeper level in order to understand what's going on here. 6.041 is a particularly good undergraduate course. I think it's probably taught here better than it's taught at most places. But you still don't understand it the first time through. It's a tricky, subtle subject you really need to understand enough of it, If you have no exposure to it. If you've taken a course called statistics and probability, which first teaches you statistics and then tucks in a little bit of probability at the end, go learn probability first because you don't have enough of it to take this subject.

The other part of dealing with these channels is that suddenly we have to deal with 4AA analysis and all of these things, if you don't have some kind of subject in signals and systems. Some computer scientists don't learn that kind of material anymore. Again, you can't learn it on the fly because you need a little bit of background for it. Those two prerequisites are really essential. Nothing else is essential. The more mathematics you know the better off you are, but we will develop what we need as we go.

Now, we talked about this fundamental layer in all communication systems, which is between source coding and channel coding. You first turn the source into bits, and then you turn the bits into something you can transmit on the channel. Source coding breaks down into three pieces itself. It doesn't have to break down into those three pieces, but it usually does. Most source coding you start out with a wave form, such as with voice, or with pictures you start out with what's more complicated than a wave form, some kind of mapping from x-coordinate and y-coordinate and time, and you map all of those things into something. Our problem here is we want to take these analog wave forms or generalizations of. It turns out that all we have to deal with here is the analog wave forms because everything else follows easily from that. So there's one question. How do you turn an analog wave form into a sequence of numbers? This is the common way of doing source coding. Many of you who have been exposed to the sampling theorem, you think that's the way to do it, this is one way to do it. We will talk a great deal about this. You will find out that the sampling theorem really isn't the way to do it. Although again, it's a simple model, it's the way to start dealing with it. Then you learn the various other parts of that as you go along. After you wind up with sequences, sequences of numbers, we worry about how to quantize those numbers, because fundamentally we're trying to get from wave forms into bits. So the three stages in that process are first go to sequences, go from sequences to symbols -- namely, you have a finite number of symbols, which are quantized versions of those analog numbers which can be anything. Then finally, you encode the symbols into bits.

The coding goes in the opposite order. Here's a picture which shows it better than the words do. From the input wave form, you sample it or something more generalized than sampling it. That gives you a sequence of numbers. You quantize those numbers, that gives you a symbol sequence. You have a discrete coder, which turns those symbols into a sequence of bits in some nice way. Then you have this reliable binary channel. Notice what I've done here. This thing is that entire system we were talking about before. This has a channel encoder in it, a channel, a channel decoder and all of that. But what we've been saying in this layering idea is when you're dealing with source coding you ignore all of that. You say OK, it's Tom's job who was designing the channel encoder to make sure that my bits come out here as the same bits. If the bits are wrong it's his fault. If the bits are right, and this output wave form doesn't look like the input wave form, it's my fault. So part of the layering idea is we just recreate this whole thing here as the same bits and we don't worry about what happens when the bits are different.

The other part of this is that all of this breaks down in the same way. Namely, at this interface between here and here, the idea is the same. Namely, there's an input wave form that comes in, there's a sequence of numbers here. The job of this analog filter, as we call it, and you'll see why we call it that later, is to take numbers here, turn them back into wave forms. These numbers are supposed to resemble these numbers in some way or other. I can deal with this part of the problem totally separately from dealing with the other parts of the problem. Namely, the only problem here is how do I take numbers, turn them into wave forms. There are a bunch of other problems hidden here, like these numbers are not going to be exactly the same as these wave forms because I have quantization involved here. Since the numbers are not exactly the same, we have to deal with questions of approximation. How to approximations in numbers come out in terms of approximations in terms of wave forms? That's where you need things like the sampling theorem and the generalizations of it that we're going to spend a lot of time with.

Then you go down to this point. At this point the quantizer produces a string of symbols. Whatever those symbols happen to be, but those symbols might, in fact, just be integers whereas the things here were real valued numbers, and in quantizing we turn real numbers into intergers by saying -- well, it's the same way you approximate things all the time. You round off the pennies in your checkbook. It's the same idea. So the problem here is the quantizer produces these symbols here. The job of all of this stuff is to recreate those same symbols here. So the symbols now go into something that does the reverse of a quantizer -- we'll call it a table look-up because that's sort of what it is. So we can study this part of the problem separately, also. You don't have to understand anything about this problem to understand this problem. Finally, we have symbols here going into a discrete encoder. We have an interesting problem which says how do we take these symbols which have some kind of probabilistic structure to them and turn them into binary digits in an efficient way. If the symbols here, for example, are binary symbols and the symbols happen to be 1 with probability 999 out of 1,000, and zero with probability 1 in 1,000, you don't really just want to take these symbols and map them in a straightforward way, 1 into 1 and zero into zero. You want to look at whole sequences of them. You want to do run length coding, for example. You want to count how long it is between these zero's, and then what you do is send those counts and you encode them. So there are all kinds of tricks that you can use here. But the point is, this problem is separate, and this problem is separate from this problem.

Now what we're going to do in this course is start out with this problem, because this is the cleanest and the neatest of the three problems. It's the one you can say the most about. It's the one which has the nicest results about it. In fact, a lot of source coding problems just deal with this one part of it. Whenever you're encoding text, you're starting out with symbols which are the letters of whatever language you're dealing with and perhaps a bunch of other things -- ask a code, you've generated 256 different things, including all the letters, all the capital letters, all of the junk that used to be on typewriters, and some of the junk that's now in word processing languages, and you have a way of mapping those into binary digits. We want to look at better ways of doing that. So this, in fact, covers the whole problem of source coding when you're dealing with text rather than dealing with wave forms. This problem here combined with this problem deals with all of these many problems where what you're interested in is taking a sequence of numbers or a sequence of whatjamacallits and quantitizing them and then coding. Finally, when you get to the input wave forms and output wave forms, you've solved both of these problems and you can then focus on what's important here.

This is a good case study of why the information theoretic approach works and why theory works. Because if you started out with the problem of saying I want to build something to encode voice, and you try to do all of these together, and you look up all the things you can find about voice encoding on the web and you read all of them, what you will wind up with, I can guarantee you, is you will know an enormous amount of jargon, you will know an enormous number of different systems, which each half work, you will not have the slightest clue as to how to build a better system. So you have to take the structured viewpoint, which is really the only way to do things to find new and better ways of doing things.

Now there are some extra things. I have over-simplified it. I want to over-simplify things in this course. What I will always do is I'll try to cheat you into thinking the problem is simpler than it is, and then I will talk about all of the added little nasties that come in as soon as you try to use any of this stuff. There are always nasties. What I hope you will all do by the end of the term is find those little nasties on your own before I start to talk about them. Namely, when we start to talk about a simple model, be interested in the simple model, study it, find out what it's all about, but at the same time, for Pete's sake, ask yourself the question. What does this have to do with the price of rice in China or with anything else? And ask those questions. Don't let those questions interfere with understanding the simple model, but you've got to always be focusing on how that simple model relates to what you visualize as a communication system problem. It usually will have some relation but not a complete relation.

Here the problems are in this binary interface. The source might produce packets or the source might produce a stream of data. In other words, if what you're dealing with is the kind of situation where you're an amateur photographer, you go around taking all sorts of pictures, and then you encode these pictures -- what do I mean by encoding the pictures? You turn them into binary digits. Then you want to send your pictures to somebody else, but what you're really doing is sending these packets to someone else. You're not sending a stream of data -- you have 10 pictures, you want to send those 10 pictures. At the output of the whole system, you hope you see 10 separate pictures. You hope there's something in there which can recognize that out of the stream of binary digits that come across the channel, there's some packet structure there.

Well we're not going to talk about that really. Whenever you start dealing with this problem of source encoding, you also need to worry a little bit about the packet structure. What are the protocols for knowing when something new starts, when something new ends. Those kinds of problems are dealt with mostly in a network course here, and you can find out all sorts of things about them there. But there are these two general types of things -- packets and streams of data. In terms of understanding voice encoding, you can study them both together. Why can you study them both together? Because the packets are long and because the packets are long because they include a lot of data, you have a little bit of end effect about how to start it, a little bit of end effect about how to end it, but the main structural problem you'll be dealing with is what to do with the whole thing. It's an added piece that comes at the end to worry about how do you denote the beginning and the end. That's a problem you have with stream data also, because stream data does not start at time minus infinity. Whatever the stream data is coming from, what it's coming from did not at time, t equals infinity. You just think of it that way because you recognize you can postpone the problem of how do you start it and how do you end it, and hopefully you can postpone it until somebody else has taken over the job.

What the channel accepts is either of these binary streams or packets, but then queueing exists. In other words, you have stuff coming into a channel -- sometimes I'm sitting at home and I want to send long files, I want to send this whole textbook I'm writing to somebody. It queues up, it takes a long time to get it out of my computer and into this other person's computer. It would be nice if I could send it at optical speeds. But I don't care much. I don't care much whether it takes a second or a minute or an hour to get to this other person. He won't read it for a week anyway, if he reads it at all, so what difference does it make? But anyway, we have these queueing problems, which are, again, separable. They're dealt with mostly in the network course. What we're mostly interested in is how to reduce bit rate for sources. How do you encode things more efficiently? Data compression is the word usually given to this. How do you compress things into a smaller number of bits using the statistical structure of it? Then how do you increase the number of bits you can send over a channel? That's the problem with channels. If you have a channel that'll send 4,800 bits per second, which is what telephone lines used to do. If you build a new product that sends 9,600 bits per second, which was a major technological achievement back in the '70s and '80s, suddenly your company becomes a very important communication player. If you then take the same channel and learn how to communicate at megabits over it, that's a major thing also. How did people learn to do that? Mostly by using all the ideas that the people used in going from 4,800 bits per second to 9,600 bits per second. It was the people who did that first job who were the primary people involved in the second job. The point of that is it doesn't really make any difference as an engineer whether you're going from 4,800 to 9,600 or from 9,600 to 19.2 or from 19.9 to 38.4 or whatever it is, or so forth up. Each one of these is just putting in a few new goodies, recognizing a few new things, making the system a little bit better.

Let's talk about the channel now. The channel is given -- in other words, it's not under the control of the designer. This is something we haven't talked about yet. In all of these communication system design problems, one of the things that gets straight is when you're worrying about the engineering of something, what parts of the system can you control and what parts of it can't you control. Even more when you get into layering -- what parts can you control and what parts can other people control. Namely, if I have some source and I'm trying to transmit it over some physical channel and I have two engineers -- one of them is a data compressor, and the other is a channel guy. What the channel guy is trying to do is to find a way of sending more bits over this channel than could be done before. What the data compressor is trying to do is to find a way to encode the source into fewer bits. If the two come together, the whole thing works. If the two don't come together, then you have to fire the engineers, hire some new engineers or do something else or you get fired. So that's the story. The things you can't change here, you can't change the structure of the source usually, and you can't change the structure of the channel.

There's some very interesting new problems coming into existence these days by information theorists who are trying to study biological systems. One of the peculiar things in trying to understand how biological organisms, which have remarkable systems for communicating information from one place to another, how they manage to do it. One of the things you find is that this separation that we use in studying electronic communication does not exist in the body at all. In other words, what we call a source gets blended with the channel. What we call the statistics of the source get very much blended in with what this organism is trying to do. If the organism cannot send such refined data, it figures out ways to get the data that it needs which is essential for its survival or it dies and some other organism takes over. So you find the system which has no constraints in it, and it evolves with all of these things changing at the same time. The channels are changing, the source statistics are changing, and the way the source data is encoded is changing, and the way the data is transmitted is changing.

So it's a whole new generalization of this whole problem of how do you communicate. Here though, the problem we're dealing with is these fixed channels, which you can think of, depending on what your interests are, you can view the canonical channel as being a piece of wire in a telephone network, or even view it as being a cable in a cable TV system, or you can view it as being a wireless channel. We will talk about all of these in the course. The last three weeks of the course is primarily talking about wireless because that's where the problems are most interesting. But in any situation we look at, the channel is fixed. It's given to us and we can't change it. All we can do fiddle around with the channel encoding and the channel decoding. The channels that we're most interested in usually send wave forms. That brings up another interesting question. We call this digital communication, and what I've been telling you is that the sources that we're interested in are most often wave form sources. The channels that we're interested in are most often channels that send wave forms, receive wave forms and add noise, which is wave forms. Why do we call it digital communication? Anybody have a clue as to why we might call all of this digital communication? Yeah?

AUDIENCE: [INAUDIBLE]

PROFESSOR: Because the analog gets represented in binary digits, because we have chosen essentially, because of having studied Shannon at some point, to turn all the analog sources into binary bit streams. Namely, when you talk about digital communication, what you're really doing is saying I have decided there will be a digital interface between source and channel. That's what digital communication is. That's what most communication today is. There's very little communication today that starts out with an analog wave form and finds a way to transmit it without first going into a binary sequence.

So here's a picture of one of the noises that we're going to study. We have an input which is now a wave form. We have noise, which is a wave form. And we have an output, which is a wave form. This input here is going to be created somehow in a process of modulation from this binary stream that's coming into the channel encoder. So somehow or other we're taking a binary stream, turning it into a wave form. Now, think a little bit about what might be essential in that process. If you know something about practical engineering systems for communication, you know that there's an organization in the U.S. called the FCC, and every other country has its companion set of initials, which says what part of the radio spectrum you can use and what part of the radio spectrum you can't use. You suddenly realize that somehow it's going to be important to turn these binary streams into wave forms, which are more or less compressed into some frequency bands. That's one of the problems we're going to have to worry about. But at some level if I take a whole bunch of different binary sequences, one thing I can do, for example, I could take a binary sequence of length 100. How many such binary sequences are there? Has length 100, the first bit can be 1 or zero, second bit can be 1 or zero -- that's four combinations. Third bit can be 1 or zero also -- that makes eight combinations of three bits. There are 2 to the 100th combinations of 100 binary digits. So a theoretician says fine, I'm at those 2 to the 100th different combinations of binary digits into evenly spaced numbers between zero and 1. Then I will take those evenly spaced numbers between zero and 1 and I'll modulate some wave form, whatever wave form I happen to choose, and I will send that wave form. In the absence of noise, I pick up that wave form and turn it back into my binary sequence again. What's the conclusion from this? The conclusion is if you don't have noise there's no constraint on how much data you can send. I can send as much data as I want to, and there's nothing to stop me from doing it, if I don't have noise. Yeah.

AUDIENCE: So, is that where something that [UNINTELLIGIBLE] basically comes in. Like a difference between one signal and the next signal.

PROFESSOR: Somehow I have to keep these things seperate. I don't necessarily have to separate one binary digit from the next binary digit in time. I could, in fact, do something like creating 2 to the 100th different wave forms or I could do anything in between. I could take each sequence of eight bits, turn them into 1 of 256 wave forms, transmit one wave form, then a little bit later transmit another wave form and so forth. So I can split up the pie in any way I want to, and we'll talk about that a lot as we go on. Now, I split them up into different wave forms, which I send at spaced times. I somehow have to worry about the fact that if I send them over a finite bandwidth, there is no way in hell that you can take finite bandwidth wave forms and send them in finite time. Anything which lasts for a finite amount of time spreads out in bandwidth. Anything which is in a finite amount of bandwidth spreads out in time. That's what you ought to know from 6.003. So we're dealing with an unsoluble problem here.

Well, Nyquist back in 1928 solved that problem. He said well no, you can't make the wave form separate, but you can make the sample separate. People earlier had figured out they could do that with the sampling theorem, but the sampling theorem wasn't very practical, but Nyquist's way of doing it was practical. So, in fact, you can separate these wave forms. But you still have the problem how do you deal with the noise. Well, one of the favorite ways of dealing with noise is to assume that it's something called white Gaussian noise. What is white Gaussian noise? White Gaussian noise is noise which no matter where you look it's sitting there. You can't get away from it. You move around to different frequencies, the noise is still there. You move around to different times, it's still there. It is somehow uniformed throughout time and throughout frequency. Kind of an awkward thing because if I developed a receiver which didn't have any bandwidth constraint on it, and I looked at this noise coming in, which was spread out over all frequencies, the noise would burn out the receiver. In other words, this white Gaussian noise has infinite power. That shouldn't bother you too much. You deal with unit impulses all the time. A unit impulse has infinite energy also. You probably never thought about it because most undergraduate courses try very hard to conceal that fact from you because they like to use impulses for everything, but impulses have infinite energy associated with them, and that's kind of a nasty thing. But anyway, we will deal with the fact that impulses have infinite energy, and therefore, not use them to transmit data. And we will deal with the fact that this noise has infinite power, and find out how to deal with that. And we will learn how to understand the statistical structure of that noise.

So we'll deal with all of these things when we start studying these channels. As we do that we will find out that no matter what you do on a channel like this, there are some fundamental constraints on how much data can be transmitted. I can take you through a little bit of a thought experiment which will sort of explain that, I think. It's partly what we were just starting to say over here. A simple-minded way to look at this problem is I have binary digits coming in. I take these binary digits and I separate them into shorts sequences of m binary digits each. So, I take the first m binary digits, then I take the next m binary digits, and the next m binary digits and so forth. A sequence of m binary digits, there are 2 to the m combinations of this. So I map these 2 to the m combinations into one of 2 to the m different numbers. Now, you look at this noise and you say I have to keep some separation between these numbers. So, with a separation between the numbers, say the separation is 1, let this define the number 1. That's something we theoreticians do all the time. So I have 2 to the m different levels. I have to send this in a certain bandwidth. The sampling theorem says I can't send things that are more than twice the bandwidth of this system, so I have a bandwidth of 2w. I can send m binary digits in this bandwidth, w. Therefore, I can send bits at a certain rate. This is all very crude but it's going to lead us to an interesting trade-off. I can increase m and by increasing m I can get by with a smaller bandwidth, because I have these symbols, these m bit symbols coming along at a slower rate. So by increasing m, I can decrease the bandwidth, by increasing m, I can reduce the bandwidth, but by reducing the bandwidth, the power is going up. You can see that as I change m, the power is going up exponentially with m. In other words, the trade-off between bandwidth and power is kind of nasty in this simple-minded picture.

Let me jump ahead. For this added of white Gaussian noise with a bandwidth constraint, Shannon showed that the capacity was w times log of 1 plus the power in the system times the noise power times w. Now, I don't expect you to understand this formula now, and there's a lot of very tricky and somewhat confusing things about it. One thing is that the power and the noise is going up with the bandwidth that you're looking at. Because I said the bandwidth is everywhere, you can't avoid it, and therefore, if you use a wider bandwidth system, you get more noise coming into it. If you have a certain amount of power that you're willing to transmit and a certain amount of noise, and the number of bits per second you can transmit is going up with w. But look at this affect with p here, the effect with p is logarithmic, which says as I increase this parameter m and I put more and more bits into one symbol, this quantity is going up exponentially, which means the logarithm of this is going up just linearly with m.

This gives you, if you look at it carefully, the same trade-off as the simple-minded example I was talking about. That simple-minded example does not explain this formula at all. This formula's rather deep. We probably won't even completely have proven this term. What Shannon's results says is that no matter what you do, no matter how clever you are with this noise that exists everywhere, the fastest you can transmit with the most sophisticated coding schemes you can think of is this rate right here, which is what you would think it might be just from the scaling in terms of w and m, where when you increase m the power goes up exponentially. It's the same kind of answer you get with that simple thought experiment. The fact that you have a 1 in here is a little bit strange, and we'll find out where that comes from. The idea here is that this noise is something you can't beat. The noise is fundamental. It's a fundamental part of every communication system. As I always like to say, noise is like death and taxes, you can't avoid them, they're there, and there's nothing you can do about them. Like death and taxes, you can take care of yourself and live longer, and with taxes, well, you can become wealthy and support the government and make all sorts of payments and reduce your taxes that way. Or you can hire a good accountant. So you can do things there also. But those things are still always there. So you're always stuck with this. One of the major parts of the course will be to understand what this noise is in a much deeper context than we would otherwise. So, I'll save that and come back to it later.

We have the same kind of layering in channel encoding. When I talk about channeling encoding, I'm not talking about any kind of complicated coding technique. 6.451 talks about all of those. What I'm talking about is simply the question of how do you take a sequence of bits, turn it into a sequence of wave forms in such a way that at the receiver we can take that sequence of wave forms and match them reliabily back into the original bits. There's a standard way of doing this, which is to take the sequence of bits, first send them through a discrete decoder. What the discrete encoder code will do is something like this. And match bits at one rate into bits at a higher rate, and this let's you correct channel errors. A simple example of this is you match zero into zero, zero, zero. 1 into 1, 1, 1. If the rest of your system, namely, if this part of the system makes a single error at some point, this part of the system escapes from it. I sort of put these on one slide but I couldn't. This takes a single zero, turns it into three zeros. These three zeros go into this modulator, which maps this into some wave from responsive to these three zeros. Wave form goes through here, noise gets added, the modulator, the text, those binary digits, comes out with some approximation to these three zeros. Every once in awhile because of the noise, one of these bits comes out wrong, and because they come out wrong this discrete decoder says ah-ha, I know that what was sent/put in was either zero, zero, zero or 1, 1, 1. It's more likely to have one error than to have two errors, and therefore, any time one error occurs I can decode it correctly. Now, that is a miserable code.

A guy by the name of Hamming became very famous for generalizing this just a little bit into another single error correcting code, which came through at a slightly higher rate but still corrected single errors. He became inordinately famous because he was a tireless self-promoter and people think that he was the person who invented error correction coding. He really wasn't. And his heirs will probably -- I don't know. Anyway, coding of this type, namely mapping binary digits into longer strings of binary digits in such a way that you can correct binary errors, has always been a major part of communication system research. A large number of communication systems work in exactly this way. In fact, the older systems which use coding tended to work this way. This was a very popular way of trying to design system. You would put a total layering across here. Namely, this part of the system didn't know that there was any coding going on here. At this point where you're doing discrete decoding, you don't know anything about what the detector is doing here. It doesn't take a lot of imagination to say if you have a detector here and there's this symbol that comes through here, noise gets added to it, you're trying to decode this symbol, and there's this Gaussian noise, which is just sort of spread around, it's concentrated. Usually when errors occur, errors occur just barely. What you see at this point is you almost flip a coin to decide whether a zero or a 1 was transmitted at this point. If this poor guy had that information, this poor guy could work much, much better. Namely, this guy could correct double errors instead of single errors, because if one bit came through with a clear indication that that's what it was, and the other two bits came through saying well, I can't really tell what these are, then this guy could correct two errors instead of one error.

So this is a good example where this layering in here is really a rotten idea. Most modern systems don't do that strict layering. But a whole lot of modern systems do, in fact, do this binary encoding here and they use the principles that came out of the binary encoding from a long time ago. The only extra thing they do here is they use this extra information from the demodulator. This is something called soft decoding instead of hard decoding. In other words, what comes out of the demodulator is soft decisions, and soft decisions say well, I think it was this but I'm not sure or I am sure and so forth. There's a whole range of graduations. We will study detection theory and we'll understand how that works and we'll understand how you use those soft decisions and all of that stuff and it's kind of neat.

The modulation part of this is what maps bit sequences into wave forms. When we map bit sequences into wave form, there's this very simple idea. If I take a single bit that's either zero or 1, I map a zero into one wave form, I map a 1 into another wave form, I send either wave form a or wave form b. I somehow want to make those wave forms as different from each other as I can. Now, what do you mean by making wave forms different? I know what it means to make numbers different. I know that zero and 3 are more different than zero and 2 are. Namely, I have a good measure of distance for a sequence of numbers. One of the things that we have to deal with is how do you find an appropriate measure of distance for wave forms. When I find the measure of distance for wave forms, what do I want to make it responsive to? What I'm trying to do is to overcome this noise, and therefore, I have to find the measure of distance which is appropriate for what the noise is doing to me. Well that's why we have to study wave forms as much as we do in this course. What we will find is that you can treat wave forms in the same way as you can treat sequences of numbers. We will find that there's a vector space associated with wave forms, which is exactly the same as a vector space associated with infinite sequences of numbers. It's almost the same as the vector space associated with finite sequences of wave form, and that vector space associated with a finite sequence of numbers is exactly the vector space that you've been looking at all of your lives, and which you use primarily as a notational convenience to talk about a sequence of numbers. What's the distance that you use there? Well, you take the difference between each number in the sequence, you square it, you add it up and you take the square root of it. Namely, you look at the energy difference between these sequences. What we will find remarkably is when we do things right, the appropriate distance to talk about on wave forms, is exactly what comes from the appropriate looking at sequences. In fact, we'll take wave forms and break them into sequences. That's a major part of this modulation process. You think of that in terms of the sampling theorem. Namely, you take a bunch of numbers, you take each number in the sampling theorem and you put a little sin x over x hat around it and you transmit that, and then you add up all of these different samples, which is a sequence of numbers, each of them with these little sin x over x caps around them. The neat thing about the sin x over x caps around them is they all go to zero with these sample points and it all works. Therefore, the sequences look almost the same as a wave form. We'll find out that that works in general, so we will do most of that.

Modern practice often combines all these layers into what's called coding modulation. In other words, they go further than just this idea of soft decisions, and actually treat the problem in general of what do you do with bits coming into a sequence, into a system, how do you turn those into wave forms that can be dealt with an appropriate way. And we'll talk just a little bit about that and mostly 6.451 talks about that.

I want to talk a little bit right at the end, I'm almost finished, about computational complexity in these communication systems that we're trying to worry about. One of the things that you will become curious about as we move along is that we are sometimes suggesting doing things in ways that look very complex. The word complex is an extraordinarily complex word. None of us know what it means because it means so many different things. I was saying that these telephone systems are inordinately complex. In a sense they aren't because they have an incredible number of pieces, all of which are the same. So you can understand a telephone system in terms of understanding a relatively small number of principles. You can understand these Gaussian channels in terms of understanding a small number of principles. So they're complex if you don't know how to look at them, they're simple if you do know how to look at them. Here what I'm talking about is how many chips do you need to build something? How complicated are those chips? Fundamentally, how much does it cost to build the system? Well, one of the curious things and one of the things that has made information theory so popular in designing communication systems is that we're almost at the point where chips are free. Now, what does that mean? It means if you want to do something more complicated it hardly costs anything. This is sort of Moore's law. Moore's law says every year you can put more and more stuff on a single chip. You can do this cheaply and the chips work faster and faster so you can do more and more complicated things very, very easily.

There are some caveats. Low cost with high complexity requires large volume. In other words, it takes a very long time to design a chip, it takes an enormous amount of lead time -- I mean we're not going to study these details in the course, but I want you to be aware of why it is that in a sense complexity doesn't matter anymore. If you spend a long enough time designing it and you can produce enough of them, you can do extraordinarily complex things very, very cheaply. I mean you see this with all the cellular phones that you buy. I mean cellular phones now are actually give-away devices. You look at them to see what's in them and they're inordinately complicated. You buy a personal computer today and it's 1,000 times more powerful than the biggest computers in the world 30 years ago. You can do everything more cheaply now than you could before.

What's the hitch? Well, you see the hitch when you program a computer, namely, if you look at word processing languages, yes, they become bigger and bigger every year, they become more and more complex, they will do more and more things. As far as their function of word processing, some of us think there have been advances in the last 30 years. Others, like myself, feel that there have not been advances and, in fact, we're going backwards. The problem is we have so much complexity we don't know how to deal with it anymore. Most of us have become blinking 12's. You know what a blinking 12 is. It's all of the electronic equipment you have in your home, whenever you don't program it right you see a 12 blinking on and off, which is where the clock is and the clock hasn't been set, and therefore, it's blinking at you. Most people who have complicated devices, whether they're audio devices or what have you, are using less than one percent of the fancy features in them. You spend hours and hours trying to bring that from one percent up to two percent. So that the cost of complexity is not in what you can build, but it's in this conceptual complexity. If you design an algorithm and you understand the algorithm, it hardly makes any difference how complicated it is, unless the complexity is going up exponentially with something. That's the first caveat. It's actually the second caveat, too.

Complex systems are often not well thought through. They often don't work or are not robust. The non-robustness is the worst part of it. The third caveat is when you have special applications. Since they involve small numbers, you're not going to build a lot of them, it takes a long time to design them. The only way you can build special applications is to make them minor modifications of other things that have been done before. So this question of how complicated the systems we can build are, if you can make enough of them they become cheap. If you have enough lead time they become cheap. Which says that this layering of communication system that we're talking about really ultimately makes sense.

Starting next time, we're going to start talking about this first part of source coding, which is the discrete part of source coding. We will have those notes on the web shortly. You can look ahead at them, if you would like to, before Monday. Please review the probability that you are supposed to know because you will need it very shortly. Thanks.