Lecture 17: Detection for Random Vectors and Processes

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation, or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu

PROFESSOR: Review what random processes are a little bit now. And remember, in what we're doing here, we have pointed out that random processes in general, have a mean. The mean is not important. The mean is not important for random variables or random vectors either. It's just something you add on when you're all done. So the best way to study random variables, random vectors, and random processes, particularly when you're dealing with things which are based on base things being Gaussian, is to forget about the means, do everything you want to with the fluctuations, and then put the means in when you're all done. Which is why the notes do most of what they do, in terms of zero mean, random variables, random vectors, and random processes, and simply put in what the value of the mean is at the end.

OK so some of this has the mean put in. Some of it doesn't. So to start out with, a random process is defined by its joint distribution at each finite set of epochs. OK, that's where we started with all of this. How do you define a random waveform? I mean, you really have to come to grips with that question before you can see how you might answer the question. If you think it's obvious how to define a random process, then you really have to go back and think about it. Because a random process is a random waveform. It's an uncountably infinite number of random variables. So that defining what it is, is not a trivial problem. And people have generally agreed that the way to define it, be the test for whether you have defined it or not, is whether you can find the joint distribution at each finite set of epochs.

Fortunately most processes of interest can be defined in a much, much simpler way in terms of an orthonormal expansion. When you to find it in terms of an orthonormal expansion, you have the orthonormal expansion which is the sum over k, of a set of random variables, Z sub 1, Z sub 2 and so forth Z sub k, times a set of orthonormal functions. So all the variation on t is stuck into the set of orthonormal functions. All the randomness is stuck into the sequence of random variables. So at this point, you have a sequence of random variables, rather than a waveform.

Why is it so important to have something countable instead of something uncountable? I mean, mathematicians understand, in a deep mathematical sense, why that's so important. For engineers the argument is a little bit different. I think for engineers the argument is, no matter how small the interval of a waveform you're looking at, even if you shrink it, no matter what you do with it, you can't approximate it in any way and get rid of that uncountably infinite number of random variables. As soon as you represent it in terms of an orthonormal expansion though, at that point you have represented it as a countable sum. As soon as you represent it as a countable sum, you can approximate it by knocking off the tail of that sum. OK, and at that point you have a finite number of random variables, instead of an infinite number of random variables. We all know how to deal with a finite set of random variables. You've been doing that since you were in 6.041 OK. So if you have a countable set of random variables, the way to deal with them is always the same. It's to hope that they're defined in such a way that when you look at enough of them, all the rest of them are unimportant.

That sort of is the underlying meaning of what countable means. You can arrange them in a sequence, yes. But like when you count in school, after you name the first hundred numbers you get tired of it, and you realize that you understand the whole thing at that point. OK in other words, you don't have to count up to infinity to understand what it means to have a countable set of integers. You all understand the integers. You understand them intuitively. And you understand that no matter how big an integer you choose, somebody else can always choose one bigger than you've chosen. But you also understand that that's not important. OK?

So this is the way we usually define a stochastic process, a random process to start off with. The other thing that we have to do, is given a random process like this, we very often want to go from this random process to other random processes which are defined in terms of this random process. So almost always we start out with a random process which is defined this way. And then we move from there to various other processes which we define in terms of this. So we never really have to deal with this uncountably infinite number of random variables. If we really had to deal with that, we would be in deep trouble. Because the only way we could deal with it, would be to take five courses in measure theory. And after you take five courses in measure theory, it's not enough. Because at that point, you're living in measure theory. And most of the mathematicians that I know can't come back from measure theory to talk about real problems. So that every time they write a paper, which looks like beautiful mathematics, it's very difficult to interpret whether it means anything about real problems. Because that's where things get hard. OK so the point is, if you define random processes in this way, then you always have some sort of grounding about what you're talking about. Because you have a countable number of random variables here. You know that if you're dealing with anything that makes any sense, only a finite number of them are going to be important. The thing you don't know is how many of them you need to be important. OK, so that's why we take an infinite sum here. Because you don't know ahead of time, how many you'd need to deal with.

OK, so we then started to talk about stationarity. The process Z of t is stationary if Z of t sub 1 up to Z of t sub k and Z of t sub 1 plus tau up to Z of t sub k plus tau have the same distribution. OK, so you take any finite set of random variables in this process, you shift them all to some other place, and you ask whether they have the same distribution. How do you answer that question? God only knows. I mean, what you need to be able to answer that question is some easy way of finding those joint probabilities. And that's why you deal with examples. That's why we deal with jointly Gaussian random variables. Because once we're dealing with jointly Gaussian random variables, we can write down those joint distributions just in terms of covariance matrices. Well it means, of course, if you want to deal with things that are not a zero mean. And after you learn how to deal with covariance matrices, at least you're dealing with a function of a finite number of variables. So it's not so bad.

OK so the argument is, this is what you need to know if you're going to call it stationary. But we have easier ways of testing for that. And in fact this Wide Sense Stationary we said, if the covariance matrix, mainly the expected value of Z of t sub 1 times the expected value of Z of t sub 2 is equal to some function of just t sub 1 minus t sub 2. In other words, where the expected value of this value times this value is the same, if you shift it over by some amount. OK, you see the difference between? I mean the difference between these two things is one, that in the definition for stationarity, you need an arbitrarily large set of random variables here. You shift them over to some other point here. And you need the same joint distribution over this arbitrarily large set of random variables. When you get into dealing with the covariance matrix, all you need for Wide Sense Stationarity is you only have to deal with two random variables here, and the shift of those two random variables here. Which really comes out to a problem involving just two random variables t sub 1, t sub 2, and the fact that it's the same no matter where you shift the t sub 1 to. It's only a function of the difference between t sub 1 and t sub 2. So it's a whole lot easier.

And then we pointed out that since jointly Gaussian random variables, and since the Gaussian random process, it's completely determined by its covariance function, a zero mean Gaussian process. If you'll forgive me, I'm not going to keep coming back and saying that. I will just be thinking solely today in terms of zero mean processes. And you should think solely in terms of zero mean processes. You can sort out for yourselves whether you'd need a mean or not. That's sort of a trivial addition to all of this.

OK, so it's Wide Sense Stationary. Well we already said that. Well, here I put in the mean. OK, Wide Sense Stationary implies that it's stationary for Gaussian processes. OK, so in other words, this messy looking question of asking whether a process is stationary or not, with all of these random variables here and all of these random variables here, becomes trivialized for the case of a Gaussian random process, where all you need to worry about is the covariance function. Because that's the thing that specifies the whole process. OK. That's part of what we did last time.

We set an important example of this. And you don't quite know how important this is yet. But this is really important because all of the Gaussian processes you want to talk about can be formed in terms of this process. So it's nice in that way. And the process is Z of t is the sum of an orthonormal expansion. But the orthonormal functions now are just the time shifted sinc functions. We know the time shifted sinc functions are orthogonal. So we just have this set of random variables. And we say OK, what we're going to assume here is that these random variables, well here I put in the mean. Let's forget about the mean. The expected value of V sub k times V sub i, namely any two of these random variables expected value of the pair, is equal to zero if K is unequal to i. OK, in other words, the random variables are in some sense orthogonal to each other. But let's save the word orthogonal for functions, and use the word correlated or uncorrelated here for random variables. So these random variables are uncorrelated. So we're dealing with an expansion where the functions are orthogonal, and where the random variables are uncorrelated.

And now, what we're really interested in, is making these random variables Gaussian. And then we have a Gaussian random process with this whole sum here. We're interested in making these have the same variance in all cases. And therefore what we have is process which is stationary. And its covariance function is just sigma squared times this sinc function here. There's an error in the notes, in lecture 16, about this. It will be corrected on the web. This is in the notes. This quantity here is left out. So you can put that back in if you want to. Or you can get the new notes off the web after a few hours or so. This is not put on the web yet. I just noticed yesterday. In fact I noticed it this morning.

OK, the sample functions of a Wide Sense Stationary non-zero process are not L sub 2. When I talk about a non-zero process I'm not talking about zero mean, I'm saying this peculiar random process which is zero everywhere. In other words, a random process which really doesn't deserve to be called a random process. But unfortunately by the definitions, it is a random process. Because with probability one, Z of t is equal zero everywhere. And that just means you have a constant, which is zero everywhere. So you're not interested in it.

OK, if you take the sample functions of any non-trivial Wide Sense Stationary random process, they dribble on forever. And they have the same-- no matter how far you go out in time-- the V sub k's which are the samples of this process, sample random variables of the process, weigh out. These things all have the same variance. So that this is stationary because it just keeps going on forever. OK and as we said last time, that runs into violent conflict with our whole idea here of trying to understand what L sub 2 functions are all about. Because what we would like to be able to do is stick the L sub 2 functions as sample values of these processes, have the waveforms that we send the L sub 2. When you add up two L sub 2 functions, you get an L sub 2 function. And therefore, all the sample values of all the things that happen with probability one are all L sub 2 functions. As soon as you do the most natural and simple thing with random processes, you wind up with something which cannot be L sub 2 anymore. But it's not L sub 2 in only a trivial way. OK in other words, it's not L sub 2 because it keeps going forever. It's not L sub 2 for the same reason that sine x is not L sub 2, or 1 is not L sub 2.

OK, and we've decided not to look at those functions as reasonable functions for the waveforms that we transmit or receive. But for random processes, maybe they're not so bad. OK. But we don't know yet. We started to talk about something called effectively Wide Sense Stationary, which the notes talk about quite a bit. Which to say, OK we will assume that the process is stationary over some very wide time interval. We don't know how wide that time interval is, but we'll assume that it's finite. In other words, we're going to take this random process, whatever it is, you can define it, a nice random process this way. And then we're going to get our scissors, and we're going to cut it off over here and we're going to cut it off over here. So it's then, time-limited over this very broad band.

Why do we want to do this? Because again, it's like the integers which are countable. You don't know how far out you have to go before you're not interested in something anymore. But you know if you go out far enough, you can't be interested in it anymore. Because otherwise you can't take limits. You can't do anything interesting with sequences. OK, so here the idea is. If you go out far enough, you don't care. This has to be the way that you model things because any piece of equipment that you ever design is going to start being used at a certain time, and it's going to stop being used at a certain time. And about halfway after when it stops being used, the people who make it will stop supporting it. And that's to hurry on the time at which you have to buy something new.

I bought a new computer about six months ago. And I find that Microsoft is not supporting any of the stuff that it loaded into this damn computer anymore. Six months old, so you see why I'm angry about this matter, of things not being supported. So in a sense they're not stationary after that. So you can only ask for things to be stationary over a period of six months or so. OK. But for electronic times, when you're sending data at kilobits per second or megabits per second, or gigabits per second as people like to do now, that's an awful long time. You can send an awful lot of bits in that time. So stationary means stationary for a long time. The notes do a lot of analysis of this. I'm going to do a little of that analysis here in class again. Not because it's so important to understand the details of it, but to understand what it really means to have a process be stationary, and to understand that doesn't really destroy any of the L sub 2 theory that we built up.

OK. So the covariance function is L sub 2 in cases of physical relevance. That's the thing. That's one of the things we need. We want the sample functions also to be L sub 2 in cases of physical relevance. If you have a function here, this is just a function of time now at this point. There's nothing random about this. It's just a statistic of this random process. But it's a nice well-defined function. We're talking about real random processes here. So what can you say about this function? Is it symmetric in time? How many people think it must be symmetric? I see a number of people. How many people think it's not symmetric? Well it is symmetric. It has to be symmetric because its expected value of Z of t sub 1 times Z of t sub 2, and if you flip the roles of those two, you have to get the same answer. So it is symmetric. It is real.

We're going to talk about the Fourier transform of this before trying to give any relevance or physical meaning to this thing called spectral density. We'll just say this is the Fourier transform of the covariance function for a Wide Sense Stationary process. So we have some kind of Fourier transform. We'll assume this is L sub 2 and everything. We won't worry about any of that. So there is some function here which makes sense. Now if k is both real and symmetric, what can you say about its Fourier transform? If you have a real function, what property does this Fourier transform have?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: It's conjugate symmetric, yes. And furthermore, if k is symmetric itself, then you can go back from this thinking you have a symmetric transform, and realize that this has to be real. OK. So spectral densities of real processes are always both real and symmetric. We will define what we mean by spectral density later for complex processes. And spectral densities are always real. They aren't always symmetric if you have a complex process. But don't worry about that now. There are enough peculiarities that come in when we start dealing with complex processes, that I don't want to worry about it at all.

So spectral density is real and symmetric. And so far it's just a definition. OK, but the thing we found about this sinc process, is that the sinc process in fact, is a stationary process. It's a Wide Sense Stationary process for whatever variables you want to put in here. And if these variables are IID and they're Gaussian and zero mean, then this process is a zero mean, stationary Gaussian random process. And it has a spectral density. This turns out to be a sinc function. The Fourier transform of it is a rectangular function. So this process has a spectral density which is constant out to a certain point, and then drops off to zero. So it's nice in that sense. Because you can make this be flat as far as you want to, and then chop it off wherever you want to. And we'll see that when we start putting a process like this through filters, very nice things happen. OK.

So we're familiar, relatively familiar, and conversant with at least one Gaussian random process. OK, we talked about linear functionals last time before the quiz. And we said a linear functional is a random variable, V, which is simply this integral here. We've talked about this integral a lot where Z of t is a function, and g of t is a function. If the sample values of Z of t are L sub 2, then this integral is very well-defined. It turns out, also, that if g of t is L sub 2, and these sample values are bounded, then all of this works out very nicely also. And we'll find other ways to look at this as we move on. OK but what this means is that for all sample points in this entire sample space, in other words we're dealing with a probability space where we have a bunch of processes running around, we have data coming in. We're transmitting the data. We have data that gets received. We're doing crazy things with it. All of this stuff is random. We might have other processes doing something else. Big complicated sample space here for all of the elements in it, the sample value of the random variable, V, is just the integral of the sample value of the process here. In other words for each omega, this process is simply a waveform. So it's this waveform times the function g of t, which we think of like that. So if g of t is time limited in L sub 2, if g of t is time limited as far as these sample functions are concerned, we don't give a fig about what Z of t is outside of that interval. And if we're going to only deal with linear operations on this process where those linear operations are constrained to some interval, then we don't care what the process does outside of that interval. And since we don't care what the process does outside of that interval, we can simply define the process in terms of what it's doing from some large negative time to some large positive time. And that's all we care about.

And so we wanted to find effective stationarity, as something which obeys the law of stationarity over that interval. And we don't care what it does outside of that interval, OK? In other words, if Microsoft stops supporting things after six months, we don't care about it. Because everything we're going to do we're going to get done within six months. And if we don't get it done within six months we're going to have a terrible crash. And we're going to throw everything away and start over again. So it doesn't make any difference at that point.

OK, so we have a process now we'll assume is effectively stationary within two time limits. And our definition is if the covariance function, which is a function of two values, t and tau, namely it's the expected value of Z of t times expected value of Z of tau. And our definition of effective stationarity is that this is equal to a function of t minus tau, whenever t and tau are in this box. OK, and we drew figures last time for what that meant. We drew this box, you know, bounded by T sub 0 over 2. And what this effective stationarity means, is that on all these diagonal lines, the covariance function is constant on those diagonal lines. And that gives us the same effect whenever we're passing this process any time we're calculating the inner product of a sample value of the process with some function, which is contained within those limits. All we need to know about is just what the process is doing within minus T sub 0 over 2 to plus T sub 0 over 2. Nothing else matters. And therefore, something is effectively stationary within those limits means that we get the same answer here, whether or not it's stationary. Why is that important? It's important because the things that you can do, the simple things that you can do with stationary processes are so simple that you'd like to remember them, and not remember all these formulas with capital T sub 0's floating around in them. Because having a capital T sub 0 in it is just as crazy as assuming that it's stationary, because you never know how long it's going to be until Microsoft stops supporting its software. I mean, if you knew you'd never buy their products, right? So you can't possibly know. So you assume it's going to go on forever. But then, when we try to ask what does that mean, we say what it really means is over some finite limits, which are large compared to anything we're going to deal with that at all of these results hold. So eventually we're trying to get to the point where we can leave the T sub 0 out of it. But we're going through this so we can say, OK there's nothing magical that happens in the limit as T sub 0 goes to infinite. Mainly we're trying to really derive the fact that all these results can be established over a finite interval of time. And therefore you don't care.

OK. So if this covariance function, single variable covariance function, is less than infinity, what does that mean? What is a single variance, single value, covariance function of zero mean? It's the expected value of Z of T, times Z of T. In other words, it's the variance of the process at T for any T within minus T sub 0 to plus T sub 0, or T sub 0 over 2 to plus T sub 0 over 2. OK? So if that variance is finite, then you can just integrate over the sample values of the process, and you get something finite. In other words this is what you need for the sample functions to be L sub 2 with probability one. As soon as you have this, then you're in business with all of your L sub 2 theory. And what does your L sub 2 theory say? It really says you can ignore L sub 2 theory.

OK, that was the nice thing about it. That was why we did it. OK in other words, the whole thing we're doing in this course is we're going through complicated things sometimes so that you can know how far you can go with the simple things. And we always end up with the simple things when we're all done. OK, and that's the whole principle. I mean we could just do the simple things like most courses do. Except then, you would never know when it applies and when it doesn't apply. So, OK. So there we are.

Let's talk about linear filtering of processes. The notes don't do quite enough of this. So when I get to the place where you need it I will-- and some of it's on this slide-- I'll tell you. We're taking a random process and we're passing the sample waveform of that random process through a linear filter, a linear time and variant filter, so some other random process comes out. OK, and now the output at some time tau, is just the convolution of the input Z of t times this filter response h of tau minus t. And you can interpret this in terms of these sample functions the way we did before. But now we already sort of understand that. So we're just interpreting in terms of if there's a random variable V, which is the value of the process at time at epoch tau, which is given in this way. It's a random variable when you pass the process through the filter.

OK, if Z of t is effectively Wide Sense Stationary with L sub 2 sample functions, and if h of t is non-zero only within finite limits, minus A to plus A, what's going to happen? When you pass the sample waveforms through a linear filter, which is bounded between minus A and plus A, that linear filter cannot do anything with the inputs. Namely the output that comes out of there at some time, T, can't depend on the input anymore than A away from that output. OK in other words, let me draw a picture of that. Here's some time where we're observing V of t. And V of t can depend on Z of t only in this region here, Z of t minus A up to Z of t plus A. OK that's what the convolution says. That's what those filter equations say. It says that this output, here at this time, depends on the input only over these finite limits. And again remember we don't care about realizability here. Because the timing at the receiver is always different from the timing at the transmitter.

OK so what that says then is, what does it say? It says that we know that Z of t is Wide Sense Stationary over these big limits minus T sub 0 to t sub 0 from six months ago until six months from now. And this linear filter is non-zero only over one millisecond. It says that the process which comes out, is going to be Wide Sense Stationary over minus six months plus one millisecond to six months minus one millisecond. And that's just common sense. Because the filter isn't moving the process any more than that little bit.

OK. So V of t then, is going to be Wide Sense Stationary in L sub 2 within those slightly smaller limits. The covariance function is going to be the same thing that it was before if you had a completely stationary process, if you only worry about what's going on within those limits, T sub 0 minus A to T sub zero plus A. You can take that big mess there, and view it more simply as a convolution. Let's see, this part of it is a convolution of h of t with K tilde, which is the covariance, which is a function of one variable. And then it's convolved with-- and here I put in the complex conjugate of h, because it's easier to do it here. Because at some point, we want to start dealing with complex random processes. And for filters it's easy to do that. For linear functionals it's a little harder. So I want to stick to real processes. If we're dealing with filtering, I can simply define this covariance function as the expected value of Z of t times Z complex conjugate of tau. And when you do that, we're taking this integral, which is the convolution of h of t with K tilde. And then we're convolving it with h complex conjugate of minus t. Because the t gets turned around in there. And what that says when you take the Fourier transform of it, at least what I hope it says-- I'm not very good at taking Fourier transforms. And you people, some of you are very good at it. We get the spectral density of the process, Z of t times the magnitude of h hat of f.

Now this is the formula, which is not in the notes I just found out. And it's a very important formula. So you ought to write it down. It says what the spectral density is on the output of the filter, if you know the spectral density of the input of the filter. And it's a kind of a neat, simple formula for it. It says, you pass a random process. I mean, it's like the formula that you have when you take a waveform and you pass it through a linear filter. And you know it's kind of easier to look at that in the frequency domain where you multiply, than it is to look at it in a time domain where you have to convolve. Here things become even simpler because all we have to do is take the spectral density, which is now a function of frequency, multiply it by this magnitude squared, and suddenly we get the spectral density at the output.

Now what happens when you take this nice sinc filter, this sinc Gaussian process that we have, which is flat over as large a bandwidth as you want to make it. And you pass that process through a linear filter. What do you get out? You get a process out which has any spectral density that you want within those wide limits. OK, so just by understanding the sinc Gaussian process and by understanding this result, you can create a process which has any old spectral density that you want. And if you can create any old spectral density that you want, you know you can create any old covariance function that you want. And all of this is good for Wide Sense Stationarity so long as your filter doesn't extend for too far. Which says all of our theory works starting at some negative time going to some positive time, taking filters that only exist for a relatively small time. If it's Wide Sense Stationary and it's Gaussian, then everything works beautifully. You can simply create any old spectral density that you want. If you have some waveform that's coming in, you can filter it and make the output process look whatever you want to make it look like, so long as you have the stationarity property. Now you know the secret of why everybody deals with stationary processes. It's because of this formula. It's an incredibly simple formula. It's an incredibly simple idea. You see now we know something else. We know that it also applies over finite, but large time intervals.

OK. So just to spell out the conclusions once more, a Wide Sense Stationary process is Wide Sense Stationary after filtering. So if you start out with Wide Sense Stationary, it's Wide Sense Stationary when you get done. If it's effectively Wide Sense Stationary, it's effectively stationary with a reduced interval of effective stationarity after you filter. OK in other words, if you start out with a process which is effectively stationary from minus T sub 0 to plus T sub 0, then after you filter it with any filter whose impulse response is limited in time, you get something which is effectively stationary within that six months minus one millisecond period of time. Yes?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: What?

AUDIENCE: Don't you differentiate [UNINTELLIGIBLE]

PROFESSOR: Effectively Wide Sense Stationary, thank you. I try to differentiate between them, but sometimes I'm thinking about Gaussian processes, where it doesn't make any difference. Is effectively Wide Sense Stationary with reduced interval after filtering. OK, good.

The internal covariance, in other words, the covariance function within these intervals, the spectral density, and the joint probability density aren't affected by this interval of effective stationarity. In other words, so long as you stay in this region of interest, you don't care what T sub 0 is. And since you don't care what T sub 0 is, you forget about it. And for now on we will forget about it. But we know that whatever we're dealing with, it's effectively stationary within some time period, which is what makes the functions L sub 2. We don't care how big that is. So we don't have to specify T sub 0. We don't have to worry about it. We just assume it's big enough and move on. And then if it turns out it's not big enough, if your system crashes when one of Microsoft's infernal errors come up, then you worry about it then. But you don't worry about it before that.

OK so now we can really define white noise and understand what white noise is. White noise is noise that is effectively Wide Sense Stationary over a large enough interval to include all time intervals of interest. Also on the other part of white noise, is that the spectral density is constant in f over all frequencies of interest. OK in other words, white noise, really you have to define it within terms of what you're interested in. And that's important to understand. If you're dealing with wireless channels, you sometimes want to assume that the noise you're seeing is actually white noise. But sometimes you do things like jumping from one frequency band to another frequency band to transmit. And when you jump from one frequency band to another frequency band, the noise might in fact be quite different in this band, than it is in this band. But you're going to stay in these bands for a relatively long time. So as far as any kind of analysis goes which is dealing with those intervals, you want to assume that the noise is white. As far as any overall analysis that looks at all of the intervals at the same time, in other words, which is dealing with some very long-term phenomenon, you can't assume that the noise is white. OK in other words, white noise is a modeling tool. It's not something that occurs in the real world. But it's a simple modeling tool. Because what do you do with it? You start out by assuming that the noise is white. And then you start to get some understanding of what the problem is about. And after you understand what the problem is about, you come back and say, is it really important for me that the noise is going to have a constant spectral density over this interval of interest? And at that point you say, yes or no.

OK. It's important to always be aware that this doesn't apply for times and frequencies outside the interval of interest. And if the process is also Gaussian, it's called white Gaussian noise. So white Gaussian noise is noise which has a flat spectral density over this effective stationarity period that we're interested in, over some very large period of time, minus T sub 0 to plus T sub 0. And over some very large frequency interval, minus W up to plus W or even better over some limited frequency band in positive and negative frequencies, the spectral density is constant. You can't define the spectral density until after you look at this effective period of stationarity. But at that point, you can then define the spectral density and see whether it looks constant over the period you're interested in. You can either see whether it is, or if you're an academic and you write papers, you assume that it is. OK. The nice thing about being an academic is you can assume whatever you want to assume.

OK let's go back to linear functionals. The stuff about linear functionals is in fact in the notes. This will start to give us an idea of what spectral density really means. So if I have a Wide Sense Stationary process, a linear functional, well this is what a linear functional is in general. But for a Wide Sense Stationary process, we have the single variance, single variable covariance function. The expected value of the product of two random variables of this sort is just this integral here. This is what we did before. If you don't remember this formula, good. It's just what you get when you take the definition of V sub i, which is the integral of g sub i of t with Z of t. And then you multiply it by V sub j. And you take Z of t and Z of tau. And you take the expected value of the two. And then you integrate the whole thing. And it sort of comes out to be that, if I've done it right. You can express this in this terms, namely as the integral of g sub i of t times a convolution of this function times this function. You can then use Parseval's identity. Because this convolution gives you a function of t, OK. When you integrate this over tau, this whole thing is just a function of t. OK. So what we're dealing with is the integral of a function of t times a function of t. And Parseval's relation says that the integral of a function of time, times a function of time is equal to the integral of the Fourier transform times the Fourier transform. OK. You all know this. This is almost the same as a trick we used with linear filters, just slightly different. So that says that this expected value is equal to this product here. And here I've used the complex conjugate for the Fourier transform of g sub j, because I really need it there if g is not a symmetric function.

OK so if these Fourier transforms, g sub i of t and g sub j of t are non-overlapping in frequency, what does this integral go to? And this is absolutely wild. I mean if this doesn't surprise you, you really ought to go back and think about what it is that's going on here. Because it's very astonishing. It's saying that stationarity says something you would never guess in a million years that it says. OK it says, that if this and this do not overlap in frequency, this integral is 0. OK in other words, if you take a linear functional over one band of frequencies, it is uncorrelated with every random variable that you can take in any other band of frequencies. So long as the function is Wide Sense Stationary, this uncorrelated effect takes place. If you're dealing with the Gaussian random process, it's also stationery. Uncorrelated means independent. And what happens then, is that expected value of V sub i and V sub j is zero. V sub i and V sub j are statistically independent random variables. And it says that whatever is happening in one frequency band of a stationary Gaussian process is completely independent of what's happening in any other band.

Now if any of you can give me an intuitive reason for why that is, I would love to hear it. Because I mean it's one of those-- well after you think about it for a long time, it starts to sort of make sense. If you read the appendix in the notes, the appendix in the notes sort of interprets it a little bit more. If you take this process, if you limit it in time and you expand it in a Fourier series, what's going to happen to those coefficients in the Fourier series? If the process is stationary, each one of them, if you think of them as being complex coefficients, the phase in each one has to be random and uniformly distributed between zero and 2 Pi. In other words, if you have a phase on any sinusoid, which is either deterministic or anything other than uniform, then that little slice of the process cannot be Wide Sense Stationary. And there's nothing in any other frequencies that can happen to make the process stationary again. OK in other words, when you expand it in a Fourier series, you have to get coefficients at every little frequency slice, which have uniform face to them. In other words, the Gaussian random variables that you're looking at, are things that are called proper complex Gaussian random variables which have the same variance for the real part as for the imaginary part. When you look at them as a probability density, you find circles. In other words, you find circularly symmetric random variables at every little frequency interval. And that's somehow, what stationarity is implying for us. So it's a pretty important thing. And it doesn't just apply to white noise. It applies to any old process with any old spectral density, so long as we've assumed that it's Wide Sense Stationary.

OK. So, different frequency bands are uncorrelated for Gaussian Wide Sense Stationary processes. Different frequency bands are completely independent of each other. OK, as soon as you have stationary noise, looking at one frequency band to try to get any idea of what's going on in another frequency band, is an absolute loser. You can't tell anything from one band about what's going on in another band. So long as you're only looking at Gaussian noise If you're sending data which is correlated between the two bands, then of course it's worthwhile to look at both bands. But if you're sending data that's limited to one frequency band, then at that point, there's no reason to look anywhere else. We're going to find that out when we start studying detection. It'll be a major feature of understanding detection at that point. OK, and one of the things you want to understand with detection, is what things can you ignore and what things can't you ignore. And this is telling us something that can be ignored. It's also starting to tell us what this spectral density means. But I'll interpret that in the next slide I hope.

OK if you take this set of functions g sub j of t, in the notes I now convert that set of functions to V sub j's instead of g sub j's. If I start out with that set of functions these-- oh dear, this is a random variable-- V sub j is the component of the process in the expansion, in that orthonormal set. So in other words, you can take this process, you can expand it into an orthonormal expansion using random variables and these orthonormal functions. If g sub j of t is a narrow band, and S sub Z of f is constant within that band-- in other words we don't need white noise here. All we need is a spectral density which changes smoothly. And if it changes smoothly, we can look at a small enough interval. And then what happens? When we take the expected value of V sub j squared, what we get is this times this times this-- OK the product of these three terms and since this-- that's supposed to be a j there.

OK so what's happening is these are narrow band, this is just a constant within that narrow band. So we can pull this out and then all we have is the integral g sub i of f times g sub i complex conjugate of f, which since these functions are orthonormal, is just one. So what we find is that random variable there is simply the value of the spectral density. Now we don't use that to figure out what these random variables are all about. We use this to understand what the spectral density is all about. Because the spectral density is really the energy per degree of freedom in the process at frequency f. In other words, if we use orthonormal functions which are tightly constrained in frequency, they just pick out the value of the spectral density at that frequency. S sub Z of f sub 0 is just sort of a central frequency in this band, or integrating over here, OK. So the interpretation of the spectral density is that it's the energy per degree of freedom in any kind of orthonormal expansion you want to use, so long as those functions are tightly limited in frequency.

So that's what spectral density means. It tells you how much energy there is in the noise, at all these different frequencies. It tells you, for example at this point, that if you start out with a sinc Gaussian process like we were talking about before, pass it through a filter, what the filter is going to do is change the spectral density at all the different frequencies. And the interpretation of that which now makes perfect sense when you're going through a filter, is the amount of energy in each of those frequency bands is going to be changed exactly by the amount that you filtered it.

OK so we have a nice thing there. S sub Z of f is the energy per degree of freedom in the process at frequency f. For a Gaussian Wide Sense Stationary process, this is the energy per degree of freedom at f. And the noise at all different frequencies is independent. That's what we were saying before. And after filtering, this independence is maintained. In other words, you pass a Gaussian random process through a filter and you still have independence between all the different frequency bands. It's an absolutely remarkable property. Just this property of the process acting the same at each time as it acts at each other time, that this leads to this amazing property of things being independent from one frequency to another is really worth thinking about. And it's one of the things that people use when they actually design systems. I mean it's almost natural to use if you don't think about it. If you start thinking about it then it becomes even more worthwhile. OK should I start detection? Let me just start to say a few things about detection, then I want to stop in time to pass the quiz back. OK let's talk about where detection fits in with our master plan of what we've been doing. We started the course talking about what was over on this side of the channel business, namely all the source coding and all of those things. We then started to talk about how do you do signal encoding. In other words, how do you map from binary digits coming in, into signals. Well we never talked about this problem. We only talked about this problem, when you didn't have any noise. We carefully avoided dealing with this problem at all. Then we talked about baseband modulation for PAM and for QAM. For PAM it was real, for QAM it was complex. Then we talked about how you go from baseband frequencies up to passband frequencies. Now we've been talking about what happens when you add white Gaussian noise, or any other noise at passband. Then you go from passband down to baseband. When I do this, we're going to have to come back at some point and deal with the question of what happens when you take passband white Gaussian noise and convert it down to baseband. Because some kind of funny things happen there. And we have to deal with it with a little bit of care. But let's forget about that for the time being, and suppose everything works out all right.

Then we go through our baseband demodulator. Finally something comes out of the baseband demodulator, which we hope is the same as what came out of the signal encoder. But what we're trying to do at this point now, is when the noise is here to say, this is in some sense, going to be what we put in plus noise. And for the time being, in terms of looking at detection, we will just assume that. We will assume that this thing down here is what went in, plus noise. And assume that all this process of passing white Gaussian noise through this junk gives us just a signal plus noise.

OK, so what a detector does is it observes a sample value of this random variable V, or it observes a vector, or observes a random process. It observes whatever it's going to observe. And it guesses the value of another random variable. And we'll call the other random variable H. We had been calling it other things for the input. It's nice to call it H because statisticians always call this a hypothesis. And I think it's easier to think of what's going on if you think of it as a hypothesis type random variable. Namely the detector has to guess. If we have a binary input, it has to guess at a binary output. It has to flip a coin somehow and say, I think that what came in is a zero, or I think what came in is a one.

The synonyms for detection are hypothesis testing, decision making, and decoding. They all mean exactly the same thing. There's no difference whatsoever between them, except the fields that they happen to deal with. And each field talks about it in a different sense. The word detection really came mostly from the radar field where people were trying to determine whether a target was there or not. And hypothesis testing has been around for a great deal longer. Because all scientists from time immemorial have had to deal with the question of have you collect a bunch of data about something. And it's always noisy data. There's always a bunch of junk that comes into it. And you have to try to make some conclusions out of that. And making conclusions out of it is easier if you know there's only a finite set of alternatives which might be true. In a presidential election, after you know who the candidates are, it's a binary decision. The people who try to guess who people are going to vote for, are really doing binary decision making. They don't have very good probability models. And it certainly isn't a stationary process. But in fact, it is a detection problem, the same as the detection problem we have here.

So all these problems really fall into the same category. We're fortunate here at having one of the cleanest probabilistic descriptions for detection that you will ever find. So if you want to study decision making, you're far better off trying to learn about it in this context and then using it in all these other contexts, than you are doing something else. If you go back and look at old books about this, you will be amazed at the philosophical discussions that people would go through about what makes sense for hypothesis testing, and what doesn't make sense. And all of these arguments were based on a fundamental misapprehension that scientists had until relatively recently. And the misapprehension that they had, was that they refused to accept models as taking an intermediate position between physical reality, and then you have models, and then you have doing something with a model. And you do something with the model instead of doing something with the physical reality. And people didn't understand that originally. So they got the analysis of the model terribly confused with trying to understand what the physical reality was.

In other words, statisticians and probabilists were both doing the same thing. Both of them were trying to simultaneously determine what a reasonable model was, and to determine what to do with a model. And because of that, they can never decide on anything. Because they can never do what we've been doing, which says take a toy model, analyze the toy model. Figure out something from it about the physical reality. Then go back and look at reality and see what you need to know about reality in order to make these decisions, which is the way people do it now I think.

Particularly decision theory is done that way now. Because now we usually assume both that we know Apriori probabilities, it's called, for these inputs here. For the binary communication problem, we usually want to assume that these inputs are equiprobable, in other words, if P sub 0 is equal to a half, and P sub 1 is equal to a half. And that's the Apriori probability of what's coming in. And we then want to assume some probabilistic model on v. Usually what we do, is we figure out what v is in terms of a conditional probability density, conditional probability density of v, conditional on either of these two outputs. You see for this model here, this is just made in heaven for us. Because this thing is an input plus a Gaussian random variable. So if we look at the conditional probability density of the output, which is signal plus noise, conditional on the signal is just a Gaussian random variable shifted over by the input.

OK. So we have these two probabilities, the input probability, this thing, which statisticians call a likelihood, which is the probability of this conditional on this. And in terms of that, we try to make the right decision. Now, I think I'm going to stop now so we can pass back the quizzes and so on. Before Wednesday, if you don't read the notes before Wednesday, spend a little bit of time thinking about how you want to make the optimal decision for this problem as we've laid it out now. What's surprising is that it really is a trivial problem. Or you can read the notes and accept the fact that it's a trivial problem. But there's nothing hard about it. The only interesting things in detection theory are finding easy ways to actually solve this problem which is conceptional trivial. OK.