Lec 7: Depth perception

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: All right, so today our topic is going to be depth perception, which, as I have mentioned to you before, is certainly one of the most intriguing achievements in vision. Because the impressions onto the retinal surface essentially two dimensional. And from that, somehow the brain needs to reconstruct the third dimension. And what is interesting about this also, that even in the most primitive animals, this is a must. And so annals with tiny brains also have mechanisms to be able to calculate depth from the information that comes in through their eyes.

And to demonstrate that, I have here a frog, which has a tiny little brain like that, has big eyes, and this frog, for its existence, needs to know exactly where things are in depth. Because if he doesn't, he would starve to death. And so what a frog does looks something like this, very crudely. He will stick out his tongue, grab a flying insect and consume it. And because of this incredible capability, it is a well-adjusted, healthy animal in most parts of the world.

Now the big question then comes up, how do we carry out these computations. What kind of mechanisms are involved in being able to compute where things are in space, either in absolute sense where it is from you and in a relative sense where one object is relative to another one. Now it turns out that this became such a serious problem, in the course of evolution, that actually several different mechanisms have evolved to make possible our ability to see things in depth.

And so when one looks at this as a list, as a fairly brief list, we can make a distinction between so called ocular motor cues and visual cues. The ocular motor cues are accommodation and vergence. So if various objects are at a very distance from you, your eyes converge or diverge. And you your lens gets thicker and thinner. And that information can be utilized in a rather crude way to tell you about where things are in that relative to you.

Now as far as visual cues are concerned, the very significant one we are going to talk about quite a bit, is a binocular cue, which is called stereopsis as you all know. And then we have a whole bunch of monocular cues motion parallax, shading, interposition, size, and perspective. And so we will talk about many of these to give you a sense of what it is like and to give you a sense of what various brain structures do with this as a result of extensive research that had been done in this area.

So now, first of all, let's talk about stereopsis. And we talk about stereopsis, we're going to talk about the basic facts of it, and then we are going to have some demonstrations. First of all, the so called stereoscope, of whichever modern version that has been handed out to you, the stereoscope was invented in the late 19th century. And when that was done, the initial approach to this was to be able to present to each eye separately an image that was taken by a camera that has two lenses, which are apart about as much as your two eyes are apart.

And each of those created a separate image of what's out there. And, of course, each eye gets a very slightly different perspective of what's there. And then when you present these two images that you had collected separately to each eye, you get a very strong sense of real depth, as you will see in just a minute. Now another way to do it, which nowadays is easier because you can barely ever find even one of these two-lens cameras, even in stores that sell ancient materials, antique stores.

So sometimes what you do instead, if you only want to take a picture of a static image, that you can take a camera, put in a track, and have it take two pictures in succession. And then you can do the same thing as you do with a serial camera. You can present one to each eye. OK, so, what we are going to do now, we are going to have a series of demos. And so we have a handout for each of you, the paper. And that you can keep and take home.

But the stereoscope that I have for each of you, that you're going to have to leave behind, because I need to use that in other classes. So what I want you to do then, there are two pictures on the first page that you put the stereoscope down onto the page so that the vertical line cuts it in half so that one goes into each eye. And then you put your head right down to it to look into it, all right?

And if you do that, if you have it properly sectioned, you're going to have a sense that that image is actually three dimensional. It's an ancient, ancient old picture on purpose. But you should still be able to see it in depth. So that's the initial thing. This became quite a parlor game and for that case, for many, many decades, whenever you went to a party, they would hand out to you a stereoscope, a handheld one, and they would show you all kinds of images.

And you can even do this today when you get on the internet to find such displays. Now, then a very important discovery was made. I shouldn't say discovery, really, I should say an invention was made by Bela Julesz who came up with a so-called random dot stereograms. By the way, don't look at the bottom one, that just tells you what it's going to look like. There's nothing to look at the bottom, the bottom set.

Now if you look in the middle set, that looks like a random dot stereogram. And the idea here was that the only cue that you provide is stereo cue, nothing else. It's pure. And so what can be done here, you can take a section here, or the same on each side, and simply move a few pixels, those images as a unit, over. And when you do that, they're going to stick out in depth. So now take you stereoscope and look in the middle display, and you look through it, you should see something sticking out in depth.

And the first question I'm going to ask you is how many of you can see something stick out in depth? What do you see? You see

AUDIENCE: A square.

PROFESSOR: A little square sticking out? All right, so now, don't try to look at the bottom one. That simply tells you what the procedure was in that center section where you see the square sticking out. The pixels were moved a few steps inward from both the left and to the right creating what is called a disparity. And that's what the brain then can calculate for depth.

So now to provide you with the acid test, go to the second page. Now you look at the second page, everybody see the letter on top? You don't even have to look through the stereoscope, obviously you see the letter E, right? That's because that section is made darker. But now if you do the same thing at the bottom, the only cue you have is the disparity cue. And the question comes up, what letter do you see there?

And let me just add, this can be used as a quick, general test. You can present these two subjects, and if it can present a whole bunch of different letters, and if they can see the letters, that means they can see stereo. If they cannot see the letters, then it looks like they may not see stereo. Now let me add one other fact here. As you move these progressively closer to each other like this, you increase the disparity. And that causes the image to be seen at increasing depths.

Our sensitivity is so great that when you look at this on a computer, a standard computer, if you move those images just one pixel from the left to the right, then from right to the left, you will see it in depth. And even monkeys can see as small step as one pixel. So now how many of you can tell me what was the letter at the bottom there?

AUDIENCE: H.

PROFESSOR: H, H, good. Anybody not being able to see the letter? Everybody sees it. Well, you guys are lucky. Because there are a significant number of people in the world who lack stereopsis, something like 5% to 10% of the population lacks stereopsis for a variety of reasons. We'll talk about that a bit more later. But one is sometimes you're born, and you're amblyopic in one eye. Sometimes you are strabismic, which means that your two eyes are not aligned, which in commonplace language is often called as being cross eyed or wall eyed.

Those types of people very seldom will have stereoscopic depth perception, even after it's corrected, especially if the correction is made by the time you're 8 or 10 years old, the correction won't help. It has to be done much, much earlier. All right, so that then is the very, very basics of the stereo procedures. And now another procedure that had been developed more recently is one which is called the auto stereogram.

So then if you go to the next page, and what you want to do is you want to look at this horizontally like that with the T on top. And then just look at it at sort of, I don't know, maybe about 20 inches from you, normal reading length. And what you want to do is to look beyond it. So stare beyond it. And if you keep doing that for a while, you will suddenly see an image, a three-dimensional image, as this comes actually from a book called The Magic Eye. There are several magic eye books in which all kinds of displays are done using these auto stereograms.

Does everybody see-- who can see what's sticking out? OK, what do you see?

AUDIENCE: Was it a shark?

PROFESSOR: You see a shark? OK. Now let me see if any of you don't see it. Keep staring at it. Look beyond it. Another thing that helps, if you look at it, bring it a little closer to you so you can look beyond it easier, gradually move it back and forth. And if you're patient, eventually you may be able to do this. The reason this is difficult is because you have to uncouple the vergence in your two eyes. You have to look beyond it slightly.

And, in fact, that is one of the reasons why testing people for stereopsis, an auto stereogram is not a very good procedure. Whereas virtually everybody can use a stereoscope without any trouble.

AUDIENCE: That's so cool.

PROFESSOR: Did you get it finally?

AUDIENCE: Yeah, that's so cool.

PROFESSOR: Yeah, all right so, if anybody is really interested in this auto stereograms is I say go to the store, the bookstore, and get one of those magic eye books. They're just a lot of fun. And you can just leaf through it. You don't even have to buy the book, just look through it at the store. And you'll see one interesting, clever image after the next. All right, so that's the stereoscope. And now let me explain to, I think I've mentioned this briefly before, the principles involved behind being able to see stereoscopic depth perception.

And what I've mentioned to you before was that if you have the two eyes fixating at a particular distance, if you then draw a circle around that, that's sometimes called a Veith-Muller circle, or the sometimes called a horopter, then any target, like this one here, will hit equivalent points on the retinal surface of the left and right eyes. However, if you do the same thing, and you put a target either beyond or closer than the Vieth-Muller circle, then they're going to hit nonequivalent points on the retinal surface.

So then by nonequivalency, we can do this and calculate this as to where the image falls relative to the central fixation spot in the foveola. And so then, when these nonequivalent points are hit, somehow the brain can measure this nonequivalence. And that is then converted into an estimate of where things are in depth. Now the idea behind this was that these nonequivalent points, that you have on the retinal surface, can connect in the cortex to single cells.

So they have a cell in the cortex that is binocular, by virtue of the fact that inputs from the left and right eyes. But they don't necessarily have to come from equivalent points. They can come from nonequivalent points. And that may be then the mechanism whereby it can tell you the degree of nonequivalence, and, therefore, convert that into depth. And, therefore, there could be single neurons in the brain that are selective to certain depths.

And so people began to do all kinds of experiments with this. And the way these experiments were done is you presented images separately to the left eyes and right eyes. And you could then present them to both eyes at the same time and vary the amount of disparity systematically to see what kind of tuning function you would get in the cortex. So this, some of the most beautiful work of this was done by a person called John [? Porgio ?] And I will tell you briefly about some of his experiments.

So here we go with then. We're going to look at the neural responses, neural responses in V1 as initially done in the monkey. So here's an example of a cell. We have here different degrees of disparity. And we have the neuron responding each time they're four repeated trials. And you can see the action potentials by these dark lines here. And what you do is you move the stimuli back and forth across the two eyes, the way it's actually done, you have a mirror. And then you have two in this experiment. You have two monitors, one to the left and one to the right. And then you can set it up almost exactly the same as what you would do with a stereoscope.

So this particular cell, as you can readily see, responds best when there's zero disparity. Now by contrast, here is a cell that responds vigorously at the far disparity and not to the close one. So then, when you do this, you can study hundreds of cells to see what kinds of distributions you have in the cortex for different degrees of disparity, cell activity. Now again, this hearkens back to what I talked about with respect to color vision where the question came up, if you want to see color, how many receptors would you need that peak at different wavelengths, right?

And one idea was that maybe you need as many photo receptors as there are colors. But in the end, it turned out that we have only three of them. And on the basis of that, we can recreate all the colors out there. Now the same thing applies to stereopsis. So when this was done systematically, here's an example of the tuning functions. This one here is the same, very much the same, as the first figure I showed you.

And so we have a bunch of different ones. And if you then study, as I've said hundreds of cells, you can come up with a distribution of this. And what John [? Porgio ?] came up with, he thought that there were four major classes. And the relative amount of activity of these four major classes then is used to compute all the very fine differences in depth. So there's a right on one. This cell is right on the fixation spot. And then you have near and far cells. And you have some in between cells.

Initially he thought there were four classes. Some people argue that there may be as many as six. But at any rate, there's a limited number on the basis of which you can calculate almost an unlimited number of depths, which is quite remarkable. All right, so now, what you can next turn to is to ask the question to what degree do various extrastriate areas contribute to stereoscopic depth perception.

And some people thought that this is a unique function for area MT, some people argue that maybe it's area V4, and so experiments were done in which it was examined to what degree stereoscopic depth deception is altered when you eliminate, say, area MT or you eliminate area V4. So that is what's been done. And you can think about it for a minute and say, well, what do you think? What do you think would happen in a monkey once you no longer had area V4? What you think would happen if the monkey no longer had area MT?

Well, the results were actually quite surprising. And they're shown here, same experiment as before. The Bela Julesz random dot stereograms. And then you're presenting one of four locations of however many little area where, like little square, that's sticks out in depth and you vary the amount of depth that sticks out by varying the number of pixels you moved the images into this place. And when that was done systematically, this is what was found. It was found that neither V4 lesion nor an MT lesion cause a significant deficit.

The only deficit that was significant had to do with a response latency. And as I should have mentioned earlier, like when we talked about the frog, one of the very important things about processing depth, again, is to be able to do it quickly. So when you have that's frog and the fly is flying along, he has to be very quick to compute it so that he can catch it, right, as you had seen. So in this case, what you see here, that there is about a 20 millisecond difference after V4 lesion, increase in latency, and quite a bit more, almost 40, 30, 40 after and MT lesion.

So that contributes to some aspect of depth processing in terms of being able to do it quickly. But neither the MT or V4 are unique in processing stereopsis. It looks like that it is processed in several different areas in the brain and inspire conjoint computation that you can arrive at the actual depth. And it's by virtue of that joint computation that you can do this very quickly.

So now you come to the next important depth cue, which is called motion parallax. This one is a capacity that we had acquired in the course of evolution, which is extremely potent and powerful. And it's based on a very simple physical fact. And the physical fact it that either when you are in motion, or something in the environment is in motion, the rate at which these images travel across the retinal surface is heavily distance dependent.

And so let me demonstrate this concretely for you. Here we have an eye that's fixed. It's always looking straight ahead. And we have two rods here, sorry, just one rod, that we are going to move into position gradually from here to here, and then back up as shown by the arrows here. And we examine the range over which these near and far and middle objects move across the retinal surface when you engage in this motion and the eye is stable.

And you can see that the far object moves over a much shorter distance than the near object. This you can readily do it yourself in an experiment. You can stick out your thumb and move your head back and forth. And you see that your thumb will move a lot more than the object that you're looking at.

So now the same thing also applies when you actually are engaged in the eye movements, which of course you do all the time. In this case, they eye is set up so that it's fixated initially on this object and then tracks it to here and then tracks it back. And when you do that, you get the same kind of effect, namely that the distance over which a far and a near object move is quite different. The near object moves over much, much greater distance than the far object, even though the eyes are tracking.

So this, then, being a basic physical fact, was then used in the course of evolution to create mechanisms that are sensitive to this differential motion. And, of course, because of the rate of motion also varies a little bit, it became possible to create mechanisms to make that computation to tell you where things are in depth. So here I'm going to show you an actual demo of this to make it clear to you.

In this case, again, we have a bunch of random dots, much like in the Bela Julesz random dot stereograms but just a single one. And everybody agrees there's no depth here. Is there any depth? Do you see any depth? So now what I'm going to do is I'm going to set this image into rocking motion. And when I do this, almost instantly, you're going to see something in depth. Are you ready?

So what you see here is are three levels, right, very clearly. In milliseconds, in 20 milliseconds, you can see this. And let me explain to you why you see this. If, OK, let me go back and do it again. If I keep this stable, you can see that the dots move over a great distance here, a lesser distance here, and practically not at all here. So there is a differential motion. And the greater the motion, the closer the image is in your analysis.

So that's called motion parallax. And then what you can do actually, you can play all kinds of games, do experiments in which you can present this kind of image. You can put this into each of the eyes separately. And you can present this image alone or you can present it paired with disparity for stereopsis. And you can do each separately or you can do the two together.

So let's now first summarize the essence of motion parallax. To derive depth information from motion parallax, neurons are needed that provide information about velocity and direction of motion and perhaps also about differential motion. Secondly, the majority of V1 cells are direction and velocity selective, as we had discussed before, and some appear also to be selective for differential motion, which I did not mention before. But, indeed, there are such cells in the visual cortex.

Now, the third important point is that such cells that are motion selective and direction selective and selective for differential motion are very, very common area MT. So those are some of the very, very basic facts. And now we can move on and ask what kind of brain activation occurs by stereopsis and motion parallax in normal and serial blind subjects using a recently developed technique, which is magnetic resonance imaging, functional, functional magnetic resonance imaging.

So how do you do this kind of stuff? Well, what you do here, here's an example, you have a very large stereoscope with a mirror at the end. And you have a subject who is lying down. And this whole unit, except not of course that part, is put into the magnet. And we have a magnet down here at MIT. Most of you probably have seen that. It's on the ground floor. So you can do this, and then you can present those images here.

And so the stereoscope will present two images and then you can vary this by rocking him back and forth either to present only motion parallax or present only stereopsis and to present both. And so now the question is, this is a very primitive question at this stage, where in the brain are these processes analyzed? And so you can find out what brain areas are active by doing this repeatedly collecting the fMRI data and then printing them out and looking at them to see what happens.

So I'm going to show you a couple examples of that. Here is the basic figure that the person sees but done in such a way that you can see it. Of course, he doesn't see anything like this. He just sees different depths. There are one, two, three, four, five, six, seven different depths here. And this rocks back and forth. And then you can, as I say, present this only with differential motion or you can present it only with disparity or you can present it with both.

And then finally, as a control, what you do is you can do the same thing. But you don't have any depth of any sort. You just have a flat surface rocking back and forth. And then when you do data analysis, you actually subtract the last one from the rest of the data so that you're not looking at the data for the activation just by the spots but for the activation that's specific for stereopsis or motion parallax.

So now if you do this experiment, here's an example of a normal subject and a stereobind subject. And we have here a sagittal cut adding up the images sideways. And what you see here, this is posterior cortex, of course. Here in the normal subject, when you present only motion parallax, you only analyze motion parallax. But you have a huge amount of activation in the visual areas.

And then if you do the same thing when you do a binocular stereopsis only, you also get a great deal of activation, in quite similar set of areas. And then the big crucial test comes up. What happens if you present the stereo under monocular conditions when you don't see stereo? And if you do that, using this same calculation procedures, there is no brain activation here. And, therefore, what we see here is due, indeed, to the analysis that we do for stereopsis.

Now if you do the same experiment in a stereoblind subject who has been tested, on tests similar we had shown you, when that person even looks at it under binocular conditions, there is no brain activation meaning that this person doesn't have any mechanisms in the brain to analyze stereopsis. Now the fortunate thing is that we have these several different mechanisms for depth perception.

And so people who are stereoblind and have no analysis for disparity, they can still see depth reasonably well. And, indeed, they can get driver's license and all that, because we have all these other mechanisms that include, that we have talked about, motion parallax. So that then is one way of looking at it. Now the other way to look at it, especially when I ask as well, are the same brain areas doing both or what.

And so what you can do is instead of doing a sagittal section, you can take sections coronally like bang, bang, bang, bang like that and see what that looks like. And here's an example in which I've isolated the stereo. And here are a bunch of sections. And this shows the activation for stereopsis. You can see there are all kinds of areas that are being activated. And then if you do the same thing and just look at the parallax alone, here we have the activation for that.

And then lastly here, we do both of them. Now we can go back. The way to look at the question, how are these two areas, these two types of depth perception analyses, differently activating in the brain. And so I'm going to go back and forth between stereo and parallax. You can see, and that, you can see the difference. Now some regions, which have a perfect overlap, and there's some regions that are quite separate.

Notable here are these areas here, which are activated by stereo but not by parallax. So this then can provide you with an initial idea that there are some brain areas in which both of these are analyzed together. And there's some brain areas in which are uniquely analyzed for either stereopsis or motion parallax alone. Now this tells you where it takes place in the brain. But how it takes place requires a totally different approach.

Namely, the most comfortably to record from individual neurons in various areas just like I had shown you that nice work done by John [? Porgio ?] recording from the one demonstrating there that there are disparity selective neurons that are tuned that then provide the hardware, if you will, for being able to analyze stereoscopic depth. So that then summarizes what I wanted to tell you about motion parallax.

And now we are going to go on and talk about yet another important depth cue that is utilized by the brain, which is called shading. Now remember that our ability to use light to illuminate things is something that was practically nonexistent for endless millions of years. And so because of that, both animals and us, we have to heavily rely on information based on light coming from the sun, coming from above.

And shading is based on those millions of years of evolution utilizing the fact that most of the light that illuminates things comes from above. So there are all kinds of nice examples of this. And here is one of them. What you can do here is you can take a bunch of disks and set them up so. You can do this on a computer to make the upper part light and the lower part dark or the other way around, the upper part dark and lower part light.

And all of you readily can see that these images, the first and third row, seem to be protruding towards you. And the images in the second and fourth row seem to be receding. Now that is because the brain is interpreting that on the basis of the fact that the light at least used to come predominantly from above. So that is the basic arrangement for seeing depth.

And now I'm going to give you some demonstrations to indicate that this cue is actually quite powerful, even when you would not necessarily expect it to be. So these shading cues have also been extensively used in art work to provide an impression of depth. And I will show you some examples that will give you a sense of how that is done. So let me make one more point before I proceed that namely it is, indeed, the degree of illumination that's crucial here.

We have the same change from red to, sorry, from some greenish to yellowish. And you have no sense of depth here whatsoever. In other words, you do need the shading information, meaning the amount of light that's being reflected from various surfaces, that is crucial for perceiving depth. So now what we are going to do is we are going to present a series of slides that will highlight the power that shading has for the perception of depth.

So here is an example of how we do this. And the reason I'm showing this in some detail because if you really are interested in stuff like this, you can do all this on your own computer. You can play games, endless games with it. You can spend hours and hours having a lot of fun thinking about how depth works on the basis of shading. So what you have here are a whole bunch of disks. And each of these can be shaded differently by many different computer programs.

So that's what you can do. That's the basics. So now what we can do is we can play a game. And we can say, present just two different objects here. But we're going to present them repeatedly on a big display. And then we can shade these differently as we please. So here is a whole bunch of them. And all the rows, this, the first, third, and so on rows are the same shape and the second, fourth and so on are the other shape, these two shapes.

So we only have two shapes here that are juxtapositioned. Now what we can do is say, well, this is a peculiar sensation. I have a vague sense that there's something maybe in the third dimension. But it's not too well defined, because this is not in accordance with the rules and laws of shading of light coming from above. So now what we can do instead, we can selectively shade these to be in accordance with the rules of light coming from above to create shading and depth.

And when you do that, here's an example of that. What you can see here is a very compelling image of these protruding elements, sort of protruding to the left, right? Everybody see, have a strong sense of depth here? So now what you can do is you can play with it and decide, well, can we do something that, keeping the very, very same shapes, shade them differently and see what it does to our perception of depth.

And so what we are going to do next is we're going to take each of these elements here, the same ones here, and we're going to reverse the contrast. You see the contrast here on top is white and the bottom is black. So we're going to reverse that contrast. And when we do so, the question is what are you going to see. And if you do that, low and behold, you still have a strong sense of depth. But it's a very confusing sense. You may see sometimes these objects pointing to the left and sometimes to the right.

It's unstable, because you're confusing those computations that have evolved over millions of years for interpreting depth in terms of shading. Now you can play also some additional games. You can make this even more complicated, make more changes, and here is another one. You still have a feeling of depth. But it's totally confusing. It's very hard. You can't organize it any way, because it is not in accordance with the law of light coming from above to a real object.

And lastly, you can also make this so that it would be in accordance with the laws. But you can change it around so that you get a completely different perception, a strong sense of depth. It's still the very, very same elements that you had seen before. But now the shading is done, again, differently. And then now gives you, again, a unified sense of a display that is not conflicting. Because in this case, it's in accordance with that some of the basic principles of shading.

So now, what we can do next, having talked about stereo and we have talked about shading, is to look at some more of the demos. So let's go back to the stereoscope. And let's go back to the handouts. And so if you now come to the next page that has a heading called stereo and shading. So, again, take the stereoscope and we are going to look at these in steps.

So let's start by looking at the top display first, which is called stereo only. So if you look at that, first of all, if you just look at it without the stereoscope, you see pretty much a sort of flat display of a truncated pyramid. Then if you put the stereoscope there and look through it, you should see, if you look at it for a little while, that one of those sticks out towards you and the other one seems to recede. Does everybody see that?

So let's stop there for a minute, because I want to add one more fact here, which I should have mentioned earlier. So what you do here is-- so you have these two displays like that. And you have one image here and another image here. When you-- this is greatly exaggerated, these coming together like this. That means it's going to stick out towards you as you look at it. But if you do the opposite like that, they're further apart than the rest of them. Then actually you see it receding.

And that's why, if you now take away the stereoscope and look at it, you can see that the top left image is in each, or facing towards each other, whereas the other ones are facing away from each other. And that's what creates, that's what the brain interprets, as protruding versus receding using the stereoscope. So that is very similar, in a way, to what happens with shading.

So now if you look at the second image there, first without the stereoscope, what you see here, again, is one that sticks out just like in the original display. And the rest of them are receding. That's because the shading, the one that sticks out is light on top and dark on the bottom. And it's the obverse for the other ones. Now if you do the same thing looking through the stereoscope, what you will see is still some degree of depth, but it's not very pronounced. Because there is no corresponding disparity information.

But now if you look at the third display, where stereo and shading are in harmony, then what you see is an extremely compelling dramatic sense of depth with the top left one sticking out towards you and the other three receding. So shading appeared to have added to the compelling nature of the depth that you see through the stereoscope. Now the last image in here is that we put stereo and shading and conflict with each other. And when you do that, you can look at it, first with just one eye, then with the other eye. When you look at it with both of them, you, for a while, you see something unstable.

And when you see it well, eventually, you realize that there is a conflict there because of the shading and the stereo being in opposition to each other. Now then, this kind of effect, if you go to the next page, we're going to go now to page five, six, and seven. Again, what you need to do here is to look at it sideways with the F's on top. And when you look at this, those of you who can use your eyes so that they are divergent, and you look beyond it, then this is very much like, or it is the same actually, as an auto stereogram.

So if you look at this for a while, and you look beyond it, eventually it's going to gel. And when it gels, what you should see is where the F's are, the images are protruding towards you, and the others are receding. Now it may take you awhile. It's much more difficult than what we just did with the stereoscope. But you should be able to see that. How many of you are able to actually see those images? A little bit di-- move it back and forth a little bit slowly. And maybe eventually you manage it.

So as I say, where the F's are you see these images-- these truncated pyramids protruding towards you. And the rest of them are receding. If you have difficulty seeing this, I'm not surprised, because it takes a lot of practice. But once you get a sense of it, I think that you will enjoy doing this and actually showing it to some of your friends.

So then if you go to the next page, there we have added shading. And the shearing is the same everywhere. But the stereo cues are not. Once again, what happens is that they are stereo cues where the F's are stick out towards you much greater than the others.

They stick out a lot less because of the added stereo. And then, in the last demo there, the last page, we are putting, just like in that figure with a stereoscope, we are putting them into opposition with each other. And so when you look at these, this would be very difficult to see for a while. Because there's a tendency to see it differently for stereo and for motion parallax.

And so it's going to be an unstable percept. So what you can do then is you can play around with this at your leisure and especially once you become more proficient looking at all the stereograms, if you go and get one of these magic eye books to look at, you will be able to see these displays as well. So now, this is the one was the first one that I showed you. As I said, this one, with the F's, are the ones that should stick out closest to you. Once you see that, then you can go on to the next to add the shading or subtract the shading from it.

So now, an interesting question that arises is to what degree are we able, or are animals able, to integrate these different kinds of depth cues. And in particular, in this case, you're going to ask the question, what about integrating stereopsis, parallax, and shading. So the experiment is one done on monkeys in which you can present these cues either singly or in combination.

And we can ask the question, well, does the monkey do better with one or the other, or does he integrate really and does really much better when you provide all three cues. And so here is a procedure. What you do here, again, you have a rocking display like this, and you can present this either with shading as shown here or with motion parallax where it rocks back and forth, and lastly also with stereopsis.

So if you do that, the results you get are quite dramatic. What happens is shown here as a percent correct performance and here is the latency in milliseconds. And it shows that the monkey does extremely well when you present-- this is percent correct. This is degrees of disparity. The monkey does extremely well when you present all three cues and does worse when you present each of those cues alone.

Even more dramatic is the fact, and this is-- I keep coming back to this, that the ability for us to respond quickly to things is very important for survival. And here what we can see is that when you present all three cues, performance is much, much, much, much faster than when you present each of those alone. And, of course, as you might expect, when you present parallax only, because that's motion over time, that takes the longest to do.

So even though motion parallax cues are great, it became important in the course of evolution to create mechanisms that can detect these things more quickly and more efficiently. So now we come to yet another cue that we know very little about at the level of the brain or single units, because it's so complicated, which is called perspective. But I want you to just be aware of it and have a sense of it.

And here is one of those cartoon examples that gives you a very strong sense of depth. And you almost cringe. If you were there, you would worry that you would be falling down. This is done strictly by virtue of perspective. It's very similar to what you encounter all the time when you're driving down a road and the road seems to converge, even though you're not aware of it. But that's what's happening on the rental surface.

Because things further away are smaller than things that are close by. And that's when you look down a railroad track, the same thing happens, even though you know that the railroad track is not converging, it's going parallel. But because of the distances involved, that's what falls on the retina. And you're smart enough to know that even though that's what falls on the retina, you can make the right kind of interpretation.

Conversely, you can also compute the depth on the basis of that kind of convergence. Now here's another example of that, a much simpler way that people can do with experiments. Here we have a bunch of dots. And we have two basic cues that have to do with perspective. One of them is this gradually decreasing size of these dots. I should say elongated disks, if you will, and also that they are converging much like a railroad track converges.

And so we have a very strong sense of having a third dimension here. Now the fact that this is so strong can be mitigated by adding a few things here. If you add some more dots, it's not question as dramatic. And then if you start mixing up the sizes, you are beginning to lose it. And then if you totally mix it up, then you have no sense of that left at all. So it is that progression of steps and sizes and whatnot that gives you the sense of the depth of the images that you're looking at.

Now here's another converse example of this that is an illusory effect that what you see here is three barrels, if you will. And this barrel is a lot bigger than this barrel, right, or is it? Well, so what we're going to do here, we have an inducing element here by this hallway, if you will, with a door at the end. And we're going to remove this hallway keeping the barrels exactly as they are. And if you do that, low and behold, those barrels are all the same size.

It's induced by virtue of the surround that gives you a false sense of depth. So now let me show you another picture because of the purpose behind this. This a picture that's in a museum in Worcester, Massachusetts. And it was created by a fellow called Edward Savage. And it's a pretty unpleasant picture. But the main reason I'm showing this to you is that there seems to be very poor sense of depth in this picture.

Now the reason this is interesting is because when artists began, centuries ago in the 13th, 12th centuries, draw things, they did not have a concept of an understanding of how to create depth, a third dimension, in their drawings. So what they did eventually, they came up with a so-called vanishing point, and they drew very much like what we had here. Lines that converged at a point and then scaled the images accordingly rather than keeping it the same size.

And that way you got a good sense of depth. So now that has a number of interesting stories about it that we are going to discuss next time you talk about the perception of shapes, OK, patterns. But I will leave that discussion until that. What I'm going to do next, however, I'm going to try to give you a sense of how important stereopsis can be for the perception of fine depths.

And so to do that, what I'm going--I'm going to show you actually a film. And here what we have is a so-called needle test. What you have here is a fine needle protruding. And here we have a bunch of different size circular openings, a little bit like a needle, but it's round. And the task is to take these one at a time and hang them up. And one can time how quickly you can do that, or you can make a film to see how well you can do it.

And then what we can do is we can test the subject under binocular conditions, and test them under monocular conditions. So I'm going to show you a film of this, actually two films. It will just take just few seconds to do it. OK, be ready, it's going to come up in a second. OK, here's the subject under binocular viewing conditions. So that's the condition under binocular viewing.

And now I'm going to show it to you, same subject, same time, but with one eye closed off. So that then just even looking at it without taking any careful measurements. It's obvious that it's much, much more difficult to thread a needle under monocular than under binocular viewing conditions.

And so what you can do is when you go home, and next time you want to sew something up, try threading the needle with one eye closed and with the two eyes open. And you will see immediately what a huge difference it is. And that difference, therefore, is due to you're having the mechanism of steropsis. Just a few seconds here.

Another test that has been used in a similar fashion which allows you to actually calculate exactly what your error is in reaching, you can have a subject sit in front of one of these touch panels, and then do this experiment either binocularly or monocularly. And after he presses this a dot comes up, and then the person has to touch it. And you have about 30 or 40 trials like that. And then you have recorded where the person touched. And, therefore, you can calculate the error between where we touched and where the dot is.

And then, again, you get a huge effect between monocular and binocular viewing conditions. Now when you come to monocular and binocular viewing conditions, another thing important to test is to what degree a person who does or does not have stereopsis is capable of integrating information between the two eyes. So to do that, we have here examples of what is called binocular integration.

So what we do here, again, you can use a stereoscope. You look at a monitor. And this represent the left eye, this is the right eye. And you flash these on. If you integrate this, this is what you see. This would be actually what you would show in the control part of the experiment. So you see the Star of David. And if a subject is shown this, and they don't see the Star of David, you worry that their ability to integrate the information between the two eyes is deficient.

And I would say 90% of the cases, those people who are deficient on this also show major deficiency in stereoscopic viewing. Now another way to do this is an experiment in which you can present two words here, sud and try, so the two are separate. And when you present them simultaneously, you actually see the word sturdy. So you ask the subject, please tell us what do you see. What is it word you see.

And the subject says sud. And the subject says try, then you know that that subject sees, if he says try, he sees mostly with his right and prefers it, doesn't see too well with his left eye. If he says sturdy, then he integrates the two. And, therefore, you can safely say that this guy has very good integration between the two eyes. So what I would like to do next then is to provide you with any questions that you have. This was a complicated topic, that you have, and then we are going to summarize.

Does anybody have a question about motion parallax, stereopsis, and so on? Let me maybe add one more important factor. Your eyes are separated only by so many centimeters. Now, can you think of an animal where there's a much larger separation?

AUDIENCE: Hammerheads.

PROFESSOR: The hammerhead shark. Yeah, that has a separation of over a foot between the two eyes. And so you could ask the question, why on earth did that animal evolve to such a huge separation between the two eyes? Well, that brings one to yet another interesting point. This I think may have started during the Second World War. It was realized that when you're flying over some territory, where there are all kinds of weapons and whatnot, which are well camouflaged, that just looking down at them, you can't see them.

But obviously if you're going to have a tank or you're going to have a gun or other that may be more like a cannon, it sticks out of the ground. So it was discovered that if you had in your airplane two lenses which are far apart, that would greatly magnify the depth. You could defeat that camouflage. And you could find those weapons down there by virtue of the fact they're sticking out of the ground.

So the fact then is that the more you separate the images from the two eyes, if you will, or your two cameras, the more likely it is that you can calculate the disparity of information between the two images. So that then is probably one of the reasons, not the sole reason, but maybe one of the reasons why, in some animals, is an excessive separation between the two eyes.

And that brings me to get me to yet another point, which is that stereopsis actually works best at relatively short distances, like threading a needle. It doesn't work too well beyond, I don't know, 10 feet or so. It becomes progressively less effective. But at short distance, it's very effective. And so I presume also many animals that have to hunt for food are able to utilize the mechanism of stereopsis, because everything is at a close distance when they hunt for food on the ground.

And by contrast, when you talk about motion parallax, that works extremely well over very long distances. So does anybody have any questions about motion parallax or about steropsis? Oh, once again, I'm crystal clear, huh? So, therefore, I think it's time for us to summarize what we had covered today. First of all, there are numerous mechanisms that have emerged for analyzing depth.

And they include the ocular motor cues, which are vergence an accommodation and then the binocular cue of stereopsis and then the monocular cues of parallax shading and perspective. Then you have several cortical structures that process stereopsis. You don't have one specific brain area that uniquely does this. The number of disparities that are represented in the brain, as studies in the area of the one by John [? Porgio, ?] is limited.

And maybe, maybe four, but there may be six, but certainly there are not a large number of them. And so it's analogous to the way things had been resolved for us to be able to process color. Utilizing motion parallax for depth processing necessitates neuron specific for direction, velocity, and differential velocity. Several areas getting V1 and MT process motion parallax, which I did not say.

But indeed, if you make a lesion in area MT, you go get a deficit in motion parallax, even thought you don't get a major deficit in steropsis. Now, area MT combines the analysis of motion parallax, depth, and flicker. However, these analyses are also carried out by several other structures as I've already said. And lastly, little is know at present about the manner in which information about shading and prospective are analyzed in the brain. And hopefully, that will be one of the future tasks by neuroscientists.

And so if any of you ever get involved in neuroscience, this certainly is a big open area that we hope people will start to analyze. So that then is the essence of what I wanted to cover today. And once again, if any of you has a question, please, please don't hesitate to ask. I'll be very happy to answer them. OK, lastly then, did everybody sign the attendance sheet? If not, please come up after the class and sign your name to it.

Very good. So next time then, we are going to talk about pattern perception. And hopefully you will find that also interesting.