Lec 9: Illusions and visual prosthesis

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: All right. Good afternoon, everyone. Today, we are going to have two topics. In the first half we are going to discuss visual illusions, and in the second half we are going to talk about prosthetic devices for the blind.

What I would like to start with is, again, to send around the attendance sheet, so please put your name to it. And then, in addition, I'm going to send around some handouts, one for each person. I'm going to hand them out in both directions. Thank you.

The handouts we're going to look at, I'll refer to them specifically as we go around. And I also will show pretty much the same handouts, figures in the handouts, on the screen. All right.

So for the first half of the presentation today we are going to talk about visual illusions. Now, the main emphasis of what I'm going to talk about, it'll be an attempt to convey to you what kinds of procedures one can use to get a better sense of why we see in illusions what we see.

Some of these approaches are strictly behavioral. And some of them have involved experiments carried out on a more thorough basis using electrophysiological procedures and so on. But those I'm not going to talk too much about. But I will relate what we can figure out through how the visual system works.

Now, because of that, there are hundreds of illusions, and I will emphasize those that allow us to think about why we see what we see. Now, one of these illusory effects is the so-called Hermann grid illusion, and that's the one I'm going to start with because it's a fascinating illusion.

If you look at this, what you see is that at the intersections you see sort of smudges. And so the big question arose as to why we see these smudges. Now, does everybody see those smudges? If you look at the handouts, I also show you-- pay attention only to the front page, the one on the left, which says the Hermann grid illusion.

So if you look at that either on your handout or you look at it on the screen here, you see these smudges at the intersections, not where you're looking at, but in the smudges other than where you're fixating. Does everybody see those smudges? OK.

So the question arose why we have these smudges. And back in the 1960s, an investigator came up with a rather clever idea of why we see these smudges. And the person who came up with this theory was a fellow called Baumgartner. And so what he proposed, that this illusory effect is not directly attributable to the center-surround organization or the receptive fields or retinal ganglion cells.

So what he proposed is that when a retinal ganglion cell falls in the center like this, there's more inhibition in the surround than when it falls along not the intersections, but along the straight lines. And because of that, there's more inhibition here than here, and that's why we see the smudges. Now, that was a very attractive theory. And in fact, it has appeared and still exists in many textbooks that you read about illusions and about vision in general.

Now, OK, so here is the exact description that I just made to tell you this. And of course, you would expect that if this is the case, you should also get the smudges if you'd reverse the contrast. And in fact, that is true. So here, once again, you see the smudges, maybe not quite as sharp. But in this case, it's whiter smudges, whereas in the previous case, this one, you have darker smudges. So that kind of fits with that basic idea.

But then, subsequently, people began to play around with various conditions to see when you do and when you don't get the smudge effect and to see whether that would fit with this theory. Now, that's the basic tenet in just about everything that psychologists and neuroscientists working in the visual system do. They try to critically examine various hypotheses. And doesn't have to be just illusions, it can be a whole bunch of other things.

And they do this then by systematically manipulating things in experiments that can present these things just under free viewing conditions, as I'm doing it in terms of these handouts, or you can do it under more controlled conditions. The subjects can look through a tachistoscope or something like that.

All right. So anyway, let's now look at this in a bit more detail and see what's wrong with this theory. Well, first of all, if you look at this figure, what you can see here that you can see the smudges irrespective of the size of the overall display. Now, that's a problem because, of course, the size of the receptive fields in the retina, in their center-surround arrangement, is constant.

So if it were the sharp arrangement that we described about the way the center and the surround are activated by the intersections and the non-intersections in the display, that should not apply to different sizes. It should only work then for a particular size where you approximate that. Now, I will elaborate on that in more detail in just a minute.

Now, another bug that arose here is that if you now turn this display just 45 degrees, all right, so that it's a diamond rather than a square, the illusory effect has declined considerably.

And then to make this even more problematic, and this appears in your handout on that backside of the first page, what you can see here is that when you make the lines arranged in such a fashion at slight angles to each other, then you don't see the illusory effect at all. Whereas here, we also see it a little bit less but still slightly visible. But here it's not visible at all. And you can compare that even better on the handout there.

So the question is, why would that be? Well, we give you another example here. If you make serrated edges-- I don't know if the people in the back can see the serrations. Those of you in front can see the serrations. And when you do that, also you fail to see it.

And the problem here, of course, is that if you make these slight changes in the angles of the intersecting lines or you make them serrated, the center-surround antagonism should be the same. And yet such a small change causes the illusory effect to disappear. So that then, in itself, questions the Baumgartner theory.

Now, there are several other factors that enter into this. This is a very subtle one. If you look at this side, OK, we have different shades for the lines. OK? On this side, we have the darkened lines, OK, in front. And in this case, they have them in the back. And here you see the illusory effect, and here you don't. And once again, how can that be if the illusory effect is due to the center-surround antagonism?

Even more dramatic is when you do this with colors. Here you have, again, the colored lines in front, and here you have them in the back. And here you can see homonymous shades, all right? This is darker yellow and so on. But it's the same color but a little bit darker at the intersections. So here you get a pretty good illusory effect. But once those white lines are in front, we get very little effect at all. So that then denotes a number of problems with the theory.

And then one more fact is that when you present the stimuli at isoluminance, you don't get an effect at all. So that then raised some serious questions about the illusory effect.

And then a quantitative study was undertaken to see just what is the layout of the receptive fields in this display. And so here what we have is a display. This square here is exactly five degrees from your fixation, OK, and you can calculate the number of midget and parasol cells that are activated by that square area.

And you can see in total there are more than 360 cells that are being activated in that little region. And here is a display, in this case, of only the ON cells, and you can double that by having the OFF cells in the same location. Saying that there's just no good correspondence whatsoever in the number of ganglion cells that are being activated and how the center-surround is truly laid out at a point where you get a very good smudge effect with this illusion.

So this then, taken all together, raises the issue then if indeed there's a retina phenomenon, why is there no fit here? And so it does look like that it's unlikely that this illusion can be explained by the center-surround antagonism theory.

So then another theory was advanced claiming that this doesn't take place in the retina, but instead that it takes place in the cortex. And so another theory that was advanced is saying that this happens in the cortex due to the fact that you have these simple cells-- we talked about those before-- that had these elongated inhibitory, supposedly inhibitory, surrounds.

And if one calculates this, that can explain the illusion a little bit better, especially since these cells, even at any given point in the visual cortex, come in several different sizes. So that's an alternate theory.

But still, further research needs to be done to be able to verify whether this theory is more valid than the center-surround theory in the retina, which certainly is totally wrong. So I think maybe in another 100 years, they're going to drop the Baumgartner theory out from textbooks and general reviews of illusions and vision.

OK. So now I want to show you one more illusory effect, which is a variant of the illusion that I have just shown you. And this one is something that can drive you nuts, I suppose. And that appears on the second front page, the second sheet, I should say, of the handout. You can look at it there, or you can look at it here.

And you can see this scintillating effect, and therefore it has been called the scintillating grid illusion. It was invented by fellow called Lingelbach. All right? That's a very strong illusory effect, very dramatic. And whether this can be explained by any particular theory about the visual cortex still remains to be settled. Right now I don't think there's a specific theory that can explain this.

Now, I will now move on and talk about yet another illusory effect, or actually a set of them, that has a much better explanation. This one we don't have a handout for. If you look at this display here, you'll agree that the left and right bars here are identical, right, going in shade from light to dark. All right?

So now what I'm going to do here-- what I want to do is to present these two images here. I want you to fixate in the fixation spot, and we'll count to about, I don't know, 20, say. Keep fixating. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. 1, 2, 3, 4, 5, 6, 7, 8, 9, 20. And then I'm going to go back here, and you can see that this side is much darker than that one. Everybody got that effect?

AUDIENCE: Yeah.

PROFESSOR: I'll do it once more. Pretty dramatic, huh? OK. Now I'll tell you what I'm going to do. What I want you to do is to look at this with one eye. OK? Don't look at it yet. Wait a while. Just let your eye get settled. You're going to look at it with one eye. And then when I switch over, I want you to switch eyes. OK?

All right, ready? Go ahead now and fixate, and we'll count to 20 again. 5, 6, 7, 8, 9, 10. 1, 2, 3, 4, 5, 6, 7, 8, 9, 20. Switch. And if you do that, you don't have an effect. OK? So what does that prove? Well, that obviously proves-- that's why it's nice to do these little, itty-bitty experiments one can do in minutes-- it proves that this is a retinal effect. And the effect is actually due to adaptation that we have talked about, when was it, a couple of times ago.

Now, I want to show you yet another effect that has a similar arrangement. What I'm going to do here is I'm going to put this display on. And what I'm going to do here is I'm going to turn these off successively as we go around the clock. OK? And what I want you to do, again, is to fixate here on the cross, OK, and then I'm going to start running it. And what are you going to see? A pretty amazing thing you're going to see. What do you see?

AUDIENCE: [INAUDIBLE].

PROFESSOR: Yeah? You see a green spot going around? Everybody see that? OK. So now the big question comes up, why on earth do we see this green spot? What is the explanation? Well, this again is a retinal effect. And to test this in more detail, we have to go back to what I told you about color vision.

And when I told you about color vision-- if you remember we did that experiment with Gaussians-- that if you look at the color circle, all right, and then you adapt to a stimulus here, which is pretty similar to what was on the screen, then the afterimage is on the opposite side cutting across the color circle.

So the afterimages that you see that are in color can be perfectly explained by the color circle. If you adapt to this, this is the afterimage. You adapt to this, this is the afterimage. It's always straight across the center line, center point, I should say, where you have the white center, which perfectly explains your afterimage.

So what I'll do again, I'll show this to you just once more. OK, here we go, ready? You can experience it again just to sort of get a sense. It's amazing how the green circle becomes more and more and more and more vivid.

Now, one more thing I'm going to tell you is when I turn this off-- keep fixating-- and then you're going to see a green afterimage, right? So it proves that, indeed, it's the afterimage that appears over a period of time with that rotating circle, and it's fully explained by the color circle.

All right. So now we're going to move on, and you look at yet another illusory effect. If you look at this display, what do you see here? You see a light square and a dark square, right? Everybody see that? Now, what if I told you that what you're seeing is a total misconception?

So I'll tell you what I'm going to do. I'm going to leave these two squares exactly as they are, but I'm going to change the background. Ready? What's the difference here? The difference is that now the background is uniform, right? And you can see these two squares are indeed identical in contrast, right?

But now if I go back to the previous one, if you look at it more carefully you can see this goes from light to dark. Of course, the projector makes this look bluish. It shouldn't be, but that's OK. And it's a gradual shift in the illumination level that fools you to think that one is light, the other is dark.

And that has to do heavily with the fact that the receptive fields of cells in the retina have center-surround organization, and they're sensitive to local contrast differences. That's their make-up. And so until I point it out to you, can't even tell that this is a gradual shift. And if it did not change in color, it would be harder to tell.

And it has to do with the fact that you're making local comparisons. And because of that, to make a distant comparison you are inaccurate about it. So this illusory effect, again, can be readily explained on the basis of what we know about the center-surround organization of receptive fields in the retina.

OK. Now, if you look at the same thing in color, which doesn't come up too well here because of the projector, but still you can tell that these are not the same color, OK, again because of the gradual shift, color shift, in the background. But exactly identical they are, as they had been before. So this works not only for contrast, but it also works for color. Now, some illusory effects don't work for color and some do.

All right. So now yet another related effect that I'm sure some of you have seen before is if you look at this circle, we agree that it's uniform in color. Now, if I cover half of it with dark and the other with light, it still looks pretty the same. Maybe it's a little darker here than here.

But then if I put a partition here, as you can see here, then this side definitely looks darker than that side. OK? So this, again, looks at contrast effects, and we'll talk about these contrasts in just a bit with more detail.

Next, let me turn to another illusory effect that you can see in many, many textbooks, which is called the wallpaper illusion. If you look at this-- I'll come to it in a second in your display, where it will appear on the back of the second page-- what you see here, it looks like that these are not parallel lines here. I shouldn't say lines, but parallel black and white squares.

It looks like that they're going like back and forth like that, OK, and it's quite dramatic. And so now we can ask the question, how can we see this? What is this due to? It could be due to the way they're aligned. It could be due to the fact that there is space between them, that gray space.

And so one can do experiments like the one you have on your next page there. Let me first-- I will prove that, indeed, we have parallel lines here. These are fully parallel lines, and they go back and forth. To convince you, you can see that the lines are identical, and yet they do not look parallel at all because of this illusory effect.

OK. So now what we can do is we can fiddle around with this a bit. Now, that's the kind of thing you guys would probably enjoy doing yourselves-- look at various illusions and start playing around with them. This is an example. This is a basic wallpaper illusion. And now you turn this into an equal checkerboard, OK? And if you keep the lines in between, and you get no effect whatsoever.

And then here we eliminate those gray bars, and the illusory effect is greatly reduced. So consequently, the illusory effect has to do with both the fact that you have space between the alternating black and white bars, meaning these gray lines here, and that you have them offset. So when it's not offset, you don't get it. And when you don't have the bars, you don't get it. So these two factors play an important role in giving rise to the illusory effect.

Now, what we also can examine is what happens at isoluminance. And in this case, once again, what happens is that the illusory effect is still there, but it is reduced under isoluminant conditions. And this is not the isoluminant actually because of the projector. But when it's perfectly isoluminant, then the illusory effect is largely lost.

All right. So now we have a variant of this illusion which is fascinating. And this one I will show you on the next page, OK? And if you look at just this display, and you are convinced that we have lines that are alternating that are filtered back and forth, right? Well, is that really true? Well, it turns out if you take a bunch of red lines here that are perfectly parallel, OK, and then you superimpose those on the display, you can see that actually the display is parallel. OK?

The reason it doesn't look parallel is due to the way these individual little pieces that you're not even conscious of are arranged. So let's look at that in more detail. Let me add here one more thing. At isoluminance, you get practically no effect. So these illusions, this one and the previous one, heavily depend on contrast differences.

OK. So here we go. Here's the basic illusory effect. And the way this was constructed is shown enlarged on top. Alternate lines have 1, 2, 1, 2, 1, 2, 1. OK? What you have in the top line, and the third, and so on, you have it in this orientation. Here you have them reversed. And that vague cue is what somehow provides the illusory effect for you.

Now, if you make all the lines similar, OK, in other words don't reverse the alternate lines, then what you see here is actually the bars are all tilted slightly upward. OK? And because of that, the illusory effect is not as dramatic as this one, but you still have it. All right?

So what then you can do with this is to enlarge on this effect as shown here. It looks like that you have somehow a very unequal set of lines, as a result of which you do not have a bunch of clear-cut squares. And if you, again, superimpose true parallel lines through it, you see that they are actually parallel.

All right. So that then explains that illusory effect in terms of how we somehow integrate information of shape that we are not even aware of to have a perception of a whole thing. Now, that's not much of an explanation. It doesn't tell you what neurons are involved or anything like that. But it gives you a general concept of how that illusory effect is created.

Now we come to yet another fascinating illusory effect, and that one is in the next few pages in the handout. So if you take the backside of page 3, you will see this illusory effect here. Does everybody see that? Now, this illusory effect is one which gives you a sense that you have a whole bunch of swirls. All right? Does everybody see those swirls?

Instead of seeing 1, 2, 3, 4 circles, you see some swirls. And you say what on earth is giving us that sensation? So what you want to do then is to look at it analytically. And what you see here is you have alternating black and white squares, first of all. And you say, well, is alternating black and white squares the central point?

But then if you look at it more closely, you can see this is-- I don't know how they came up with this one-- what you see here, if you look at each of these, that there's a gradual shift as you go around the circle in the orientation of the square. So this square becomes progressively, very slowly, more rotated. OK? And that goes around the clock, and that's true for every one of those circles.

And you can say, well, is it due to the black and white squares? Is it due to the change in the orientation of the squares that you're not even aware of? Or what? So what kind of experiment do you do to check this out, to come up with some understanding? What kind of game can you play?

Well, the first game you can play, so you can say, well, let's eliminate these slight changes in orientation. OK? And so what you can do instead, as shown on the next page, is that you have a bunch of circles, black and white circles, which are all the same.

And as you can see here, there's no swirls. You have four circles, very clear. So therefore, obviously, the squares, changing their orientation in a subtle fashion going around in the clock, is very important in introducing this illusory effect.

Well, then you can say, OK, but is that all of it? And so what we can do here, as shown on the next page, is we can maintain the orientation shift, but we can make them all the same contrast, as we have here. And when you do that, you have have a rather weak illusory effect. There's still a little bit left maybe, but it's kind of weak. And then if you instead make it isoluminant, you get some in-between effect.

So on the basis of that, we can conclude that very important in this display is that you have a gradual shift in the orientations of these squares. And that, in turn, suggests that somehow the way these squares activate orientation selective cells in V1 plays an important role in giving rise to this strange illusory effect.

OK. So now what we are going to do is to look at yet another illusory effect, which has a different source, and that one is the famous Muller-Lyer illusion. The fact is that these two bars here are identical in size-- do I have the picture-- like that. OK? But when you put these inducing elements, as we will call them, OK, on there, you'll swear that this one is longer than this one is. OK?

So now we can ask the question, what kind of manipulations can we undertake to see what that is all about? And let me come back to that in a minute. So what you can do here is to make the inducing elements and the bars different in color or in contrast, and you still get a strong illusory effect. OK? So making them isoluminant or making them opposite in contrast does not eliminate this illusion.

So the theory that somebody has come up with is shown here, which I'm not sure I buy, but that's nevertheless a theory, it claims that the brain interprets these two displays as differences in depth, as if here we had a protruding edge in a building inside or a receding edge as shown here.

And that difference that we almost unconsciously, if you will, interpret as being in different depths is what gives rise to the illusion according to this theory. Of course, there's a simpler theory simply saying that we just integrate these two bits of information.

All right. So now we can ask the question, if you have illusory effects of this sort, is this something that really takes place higher up in the brain, or does it take place in the retina where we cannot separate these inducing and testing elements?

Well, the way you can do this-- I'm going to show you another illusion which is similar to this. Here we have the so-called Ebbinghaus illusion. This is a very similar kind of illusion. This is a test element and an inducing element. The test element here are these two circles, and you swear that this one is a smaller circle than that one is, right? And we have a bunch of small circles, and here we have bunch of large circles.

So now we can ask, well, is this something that takes place in the retina or what? So what kind of experiment can you do? Well, the experiment that you can do is to present these displays interocularly. What does that mean? What that means is that you present this to the left eye and this to the right eye.

And you present this device that's used by psychologists for this kind of thing. It's called a tachistoscope. You look through it. It can present things separately through the two eyes, and you can flash them on briefly so you don't get any binocular rivalry.

And when you do that, then you can systematically vary the size of these to make sure that you can get a quantitative measure of it. And when you do that, you find that the illusory effect is just as good under interocular conditions as it is under binocular viewing conditions.

Because of this then-- this has been done with many kinds of illusions including the Muller-Lyer one. And when you do that, you show, in essence, that this is an effect that takes place most likely in the cortex, certainly not in the retina or the lateral geniculate nucleus. It takes place someplace where the images from the left and two eyes converge.

All right. So now yet another set of illusions that I want to show you. I think that these illusory effects are done. You can take this home and enjoy it or show it to your friends. Now I'm going to show you a set of just two kinds of motion illusions.

OK, here's the first one. I want you to look at the screen, and I'm going to flash something on briefly, and I'm going to do it repeatedly. Ready? What do you see? Do you see something moving? A very brief sense of motion that you see. This twirls a little bit. And this is called the snake illusion, which is really curious. This has, again, appeared in quite a number of textbooks, and we do not have a really good explanation for it.

But we have another one, which I think is related. OK. So what I'm going to do here, this is called the waterfall illusion. So what we are going to do here is I'm going to set this in motion. In this case, I don't want you to fixate.

What I want you to do is just keep looking at it in various places. You can look here, here, here, and here, and so on. OK? And I'm going to let it run for a while, and then I'm going to stop it. OK? And I want you to tell me what is your aftereffect. Ready?

OK. I'm about to stop it. Are you ready? Pay close attention to what you're going to perceive once it stops. Anybody see what happened? What did you see?

AUDIENCE: [INAUDIBLE].

PROFESSOR: You see the water running up here, right? OK. I'll do it once more so you can appreciate it. Pretty nice, huh? OK. So this illusory effect is one that also occurs interocularly. And it has been shown that when you record from area MT, we talked about the fact that area MT is a region where cells respond selectively to direction of motion.

Well, people have shown that if you, for an extended period of time, a minute maybe or whatever, you stimulate a cell in MT with this kind of a display, then after you stop it, OK, the cells with the opposite sense of motion will discharge some, which then is attributable to the upward sensation that you have. So it's clear enough that this is something that does take place predominantly by virtue of direction-selective cells and, probably, by direction-selective cells in area MT.

All right. So then what we are going to do here is we're going to look at some other limitations and ambiguities in perception. And in particular, we're going to examine some figure/ground relationships. The first one I'm going to show you is this one. If you look at these two diagonals, OK, a and b, they certainly look very, very different, don't they? OK? One looks shorter and a different angle.

But now if I eliminate the trapezoid in the surround, you can see that they're actually identical. OK? I'll show it to you again. Pretty dramatic. So that's an interesting illusory effect. And it's clear enough that this is something that takes place in higher cortical areas.

Yet another one here that I'm going to show you also has something to do with figure/ground relationships, which has been extensively studied, especially by Gestalt psychologists. And I spoke about this a little bit the last time. What you see here is a bunch of vases, so to speak, right? That's good enough.

So what I'm going to do now, I'm going to show you another set of vases. Are you ready? There is another set of vases. Everybody see the vases? Now, what if I tell you that there's much more to this than just vases? So if you look really closely, you're going to see something else there. What else do you see there?

AUDIENCE: Faces.

PROFESSOR: Oh, OK, two faces, two guys facing each other. Now, if I change the relative number of elements in here, which changes the figure/ground relationships, you get a different effect. Here you have-- predominantly you see the faces rather than the vases. OK? And that's a cute way of putting it. I see the faces, not the vases. And then you switch it, and you say, I see the vases, not the faces. All right?

So that then is a way you can play around, again, with various illusory effects by fiddling around on your computer and creating things like that.

OK. Now, yet another strange effect is where you can have difficulty telling what's what is this kind of a display. You say, well, what's happening here? Are arrows pointing to the right, or are they pointing to the left?

And that reminds one of a fellow called Yogi Berra. How many of you have ever heard of Yogi Berra? Oh, my goodness, very good. OK. So he was a famous catcher for the New York Yankees, right? Well, he was a very interesting guy, had all kinds of interesting statements. And one of those many curious statements, one of them is-- it doesn't relate to this. It just comes into my head. "It isn't over until it's over." That's one of his, OK?

This one here that's relevant is, "When you get to a fork in the road, take it." Yeah? And that's a Yogi Berra quote. And so what we have here is a sign that you come to a fork in the road, and you have to decide to take it. What are you going to do? Are you going to go this way, or are you going to go that way? So that's a kind of game you can play with this kind of strange effect that you can create that is confusing your ability to organize your visual percepts.

Another famous one, maybe not that famous, actually. Maybe some of you have seen this. This, again, has appeared many, many years ago in The New. Yorker. It says, "I'm turning into my mother." Now, keep looking at it and tell me, why do you think that that comment is being made? What we have here, a young woman looking at another young woman, right?

But now if you keep looking at this display, look. This is a nose here, and this is a mouth. You see an old lady now? OK. You played around with this, it gives you a confusing percept. And ultimately, you can see the young person looking that way or an old lady looking the other way. I mean looking not backwards but across. And that's why this statement says, "I'm turning into my mother."

So the essence here is that playing around with these kinds of illusory effect is a lot of fun, and we all very much enjoy looking at illusions. And some of us like to play around with them to try to figure out what their existence is for the reasons.

Now, I want to show you one more, which I think is very appropriate at this time. This is the famous Greek key motif. Everybody's seen this, I presume. Does everybody know where this is displayed, actually, at several places? In the Senate. OK? Whenever you have a picture of the Senate Building, you will see this. All right?

Now, that's a little bit amusing, actually, if you think about it. So this has become symbolic of democracy. And considering just what happened most recently, you could almost say that, perhaps, what's happening here is that you cannot tell whether we see this guy or this guy here going around. OK?

And one of them you could say, oh, these are the Democrats. These are the Republicans. And you have sort of a conflict as to which ones you see. All right? And that is, in a way, symbolic of what's going on today.

But to educate you a little bit, which I suspect probably most of you know already anyway, this Greek key motif actually has its source in the so-called myth of the Labyrinth that had imprisoned the Minotaur. Who knows who the Minotaur is? Oh, very good. Many of you do. All right.

So the Minotaur was a creature that was half bull and half human. All right? And what happened, according to this myth-- and let me tell you it's a myth, and it's a bit of risque thing. There was a king called King Minos whose wife Pasiphae-- Pasiphae, I'm not sure how you pronounce it properly-- is the one who had intercourse with a bull and gave birth-- sorry, had intercourse with a bull.

And as a result of that, gave birth to the Minotaur, which was a half-bull, half-human monster, if you will. All right? And that became so threatening and so irritating, I guess, to King Minos that he asked one of his subjugates, Daedalus, to create a maze, a huge maze, and he put the Minotaur in it. So it was such a maze that the Minotaur just couldn't get out of it. All right?

But then things got progressively worse, and King Minos decided that he had better get this Minotaur killed. And so he asked a person whose name is Theseus, OK, to go into the maze and kill the Minotaur.

Now, the king also had another offspring whose name is Ariadne-- nice name, Ariadne. And she said to Theseus that if you take a ball of thread and you walk through the maze, once you do what you have to do, you can follow the thread back, and that way you can get out of the maze. So that's what Theseus did, and then he went on and killed the Minotaur.

Now, the story didn't end there because somehow, for reasons I'm not sure I know the details anymore, King Minos had a fall out with Daedalus, and he jailed Daedalus and his son Icarus. And while they were in jail, they tried to figure out how they could escape.

And so Daedalus fashioned a pair of wings made out of bird's feathers and wax, and then put it on Icarus. And then Icarus, by doing this, he could fly. OK? And then he flew and escaped from the prison. But unfortunately, he flew fairly high up where the sun was very hot. That melted the wax, and Icarus, sadly, fell to his death as a result.

So that's quite the story. And that's something that obviously some of you already know. And those of you who don't, you will probably remember, as so many Greek myths are just fascinating, ancient, old stories. All right. So then here is actually a reconstruction of the kill of the Minotaur by Theseus. And this is actually a sculpture that exists in Greece. Pretty nice, huh?

OK. Now, the last confusing image that I wanted to show you is this one that had appeared-- I think this appeared also in The New Yorker, but I'm not 100% sure. What you have here is a confusion-- do we have three or two prongs here?

And the reason I show this sometimes in class is because in our department when it was originally formed by Hans-Lukas Teuber, he created what is called three prongs. OK? And then it became sort of an issue as to, well, what are the three prongs? Are there two prongs? Are there three prongs? Who belongs into which prong?

And so somebody came up with this picture that was put up as a joke in the department when the department was still in one of the old buildings that doesn't even exist anymore. All right. So that then brings me to the end of the illusory effects that I thought we'd talk about briefly.

And I will now move on and talk about visual prosthesis. All right. Now, as far as prosthetics are concerned, you're going to hear quite a bit about that from Chris Brown because in audition, an auditory prosthetic device, the cochlear implant, has become an incredible success. There are well over 50,000 people in the country now who have a prosthetic device like that that enables them to hear, to converse. It's really incredible.

Unfortunately, when it comes to visual prostheses, we have nothing comparable at this stage. It's a much needed device that will take many, many, many more years to create. But I thought I'd give you sort of a crude sense of how that kind of work progresses because there are many laboratories that are trying to come up with something that will provide an ability for people to see.

Now, I can tell you that in the world there are more than 40 million people who are blind. In the United States, there are more than a million individuals who are blind. So because of that-- that's a huge number-- it, of course, is extremely desirable to come up with some device to do this for you.

Now, this is a very, very long history, and people have tried all sorts of things. One of those is to try to induce the ability to see something in the world that you normally see with your eye by putting it through some other sense.

Now, a simple example of that is Braille. All right? Everybody knows what Braille is, right? That's you can use your hand to feel something that has protruding elements in it that can be shown in terms of letters. And so as you get into an elevator and you can't see, you can feel it, and then you can push the right button to get to the right floor. OK?

Now, that's one way it has been done. Another approach had been to present stimulation to your somatosensory system, either by putting something on your back or putting something on your fingertip, that an image would be converted into actually activating your somatosensory system that would be equivalent to something that you would perceive if you could see.

So that's the basic thing. And a tremendous amount of effort has been going on to try to come up with something to create a workable prosthetic device for the blind. Now, there's a lot of disagreement as to where such a device should be placed in the body and also a lot of debate of what kind of procedures should be used.

Now, some people advocate that it should be done on the retina. Some people advocate it should be done in the lateral geniculate nucleus. Actually, some work is being done at Harvard trying to come up with something for the lateral geniculate nucleus. And there's some people who think it would be best to do it in the visual cortex.

So let me sort of go into this in a bit more detail. And let me, first of all, raise the question of, what kinds of devices, and what kinds of necessities do we have for converting vision into something that enables blind people to see? Well, first of all, it's very important to be able to see some basic patterns, very important to see motion, and very important, often ignored, is that you got to be able to see something in the third dimension, because a person who is blind, he should be able to walk in the world. And if you're going to walk around, you should be able to see where things are in depth.

So those are the basic requirements. You're certainly not required to process color information. So that's not important. And there are several other things that are not important. But these are the three most important ones that you would like to have if you are going to create a prosthetics device for the blind.

Now, then the next question is, how are we going to deal with this, and what kinds of issues and problems are we going to deal with? First of all, the big issue is what kind of prosthetic device should we use. One of those that I've already mentioned is to try to convert visual impressions into other modalities such as your somatosensory system. All right?

The other is to create some sort of stimulating device, which there are various kinds. But one of those would be, for example, to put a bunch of electrodes into the brain, or into the retina, or whatever, and then selectively stimulate through this series of electrodes to mimic the visual field.

Now, the other one, and it is a big issue, what brain areas should be considered? And because we're at an early stage in this, I think it is desirable for people to pursue just about any area they think might work.

Now, the problem with a retinal implant-- there are two problems with it. One is the retina is extremely small, and very confined, and has these millions of receptors and ganglion cells, and how can you selectively activate them?

Now, the biggest problem, though, is that in most blind people, the retina becomes non-functional. And even worse, it degenerates in a relatively short time so that it no longer is useful for electrical stimulation or for whatever kind of stimulation you wish to engage in.

So because of that, some people have moved on to the lateral geniculate nucleus. That area has some problems because it's deep in the brain, and it's quite small. OK? And also, it tends to degenerate over time, although it takes it a lot longer than the degeneration of the retina. Thirdly, people have considered stimulating the visual cortex, and that's what I'm going to talk about in a bit more detail just now.

But the last thing that's very important here-- I think it's the last thing. Yeah, just let me go back. Ah, sorry. The third big issue is, can you create a device that has longevity? As soon as you put something into the brain and into the eye, there's going to be some sort of adverse reaction that may kill the cells at the tip of the electrodes or something like that. So all kinds of approaches have been created to try to increase the longevity of the device.

Now, I'm going to next look at what happens in the visual cortex, which is a promising, at least I think it is, a promising site where you could electrically stimulate and use that to create a visual impression. All right. If you do that-- here is a monkey brain.

The monkey is very well suited for this because, as I have mentioned before, the posterior part of the cortex-- see, this is area V1-- is lissencephalic. And so it's relatively easy to place accurately electrodes into it. OK?

So then if what you do here-- here, just a reminder, is the central sulcus. Here's the lunate sulcus. And here, of course, is area V1. If you now take the contralateral hemifield, that projects into this region. OK? This is shown out to about five degrees. That will go to around here like that. OK?

So now the next big question that, initially, had been done actually in humans, what do you see when you insert a microelectrode into the brain-- as I say, this was done on humans initially-- and you electrically stimulate? What do you see? And this work was done by Brindley many years ago. And he discovered that when you electrically stimulate in the human primary visual cortex, what you get is a small, star-like image. All right? So that's what he found.

And so that then triggered a great deal of research, including work in monkeys, to determine what could be done to create a prosthetic device that might just work. So the first step then was to try to determine what happens when you put a microelectrode in here.

This is, by the way, the layout of the visual field in the posterior cortex with much more area devoted to the fovea than into the periphery because, of course, there's a much higher packing density of receptor cells as well as retinal ganglion cells in the fovea representation than further out.

In the cortex, the thickness of the gray matter is constant. It's about roughly 2 millimeters thick. And the frequency, spatial frequency of the cells, is pretty constant as well. So therefore, you need to allocate more space to the incoming huge number of retinal ganglion cells projected to the geniculate and then to the cortex than for the periphery.

And that's why you have this kind of a layout of much more space allocated to the central than to peripheral vision. So that's a very important factor in considering how to create a prosthetic device.

So now the first step in doing this kind of stuff, if you're going to do serious experiments on monkeys, is to determine, what does the monkey see when you electrically stimulate the visual cortex? And so the experiment that's been done here is one in which a monkey has been trained to make an eye movement to the bigger of two visual stimuli or the brighter of two stimuli. In this case, we have two visual targets.

By the way, one of these is going to appear where the receptive field is of the cells that you're going to stimulate in the visual cortex. So then you may sometimes make this one brighter, this one brighter. And the monkey makes a saccade to it, and he gets a drop of apple juice for reward.

And then you can systematically vary the relative size of the two stimuli to see what kind of a function you get. And the same thing you can do for the relative brightness of it. And then on some trials interspersed with those trials, you present [INAUDIBLE] stimulation with the visual target and see which one does the monkey go to.

And then you systematically vary, as I said, the brightness or the size of those stimuli. OK? We vary the contrast in this case, and in this case, we vary the size. OK? So that's the basic experiment. And if you do that systematically, you can tell exactly what the size and the contrast is of the visual stimulus created by electrical stimulation.

If you do that-- here's an example. When you present two visual stimuli and you vary the percent contrast difference, all right, of the two stimuli, you can see that it crosses over, indeed, when they're identical. So that tells you that this system works. It says these are identical. And the same thing is true for the size series.

So this way then, if you do the same experiment with electrical stimulation, you can find out where this 50% crossover point is. This is for contrast, and this is for size. So then you can establish this with slightly different locations in the visual field.

And if you do that overall, you come up with a summary statement, which says that when you use currents between 20 and 120 microamps, OK, at eccentricities between 2 and 1/2 and 3 and 1/2 degrees, the contrast of the visual percept created in monkeys is 6% to 12%, and the size is between 15 and 20 minutes of visual angle. Now, that's quite small fortunately. So it's a local, little spot, like a star-like image.

Now, what do we mean by 6% to 12% contrast? So let me tell you this, that that's a useful thing for you to memorize. We talk about percent contrast. What you do is you measure, for example, in candelas per square meter, what the contrast is of this and what is the contest of the background.

Then you subtract one from the other, divide it by the sum of the two, and you multiply by 100. So when you do that, this is roughly 8% contrast. This is roughly 75% contrast. So it gives you a sense of what contrast means. All right? So what that means, in essence, that the visual stimulus created by the electrical stimulation is fairly low contrast.

All right. So now the next important factor, once one knows what the visual percept is that is created by electrical stimulation in the monkey, the next thing is to ask the question, well, what kind of electrode array should one put together? All right. Well, the big issue here has to do with the so-called magnification factor that I just talked about. So let's look at this analytically.

First of all, I already mentioned this. We have a very high packing density of photoreceptors in the fovea and much lower density further out. And so what you have is that this plate goes through the optic nerve up to the visual cortex to the lateral geniculate nucleus. So that's the basic arrangement, meaning that there are many, many more retinal ganglion cells per unit area that impinge on the visual cortex near the fovea than in the periphery.

So then to examine this, here we have, again, another picture. It's the geniculate again. Here's the fact that the cortex is constant in thickness. And here's the layout of the visual field on the posterior cortex with more area allocated to the fovea than to-- excuse me, than to periphery.

All right. So now I've shown you a variant of this before. If you present an arrow across the visual field here, this is the impression made with much more activation in central vision than in peripheral vision. I mean, you don't have to worry about that. That simply tells you, as we had discussed before, that the images created depend heavily on where you stimulate in the visual field.

OK. So now to look at this quantitatively, so we put a bunch of dots up in the visual field. This is in the central four degrees, and this is the actual activation in the visual cortex. OK?

Meaning that in the fovea, we have two elements here, this one and this one, here and here, that they take up a lot of space, whereas in the periphery, OK, they're very close to each other. That's due to the so-called magnification factor. And so one of the basic rules is that we've got to be able to stimulate the visual cortex taking into account the magnification factor.

So if you have this kind of spatial arrangement, if we then were to put a bunch of electrodes in like this, which we will call proportional electrodes, then if you stimulated all those, you would get a similar square with one proviso, namely that the little dots would be smaller in the fovea, obviously, than in the periphery.

So then if you have this kind of arrangement, what you can do is you can take a camera that has a fixed number of elements, and each of those elements you hook up to one of the electrodes. If you put the word "fiat lux" there, this is what you hypothetically would create, like that. OK? That's what it would look like.

So in contrast with this arrangement, if we were to use an electrode array, which you can readily buy-- you cannot buy proportional arrays. You have to somehow create them yourself. But if you buy a fixed array, which the elements are equidistant, then if you stimulated all those you would get a butterfly image, butterfly image. I will show a movie of that as well.

And that means that if you do that same conversion as before, this is what you would see in contrast to this. So in other words, it's a greatly distorted vision because you didn't take into account the magnification factor. All right.

So then what you can do, actually, you can also increase the number of electrodes in the center here. OK? And that would mean-- in this case we have 428 elements-- that you can recreate fiat lux pretty well with this increased number of elements, where with the original set you couldn't make anything out.

Now, the alternate to this would be that you could take a whole bunch of equally-spaced electrodes, which you can commercially buy, and put them in. But then to get the same acuity, you would need, in this case, something like over 600 electrodes, whereas in this case you would only need about 420.

So because of that, obviously we want to minimize the number of things you put into the brain, it's desirable to use a proportional device. So that's the first big step.

Now, let's assume that you have created a proportional device. The next thing would be to see, how do you stimulate these through a camera? So what you do then is an experiment in which you take a human subject, and you can put a camera on a helmet, and also, the person looks at a monitor. And otherwise, the rest of the world is excluded by having this bezel here. And then the camera looks at something, and then it can display the image here.

Well, one thing it can do, it can display the image as it is. But the other thing that's much more important, it can display that image as if electrical stimulation had taken place. Now, this becomes complicated because you have to do a lot of conversion of the input from the camera to the computer that's going to then drive the electrodes.

Now, the first important point, I mentioned this before, is that if you present a small spot-- this is actual real data here-- you present a small spot in the center of a small array of cells in the visual cortex, you get a vigorous response, whether a light increment or light decrement. But if you use a larger spot, you get no response, meaning that the cortical cells look at local differences. That has a lot to do with what we talked about with the illusions.

And so what you have to create is an algorithm that mimics that. And so the way you do that is you have these individual elements for each of the electrodes. And if that individual element in the computer that gets the input from the camera, if the whole element gets activated, there's no real activation of the electrode.

But if an edge appears like here, here, or here, then the electrode is activated. OK? So you create local differences by this conversion system, all right, by this algorithm that then, when the camera looks at the world, will respond whenever there's an edge, an illumination difference.

OK. So now the way this looks is like this. Again, as I've told you, whatever is in the fovea, the dots are smaller and further out, but that's OK. So that's the basic activation system.

And so now what we are going to do, having created such an algorithm, we're going to examine what the person who carries this camera on his head can actually see. Now, the way this can be done, you can either physically move the stimuli in the world, or the person looks at it and can slowly move his head around. OK?

So let's first look at what would happen if, instead of a proportional display, we used an equally-spaced display like that butterfly display. OK? So what I'm going to show you here on the left is what the camera is viewing. And here I'm going to show you what is actually perceived by this algorithm, by the person, where this image is converted into what he's looking at through that bezel. OK, you ready?

I mean, you don't see a square at all, right? What you see is, because you're not taking into account the magnification factor, you get a huge distortion.

So now let's go move on and ask the question, well, what happens when you are using a proportional display? All right? And now we get a rather nice story. Again, the visual display's on the left, and the image created is on the right. Ready?

So the rule, therefore, is that you must use a proportional image, proportional display, that can be done either by having it truly proportional as you put it into the visual cortex or by having an equally-spaced array, and the computer computes the proportional arrangement. But the disadvantage of the latter is that you need many, many more electrodes.

All right. Now, what I'm going to do is I'm going to show you whether you can read something when you use a proportional display like that. And in this case, we're going to have a bunch of big letters, and either the letters are going to move slowly across the visual field, or your head is going to move slowly with the camera like that to view that display that's up on the wall. OK?

So if we do that-- are you ready? See if you can read it. OK. Everybody able to read that one? OK. Who remembers what "fiat lux" means? I've only shown it about four or five times. Let there be light.

OK. So now, just to be [INAUDIBLE] attractive about it, I'm going to show you one more display. And this one is a display, again, a series of letters and words, is one that John F. Kennedy has used in a very, very famous statement, which I think most of you will remember.

And eventually, as you see the whole thing, you will be able to put it together. Are you ready? OK. Everybody's seen that, right? People nowadays, because of what's going on, are less inclined to take his advice.

All right. So that is then the essence of what I was going to cover today. And so let me now provide you with a brief summary. I will start with the prosthetics then. OK? Research on visual prosthetics is in its infancy, as I've noted. A great deal of basic research is needed before such a device can become effective. That's very unlike what you're going to hear about the auditory system.

The brain area that holds considerable promise for the prosthetic device based on electrical stimulation is area V1. Now, a prosthetic device for electrical stimulation of V1 must take into account the magnification factor. That's absolutely essential, and too many people are ignoring it.

Now, in a different color, we're going to talk about the illusions. There is no unitary explanation for the great many visual illusions extant. I pointed out to you that different illusions have different sources and different explanations.

The most popular theory explaining the Hermann grid illusion based on the center/surround organization of retinal ganglion cells is incorrect. That's a polite way of saying it's all wrong. A more likely theory is the one that assumes that area V1 cells are involved.

Retinal adaptation processes can explain illusions based on aftereffects, and you've seen quite a few of those. And many illusions disappear under isoluminant conditions. And lastly, there are no viable theories that explain illusions based on figure/ground relationships. We know what they are, but we don't know how the brain resolves figure and ground effects.

So that then is the end of what I wanted to cover today. And next time, we are going to talk about eye movement. And I hope that you will enjoy that as well.