Flash and JavaScript are required for this feature.
Download the video from iTunes U or the Internet Archive.
Description: Felice Frankel and Brian Hayes have a conversation about what exactly goes on, from a technical perspective, when you make an image.
Instructors: Felice Frankel and Brian Hayes
A Conversation with Brian H...
The following content is provided under a Creative Commons license.
Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free.
To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.
[FELICE FRANKEL] We're lucky to have my friend, Brian Hayes, with us today.
Brian is a writer-- a very wonderful writer, in fact, but he also takes photographs and he is very good at explaining what exactly is going on technically when you make images.
And I thought it would be interesting for he and myself to just take a little more closer look at what happens.
So I brought Brian into the studio to have a conversation with me and this is what you'll be listening to now.
[BRIAN HAYES] Hi, Felice.
It's great to be here with you.
[FF] Maybe the place to start is discussing pixels and the sensor that's inside the camera that records the image.
[BH] We know that the sensor is an array of little square pixels and we know that the picture that comes out of the camera in the end when look at on the screen or whatever, that that also is an array of little square pixels.
And you can't help yourself just imagining that there's a simple one-to-one mapping, that each of those little squares in the sensor corresponds to one of the squares in the picture.
And it isn't so.
It just isn't so and it never will be.
There's a computer graphics person named Alvy Ray Smith who wrote the most petulant academic paper that I've ever seen.
The title is "A Pixel is Not a Little Square, A Pixel is Not a Little Square, A Pixel is Not a Little Square." [FF] I think we understand what he's trying to say.
Right.
[BH] He's quite firm about this.
And of course, he also has to admit later on the paper that a pixel is a little square.
[FF] OK.
[BH] Pixels are what happened to photography when it became digitized, but they're not strictly an invention.
They existed on film, we just weren't aware of them in quite the same way.
And if you translate this discussion back 50 years, the question was then a matter of not how many pixels do you want, but how fine grained do you want your film?
[FF] You bet.
[BH] And the effect on the finished image was very similar.
If you chose a film with large grains of silver, you got very fast exposures, tended to get high contrast, but you also got a lot of noise and random fluctuations, especially in dark areas of the photograph where it looked mottled.
Now we have a much more regular geometric structure with, in spite of what Alvy Ray Smith says, little squares, pixels.
And we care about counting those pixels, how many millions of them form an image.
But the question of how much information is in the image, how closely can you look at it before you begin to see the structure of the image rather than the subject of the image, those issues have been there from the very beginning.
[FF] So, for example, just comparing it to film, although most of our students possibly don't even know what the word means, they have seen a little example of how we see the difference between setting the ISO number at 200 as compared to 6,400.
That they will have seen that there is a lot of grain and noise, or appears to be grainy-- that's the word, although that might not be accurate.
[BH] I don't think that the difference is worth bothering with.
[FF] Yeah.
[BH] It is grainy.
It's noisy.
I think that the clearest way to follow what's happening there and why that noise happens and why ISO matters is to think about the basic physics of what you're doing when you make a photograph.
What you're doing is counting photons, the quanta of light.
So you expose a piece of film or the sensor in a digital camera to the varying brightness in a scene.
It's focused on the plane of the film or the plane of the sensor and what you're doing is in each small area, the square pixels on the sensor or the chemical film grains, in each small area you're counting how many photons strike that spot within the brief moment that the shutter is open.
And there's a statistical issue here.
When a photon hits the silicon sensor, it's supposed to not lose an electron and it's actually the electrons that you wind up counting.
When it hits the film, it's supposed to catalyze a chemical reaction that produces a visible grain of silver.
Both of those things are somewhat unreliable.
It's-- in part, this goes back to the deep physics of quantum mechanics, that some of these things are probabilistic and you can't, even if you try, determine exactly what's going to happen.
But for a lot of reasons, when, say, 1,000 photons hit the film or the sensor, you may register some other number other than the exact 1,000.
You might count 900, you might actually count more than there were photons because some electrons are just spontaneously-- [FF] Yeah.
[BH] And sometimes the chemical reaction in the film happens even without light.
And there is a simple rule of thumb, a mathematical rule for explaining why it's so much better to have a lot of photons than just a few.
The error that you get in counting turns out to be roughly the square root of the number of photons.
And so if you have 10,000 photons that you're trying to count, well, the square root of 10,000 is 100 and that's about 1% error.
If you get 9,900 instead of 10,000 you're 1% off.
And when you translate that into an amount of light or the gray level on an image, 1% is probably something you're not even going to be able to notice.
On the other hand, if there were only 100 photons that you were counting, the square root of 100 is 10 and so you can have an error of 10% and something-- some pixel might be 10% brighter or darker than it's supposed to be and you will notice this.
And in fact, it looks like snow on the image.
It looks like random fluctuations darker and lighter.
And that's true whether you're looking at a digital image or a film image.
So you want to capture as many photons as you can and that's a complicated function of how long the shutter is open, how wide the aperture of the lens has been set, and how long or how sensitive the film is to light or how sensitive the sensor is.
And the ISO dial is what determines that sensitivity.
In the case of film, there was no dial.
You went to the store and bought a film with an ISO rating of whatever you chose.
With a digital camera you can adjust it was you wish.
[FF] Almost.
On my camera I cannot get-- I used to use Kodachrome 25.
[BH] I was an Ektar 25.
[FF] You're Ektar.
OK.
And I can't go down to 25 on my camera.
[BH] That's true.
I would say, however, that the whole strange historical accident that we have digital cameras with a setting that's labeled ISO is a little bit deceptive.
Setting something to ISO 26,000 on a digital camera doesn't match, in any obvious direct way, what the setting would be on a film.
And so it's with a good digital camera, the noise level that you get out of it at a low ISO setting is the equal of anything that you could have gotten on your Kodachrome 25.
[FF] So in the course, we talk about that because we have control over everything we make in this particular course, because it's in a controlled situation, we are suggesting to set the ISO at the lowest number so that we do not pick up a lot of noise.
[BH] Other things being equal, that's surely the-- [FF] Let's try to keep it simple for-- yeah.
[BH] Well, yeah.
[FF] So that's how we're taking pictures in our course.
[BH] Yes.
If you're taking pictures of something that holds still for you-- [FF] Yes.
[BH] --and especially if you have a tripod so that the camera also holds still-- [FF] This is the only way we're taking pictures in this course.
[BH] There would be no reason not to use the lowest ISO -- [FF] And we can add more light if we need to.
[BH] Yes.
[FF] So we have control.
[BH] Yeah.
[FF] This is very specific kind of photography.
[BH] And you have a luxurious situation.
[FF] Indeed.
[BH] There are occasions when it's appropriate, I think, to use a higher ISO setting.
But what you described, that kind of photography, of course-- why not?
[FF] Why not?
So shall we talk about exactly what happens to that pixel?
[BH] Sure.
I think the complexity of it is the kind of complexity that you don't really need to understand in detail.
You certainly don't need it in order to make good pictures.
There have been generations of brilliant photographers who knew nothing about the technology of cameras and certainly about the chemistry of film.
However, I think it's something that is useful to have in the back of your head.
And it may change your attitude toward the images that come out of the camera and how you think of them and what you dare to do to them.
There's a natural tendency, since we know that inside the camera there's this postage stamp sized slab of silicon that is the sensor that receives the light, the lens focuses an image onto that sensor.
And we know that's made up of these little squares called pixels in a rectangular array like seats in a theater or something, you have rows and columns full of them.
And the picture that comes out of the camera is also a rectangular array of colored squares in rows and columns, and so it's just natural to imagine that there is a simple one-to-one correspondence between the pixels on the sensor and the pixels on the image.
And if only it were that simple.
In fact, if you think about it a little more, you realize that it can't work quite that way because the square pixels on the sensor.
I've learned recently that the professionals who make these things, they call them sensels on the sensor and pixels in the picture.
But each of the picture elements on the sensor has to respond to any light that it receives of any color.
And in order to take a color photograph, you need to have separate signals for at least three color channels.
Each sensor element, then, is actually made up of four sensor elements, each of which has a color filter pasted on top of it like the kind of celluloid filters that you would use over a spotlight or something to change the lighting, except of course, they're very small.
These are arranged in a pattern so that each group of four sensor elements has two green and a red and a blue filter.
So right there if you have, say, 1,000 or a million pixels in the finished image-- 1,000 by 1,000, say-- you actually have 2,000 by 2,000 on the sensor.
There are four million elements rather than just one million.
So we're already at that point not having a one-to-one correspondence between the sensor and the picture.
And in fact, it's not even close to that.
There is no simple ratio.
There are more sensor elements on the sensor than there are in the finished picture and no pixel in the image actually corresponds to any one sensor element.
There's an algorithm that runs on the little computer inside the camera that looks at a whole neighborhood of sensor elements, and by averaging and filtering the outputs of a large number of elements it assigns a particular color to each pixel in the finished image.
[FF] So that each manufacturer probably has their own formula.
[BH] Very much so.
And they don't tell us what they are.
[FF] Of course not.
Is it fair to talk about, in this conversation about pixels, is this a place where we could talk about the difference between RAW, TIFF and JPEG?
[BH] Sure.
[FF] It would be good to have an idea about the differences.
[BH] Sure.
So each camera has a choice of outputs.
Typically, you can get either RAW or JPEG from most cameras.
And the RAW format is not the same from company to company, I guess.
But they're all, apart from details of file formats and things like that, they're all very similar.
It's a collection of numbers that represent color values in a very direct way.
If you looked at the numbers and assigned colors to each possible number, and you took a bunch of crayons and a big piece of graph paper with squares on it, you could actually reconstruct the whole image by painting the appropriate color in each square based on just looking at the numbers in the RAW file.
You wouldn't really want to do this, but it's there in principle.
Among the other common formats, the TIFF is actually very much the same.
It has the same information in it, it's just been rewritten in a different order, basically.
[FF] Good.
I have conversations with other photographers who are-- they cannot believe that I don't take RAW images.
[BH] Well, there's more to that.
If you take a RAW image and you upload it to a computer in that form and you use the appropriate piece of software, then, to convert it to a TIFF, you should have the same set of numbers in the new file as you did when it came out of the camera as a raw file.
On the other hand, if you take a JPEG and then convert that to a TIFF, the numbers will not match up one for one.
And the reason for that is that the JPEG format doesn't, in fact, store this big array of rows and columns of color values.
Instead, it takes blocks of pixels and analyzes the set of colors found within each block and then in order to store the-- well, not the same information, but substantially the same information-- in a smaller amount of data, it encodes the information into a different form.
You might, for example, look at the whole block of pixels-- I think they're eight by eight is the standard way of doing this.
You take those 64 pixels, an eight by eight square, and you ask, what's the average color in that whole block?
And so that's a first approximation of how you might represent the image.
And then you ask, so, what's the average color on the right half, and what's the average color on the left half, and on the top half and the bottom half?
And you store that information also.
And there's a little more to it than this, but not much more, actually.
What you wind up with is a way of representing the information that is more compact, but it's not exact.
And when you then reconvert it to, say, a TIFF file, you're going to wind up with something that's a little bit different.
And certainly, the purists among us would argue that there is damage done when you go to a compressed file format like JPEG.
Unquestionably, there are changes made.
That is, you have altered the information that is stored.
And the argument is that these days, with ample disk space and essentially the ability to keep every image you take in its full form, why risk losing anything?
That's a good argument, and I think that, for example, if you have the ability to keep both a RAW version and a JPEG that that's probably a good idea.
It leaves your options open.
You can always go back to the original form and work with it if it turns out that a particular JPEG isn't looking the way you want to.
However, I have to say that this is based purely on personal experience and not scientific investigation.
I've found many cameras whose JPEG processor does a better job of sharpening and color correcting the image than I can do myself.
[FF] I'm so happy you said that because I have, anecdotally, made comparisons between my images, either TIFF or JPEG, enlarged them like crazy on the screen, and I don't see the differences.
[BH] Yes.
Yes.
Cameras are very good, I'm afraid.
[FF] They are, yeah.
And it's a lot of smart people who've worked very hard to make it come out well.
Not to say that you can never beat them, but you've got to work pretty hard to do better.
And it's also worth keeping in mind, if only as a way of keeping your sanity in a world where lots of people want to tell you what to do, it's worth keeping in mind that a "RAW' file, in spite of its name, is not "raw"-- it's "cooked." The camera has already done a lot of processing and it has to.
What would truly be a "raw" file-- that is the output of the sensor directly without any processing at all-- it would hardly look like a photograph.
The dynamic range would be all wrong, the colors would be wildly incorrect.
The camera doesn't see the way our eye sees and it needs to be fixed before we even recognize it as a photographic image.
This was also true in the age of film.
If you look at, say, a color negative, a piece of color negative film that's been processed but not printed, the entire image has got a rosy salmon colored tone to it that's wildly far from the correct color.
And that needs to be corrected.
In the old days, this happened chemically.
It was the film that you were printing to from the color negative was designed explicitly to correct for that distortion.
And in fact, if you use that film to print from a negative that didn't have the orange cast to it, you would get something that was terribly wrong in the other direction.
It's just never been the case that there was a "raw" format that was truly some kind of pristine representation of nature.
It's all "cooked" formats.
And given that, it's sort of liberating to know that there's already a whole lot of processing going on before the picture ever comes out of the camera.
And so maybe you shouldn't be too afraid of turning a dial or two.
The conception of a camera as some sort of objective recording device that captures reality without distortion has never been true.
And even at a very fundamental level, when you look at the physics and technology of how the machine works.
[FF] OK.
So we talk very briefly about trying to get some really wonderful images for cover submissions, and we do talk about the submission should be 300 dots per inch because that's what the art directors usually ask for.
And if it's a full bleed cover, it's 8 and 1/2 by 11, but sometimes your images is not that size so we briefly touch on putting it into software and increasing the file size.
There's a lot that goes on when you click that button, I'm assuming.
What do you think about all that?
[BH] Increasing the file size is really very easy.
As I say, you can just press a button.
Increasing the resolution of the image is not so easy.
In some precise sense, you're not adding any information to an image when you simply increase the resolution in Photoshop, say, by making it into a larger file.
If you look at it in the simplest terms, if you were at, say 150 dots per inch, you could take each pixel and divide it into four smaller pixels.
Each square becomes four tinier squares.
Each of those four squares is identical.
And you then have a file that's four times as large in terms of megabytes and has twice the resolution and it's now 300 dots per inch.
If you look at the metadata in Photoshop or something, that's what it will tell you.
But it has no more information than the original one.
[FF] OK.
But what we have touched on with the flatbed scanner is when we tell the scanner to scan at, let's say, 72 DPI and then we tell it later to scan at 300 DPI, are we not capturing more information when we scan the image?
[BH] Absolutely.
Yes, you are.
You are dealing there with a physical process that-- again, if we look at it in the simplest terms, and this isn't terribly realistic-- but the sensor in the scanner, when you scan at reduced resolution, at, say, 150 dots per inch, it's going to use two sensor elements together as if they were one and simply record half as much information.
Then when you set the scanner for 300 dots per inch, it reads each of those pairs of sensors as two separate sensors and records twice as much information, or really four times as much because it's in two dimensions.
The scanner is moving across the surface, it's not just one line.
When you do this in software, however, you have no way of going back to the original object that you were scanning or photographing and recording more information about it.
All you have is what's in the original file.
Now, this seems very gloomy.
It seems as if you have no hope of actually increasing the resolution of an image.
Once you've made it, if you can't go back to the original and recreate it from life, then you'll never have more dots per inch than what you captured to begin with.
It's not quite that bad.
The statement that you can never put more information into it than was there to begin with, it's true in some mathematical sense in that if you have, say, an image of totally random dots that are scattered across the surface and you try to increase the resolution in order to get more dots per inch, it's not going to work.
All you'll do is make the dots you have a little bit fuzzier.
But we don't generally take pictures of random arrays of dots.
We take pictures of objects that have macroscopic form, that have sharp edges, they have areas of one color and another color, they have smooth gradients of color.
And all of that is structure at a much higher level than individual pixels.
And it is possible, if you're clever or if your software is clever, to find ways to boost the resolution of an image that if the image cooperates, if it's a picture of something that has reasonable structure to it, which could be anything from a human face, to a plant, to a technological artifact.
Because reasonably large areas of color or smooth gradients or sharp edges and that these are the things that you're interested in, it actually comes out pretty well.
I've done it myself.
[FF] Me, too.
[BH] And in most cases, the result is successful.
There is a quasi ethical question here of do you disclose to the art director you're sending this to that, in fact, this was a 150 dot per inch file that you have manipulated-- there's that terrible word-- in order to boost the resolution?
I'm not an ethicist.
I leave that to other people to decide.
It's not alteration of the content, in my view.
It's not like you're air brushing out something that wasn't supposed to be there.
But obviously, it is a deliberate act to make the picture look better.
I would also say that another problem here is that the art director who insists that the file has to be 300 DPI is being rather foolishly rigid about this.
There are images that you can print on a magazine cover that are 150 DPI that work out fine.
But changing the world in that direction is much harder than fixing the photograph.
[FF] Especially when the art director basically has to first talk to the editor, who generally knows nothing about image making.
[BH] As a former editor, I will let you have your say.
[FF] Thank you.
But no, I would say for cover submissions where we're talking about we're bringing the notion of aesthetics and communication, creating a finer image.
[BH] I entirely agree.
There are various standards that get applied.
Journalism has one set of rules that's really quite harsh.
Photojournalists working for a newspaper or an agency like Magnum, they suffer severe penalties if they're caught making changes to images.
[FF] Like moving Egyptian tombs.
[BH] Yeah, right.
[FF] Do you remember that?
[BH] I certainly do.
[FF] This was a long time ago.
[BH] In that case, if I remember correctly, it wasn't the photographer, it was the art director who "improved." [FF] They moved the pyramids so that the typeface could be dropped more properly as a full cover design.
That was a long time ago.
It's dating us, Brian, I'm afraid.
But those questions still arise.
[BH] Of course they do, yes.
And it's easier to do it in the photograph than it is to move the pyramid itself.
[FF] As of today, yes, I would have to say that.
So we are going to be discussing this in week five, how far can we go?
[BH] It's a fascinating issue with a very long history.
[FF] Yeah.
Oh, yeah.
[BH] And it is-- I think it's something that individuals, they need to make their own ethical choices, but obviously if you're working in a social setting where there are other people making the rules, you also need to be aware of that.
[FF] There's a whole issue as we talk about the whole notion of manipulation and how far can one go.
And certainly in science imaging, there are very serious restrictions, and there should be.
It isn't all about the data, but the question is defining what the data is.
The fact that you're making an image with a tool, you're already changing the data.
We have to remember, and we'll go back to this later, but making a picture is making a representation.
It is not the thing, it is a re- presentation and there is some sort of decision-making involved.
The thing that I'm trying to emphasize in the course is that we have to create a literacy, in a way, of what makes for a good picture.
And you and I might disagree on that, and we have, I think.
And I hope that students come to their own decisions about what is better than that or less than that and what is good.
But I think we have to start somewhere, and the "somewhere" for us is to create an image that well communicates what it is that this device or Petri dish or thing that we're photographing-- what picture can best communicate the structure?
We're generally interested in structure.
And if we are talking about process, for example, as we do when we make photo illustrations, we have examples later on that the students will see how I've used bits and pieces of photographs, put it all together for an illustration.
So it's not a documentary image and there's a place for that in science.
But in the end, it's about getting the finest quality to communicate the science.
That is what we're doing here in this course.
And by the way, we will be discussing using tablets and mobile devices.
Very tangentially, but there's a place for that.
Absolutely.
When you really, really need to take a picture quickly and send it to your colleague to show them that you have evidence of what you've produced.
But as far as making the finest quality image, we still are going to-- at this point-- stick to the camera and the 105 lens.
That's what we're sticking to.
[BH] I wouldn't try to argue otherwise.
I think the way people will think of photography and of pictures in years to come may be rather different from our view.
It's certainly changed within my lifetime, when photographs were physical objects that you put in scrapbooks or filed away.
We don't think of them in those terms anymore.
And I think they may change their nature again because they are so ubiquitous.
But that doesn't really have much to do with the kinds of photography that are involved in, say, scientific documentation or scientific illustration.
In art photography, you never know what's going to happen.
And there are lots of artists, I think, who are now working with cell phones and such, and even with the plastic lens Lomo cameras and things that deliberately distort the view of the world.
So I think there's lots of latitude for different approaches to this.
But for serious documentary work or for trying to create the most vivid image that you possibly can of something that holds still for you while you point the camera at it, clearly you're going to use the best tool that you can come up with.
Even mediocre ones today, I think, are considerably better than the best ones of a few decades ago.
They're probably going to get better still.