Lecture 29: Singular value decomposition

Related Resources

Readings
Table of Contents

• Download English-US transcript (PDF)

Okay. This is the lecture on the singular value decomposition.

But everybody calls it the SVD.

So this is the final and best factorization of a matrix.

Let me tell you what's coming.

The factors will be, orthogonal matrix, diagonal matrix, orthogonal matrix.

So it's things that we've seen before, these special good matrices, orthogonal diagonal.

The new point is that we need two orthogonal matrices.

A can be any matrix whatsoever.

Any matrix whatsoever has this singular value decomposition, so a diagonal one in the middle, but I need two different -- probably different orthogonal matrices to be able to do this.

Okay. And this factorization has jumped into importance and is properly, I think, maybe the bringing together of everything in this course.

One thing we'll bring together is the very good family of matrices that we just studied, symmetric, positive, definite.

Do you remember the stories with those guys?

Because they were symmetric, their eigenvectors were orthogonal, so I could produce an orthogonal matrix -- this is my usual one.

My usual one is the eigenvectors and eigenvalues In the symmetric case, the eigenvectors are orthogonal, so I've got the good -- my ordinary s has become an especially good Q.

And positive definite, my ordinary lambda has become a positive lambda.

So that's the singular value decomposition in case our matrix is symmetric positive definite -- in that case, I don't need two -- U and a V -- one orthogonal matrix will do for both sides.

So this would be no good in general, because usually the eigenvector matrix isn't orthogonal.

So that's not what I'm after.

I'm looking for orthogonal times diagonal times orthogonal.

And let me show you what that means and where it comes from.

Okay.

What does it mean?

You remember the picture of any linear transformation.

This was, like, the most important figure in And what I looking for now?

A typical vector in the row space -- typical vector, let me call it v1, gets taken over to some vector in the column space, say u1. So u1 is Av1. Okay.

Now, another vector gets taken over here somewhere.

What I looking for?

In this SVD, this singular value decomposition, what I'm looking for is an orthogonal basis here that gets knocked over into an orthogonal basis over there.

See that's pretty special, to have an orthogonal basis in the row space that goes over into an orthogonal basis -- so this is like a right angle and this is a right angle -- into an orthogonal basis in the column space.

So that's our goal, is to find -- do you see how things are coming together?

First of all, can I find an orthogonal basis for this row space?

Of course.

No big deal to find an orthogonal basis.

Graham Schmidt tells me how to do it.

Start with any old basis and grind through Graham Schmidt, out comes an orthogonal basis.

But then, if I just take any old orthogonal basis, then when I multiply by A, there's no reason why it should be orthogonal over here.

So I'm looking for this special set up where A takes these basis vectors into orthogonal vectors over there.

Now, you might have noticed that the null space I didn't include.

Why don't I stick that in?

You remember our usual figure had a little null space and a little null space.

And those are no problems.

Those null spaces are going to show up as zeroes on the diagonal of sigma, so that doesn't present any difficulty.

Our difficulty is to find these.

So do you see what this will mean?

This will mean that A times these v-s, v1, v2, up to -- what's the dimension of this row space?

Vr. Sorry, make that V a little smaller -- up to vr.

So that's -- Av1 is going to be the first column, so here's what I'm achieving.

Oh, I'm not only going to make these orthogonal, but why not make them orthonormal?

Make them unit vectors.

So maybe the unit vector is here, is the u1, and this might be a multiple of it.

So really, what's happening is Av1 is some multiple of u1, right? These guys will be unit vectors and they'll go over into multiples of unit vectors and the multiple I'm not going to call lambda anymore.

I'm calling it sigma.

So that's the number -- the stretching number.

And similarly, Av2 is sigma two u2. This is my goal.

And now I want to express that goal in matrix language.

That's the usual step.

Think of what you want and then express it as a matrix multiplication.

So Av1 is sigma one u1 -- actually, here we go.

Let me pull out these -- u1, u2 to ur and then a matrix with the sigmas.

Everything now is going to be in that little part of the blackboard.

Do you see that this equation says what I'm trying to do with my figure.

A times the first basis vector should be sigma one times the other basis -- the other first basis vector.

These are the basis vectors in the row space, these are the basis vectors in the column space and these are the multiplying factors.

So Av2 is sigma two times u2, Avr is sigma r times ur.

And then we've got a whole lot of zeroes and maybe some zeroes at the end, but that's the heart of it.

And now if I express that in -- as matrices, because you knew that was coming -- that's what I have.

So, this is my goal, to find an orthogonal basis in the orthonormal, even -- basis in the row space and an orthonormal basis in the column space so that I've sort of diagonalized the matrix.

The matrix A is, like, getting converted to this diagonal matrix sigma.

And you notice that usually I have to allow myself two different bases.

My little comment about symmetric positive definite was the one case where it's A Q equal Q sigma, where V and U are the same Q.

But mostly, you know, I'm going to take a matrix like -- oh, let me take a matrix like four four minus three three.

Okay.

There's a two by two matrix.

It's invertible, so it has rank two.

So I'm going to look for two vectors, v1 and v2 in the row space, and U -- so I'm going to look for v1, v2 in the row space, which of course is R^2. And I'm going to look for u1, u2 in the column space, which of course is also R^2, and I'm going to look for numbers sigma one and sigma two so that it all comes out right.

So these guys are orthonormal, these guys are orthonormal and these are the scaling factors.

So I'll do that example as soon as I get the matrix picture straight.

Okay.

Do you see that this expresses what I want?

Can I just say two words about null spaces?

If there's some null space, then we want to stick in a basis for those, for that.

So here comes a basis for the null space, v(r+1) down to vm.

So if we only had an r dimensional row space and the other n-r dimensions were in the null space -- okay, we'll take an orthogonal -- orthonormal basis there.

No problem. And then we'll just get zeroes. So, actually, w- those zeroes will come out on the diagonal matrix.

So I'll complete that to an orthonormal basis for the whole space, R^m.

I complete this to an orthonormal basis for the whole space R^n and I complete that with zeroes. Null spaces are no problem here.

So really the true problem is in a matrix like that, which isn't symmetric, so I can't use its eigenvectors, they're not orthogonal -- but somehow I have to get these orthogonal -- in fact, orthonormal guys that make it work.

I have to find these orthonormal guys, these orthonormal guys and I want Av1 to be sigma one u1 and Av2 to be sigma two u2. Okay. That's my goal.

Here's the matrices that are going to get me there.

Now these are orthogonal matrices.

I can put that -- if I multiply on both sides by V inverse, I have A equals U sigma V inverse, and of course you know the other way I can write V inverse.

This is one of those square orthogonal matrices, so it's the same as U sigma V transpose.

Okay.

Here's my problem.

I've got two orthogonal matrices here.

And I don't want to find them both at once.

So I want to cook up some expression that will make the Us disappear.

I would like to make the Us disappear and leave me only with the Vs.

And here's how to do it.

It's the same combination that keeps showing up whenever we have a general rectangular matrix, then it's A transpose A, that's the great matrix.

That's the great matrix.

That's the matrix that's symmetric, and in fact positive definite or at least positive semi-definite. This is the matrix with nice properties, so let's see what

will it be? So if I took the transpose, then, I would have -- A transpose A will be what?

What do I have?

If I transpose that I have V sigma transpose U transpose, that's the A transpose.

Now the A -- and what have I got?

Looks like worse, because it's got six things now together, but it's going to collapse into something good.

What does U transpose U collapse into?

I, the identity.

So that's the key point.

This is the identity and we don't have U anymore.

And sigma transpose times sigma, those are diagonal matrixes, so their product is just going to have sigma squareds on the diagonal.

So do you see what we've got here?

This is V times this easy matrix sigma one squared sigma two squared times V transpose.

This is the A transpose A.

This is -- let me copy down -- A transpose A is that.

Us are out of the picture, now.

I'm only having to choose the Vs, and what are these Vs?

And what are these sigmas?

Do you know what the Vs are?

They're the eigenvectors that -- see, this is a perfect eigenvector, eigenvalue, Q lambda Q transpose for the matrix A transpose A.

A itself is nothing special.

But A transpose A will be special.

It'll be symmetric positive definite, so this will be its eigenvectors and this'll be its eigenvalues.

And the eigenvalues'll be positive because this thing's positive definite.

So this is my method.

This tells me what the Vs are.

And how I going to find the Us?

Well, one way would be to look at A A transpose.

Multiply A by A transpose in the opposite order.

That will stick the Vs in the middle, knock them out, and leave me with the Us.

So here's the overall picture, then.

The Vs are the eigenvectors of A transpose A.

The Us are the eigenvectors of A A transpose, which are different.

And the sigmas are the square roots of these and the positive square roots, so we have positive sigmas.

Let me do it for that example.

This is really what you should know and be able to do for the SVD.

Okay.

Let me take that matrix.

So what's my first step?

Compute A transpose A, because I want its eigenvectors.

Okay. So I have to compute A transpose A.

So A transpose is four four minus three three, and A is four four minus three three, and I do that multiplication and I get sixteen -- I get twenty five -- I get sixteen minus nine -- is that seven?

And it better come out symmetric.

And -- oh, okay, and then it comes out 25. Okay. So, I want its eigenvectors and its eigenvalues.

Its eigenvectors will be the Vs, its eigenvalues will be the squares of the sigmas.

Okay. What are the eigenvalues and eigenvectors of this guy?

Have you seen that two by two example enough to recognize that the eigenvectors are -- that one one is an eigenvector?

So this here is A transpose A.

I'm looking for its eigenvectors.

So its eigenvectors, I think, are one one and one minus one, because if I multiply that matrix by one one, what do I get?

If I multiply that matrix by one one, I get 32 32, which is 32 of one one.

So there's the first eigenvector, and there's the eigenvalue for A transpose A.

So I'm going to take its square root for sigma.

Okay.

What's the eigenvector that goes -- eigenvalue that goes with this one?

If I do that multiplication, what do I get?

I get some multiple of one minus one, and what is that multiple?

Looks like 18. Okay.

So those are the two eigenvectors, but -- oh, just a moment, I didn't normalize them.

To make everything absolutely right, I ought to normalize these eigenvectors, divide by their length, square root of two.

So all these guys should be true unit vectors and, of course, that normalization didn't change the 32 and the 18. Okay.

So I'm happy with the Vs.

Here are the Vs.

So now let me put together the pieces here.

Here's my A.

Here's my A.

Let me write down A again.

If life is right, we should get U, which I don't yet know -- U I don't yet know, sigma I do now know.

What's sigma?

So I'm looking for a U sigma V transpose.

U, the diagonal guy and V transpose.

Okay.

Let's just see that come out right.

So what are the sigmas?

They're the square roots of these things.

So square root of 32 and square root of 18. Zero zero.

Okay.

What are the Vs?

They're these two.

And I have to transpose -- maybe that just leaves me with ones -- with one over square root of two in that row and the other one is one over square root of two minus one over square root of two.

Now finally, I've got to know the Us.

Well, actually, one way to do -- since I now know all the other pieces, I could put those together and figure out what the Us are.

But let me do it the A A transpose way.

Okay. Find the Us now. u1 and u2. And what are they?

I look at A A transpose -- so A is supposed to be U sigma V transpose, and then when I transpose that I get V sigma transpose U transpose.

So I'm just doing it in the opposite order, A times A transpose, and what's the good part here?

That in the middle, V transpose V is going to be the identity.

So this is just U sigma sigma transpose, that's some diagonal matrix with sigma squareds and U transpose.

So what I seeing here?

I'm seeing here, again, a symmetric positive definite or at least semi-definite matrix and I'm seeing its eigenvectors and its eigenvalues.

So if I compute A A transpose, its eigenvectors will be the things that go into U.

Okay, so I need to compute A A transpose.

I guess I'm going to have to go -- can I move that up just a little?

Maybe a little more and do A A transpose.

So what's A?

Four four minus three and three.

And what's A transpose?

Four four minus three and three.

And when I do that multiplication, what do I get?

Sixteen and sixteen, thirty two.

Uh, that one comes out zero.

Oh, so this is a lucky case and that one comes out 18. So this is an accident that A A transpose happens to come out diagonal, so we know easily its eigenvectors and eigenvalues.

So its eigenvectors -- what's the first eigenvector for this A A transpose matrix?

It's just one zero, and when I do that multiplication, I get 32 times one zero.

And the other eigenvector is just zero one and when I multiply by that I get 18. So this is A A transpose.

Multiplying that gives me the 32 A A transpose.

Multiplying this guy gives me First of all, I got 32 and 18 again.

Am I surprised?

You know, it's clearly not an accident.

The eigenvalues of A A transpose were exactly the same as the eigenvalues of -- this one was A transpose A.

That's no surprise at all.

The eigenvalues of A B are the same as the eigenvalues of B A.

That's a very nice fact, that eigenvalues stay the same if I switch the order of multiplication.

So no surprise to see 32 and What I learned -- first the check that things were numerically correct, but now I've learned these eigenvectors, and actually they're about as nice as can be.

They're the best orthogonal matrix, just the identity.

Okay.

So my claim is that it ought to all fit together, that these numbers should come out right.

The numbers should come out right because the matrix multiplications use the properties that we want.

Okay. Shall we just check that?

Here's the identity, so not doing anything -- square root of 32 is multiplying that row, so that square root of 32 divided by square root of two means square root of 16, four, correct?

And square root of 18 is divided by square root of two, so that leaves me square root of 9, which is three, but -- well, Professor Strang, you see the problem?

Why is that -- okay.

Why I getting minus three three here and here I'm getting three minus three?

Phooey.

I don't know why.

It shouldn't have happened, but it did.

Now, okay, you could say, well, just -- the eigenvector there could have -- I could have had the minus sign here for that eigenvector, but I'm not happy about that.

Hmm.

Okay. So I realize there's a little catch here somewhere and I may not see it until Wednesday.

Which then gives you a very important reason to come back on Wednesday, to catch that sine difference.

So what did I do illegally?

I think I put the eigenvectors in that matrix V transpose -- okay, I'm going to have to think.

Why did that come out with with the opposite sines?

So you see -- I mean, if I had a minus there, I would be all right, but I don't want that.

I want positive entries down the diagonal of sigma squared.

Okay.

It'll come to me, but, I'm going to leave this example to finish.

Okay.

And the beauty of, these sliding boards is I can make that go away.

Can I,-- let me not do it, though, yet.

Let me take a second example.

Let me take a second example where the matrix is singular.

So rank one.

Okay, so let me take as an example two, where my matrix A is going to be rectangular again -- let me just make it four three eight six.

Okay.

That's a rank one matrix.

So that has a null space and only a one dimensional row space and column space.

So actually, my picture becomes easy for this matrix, because what's my row space for this one?

So this is two by two.

So my pictures are both two dimensional.

My row space is all multiples of that vector four three.

So the whole -- the row space is just a line, right?

That's the row space.

And the null space, of course, is the perpendicular line.

So the row space for this matrix is multiples of four three.

Typical row.

Okay.

What's the column space?

The columns are all multiples of four eight, three six, one two.

The column space, then, goes in, like, this direction.

So the column space -- when I look at those columns, the column space -- so it's only one dimensional, because the rank is one.

It's multiples of four eight.

Okay.

And what's the null space of A transpose?

It's the perpendicular guy.

So this was the null space of A and this is the null space of A transpose.

Okay.

What I want to say here is that choosing these orthogonal bases for the row space and the column space is, like, no problem.

They're only one dimensional.

So what should V be?

V should be -- v1, but -- yes, v1, rather -- v1 is supposed to be a unit vector.

There's only one v1 to choose here, only one dimension in the row space.

I just want to make it a unit vector.

So v1 will be -- it'll be this vector, but made into a unit vector, so four -- point eight point six.

Four fifths, three fifths.

And what will be u1? u1 will be the unit vector there.

So I want to turn four eight or one two into a unit vector, so u1 will be -- let's see, if it's one two, then what multiple of one two do I want?

That has length square root of five, so I have to divide by square root of five.

Let me complete the singular value decomposition for this matrix.

So this matrix, four three eight six, is -- so I know what u1 -- here's A and I want to get U the basis in the column space.

And it has to start with this guy, one over square root of five two over square root of five.

Then I want the sigma.

Okay. What are we expecting now for sigma?

This is only a rank one matrix.

We're only expecting a sigma one, which I have to find, but zeroes here.

Okay.

So what's sigma one?

It should be the -- where did these sigmas come from?

They came from A transpose A, so I -- can I do that little calculation over here?

A transpose A is four three -- four three eight six times four three eight six.

This had better -- this is a rank one matrix, this is going to be -- the whole thing will have rank one, that's 16 and 64 is 80, 12 and 48 is 60, 12 and 48 is 60, 9 and 36 is 45. Okay.

It's a rank one matrix.

Of course.

Every row is a multiple of four three.

And what's the eigen -- what are the eigenvalues of that matrix?

So this is the calculation -- this is like practicing,

now. What are the eigenvalues of this rank one matrix?

Well, tell me one eigenvalue, since the rank is only one, one eigenvalue is going to be zero.

And then you know that the other eigenvalue is going to be a hundred and twenty five.

So that's sigma squared, right, in A transpose A.

So this will be the square root of a hundred and twenty five.

And then finally, the V transpose -- the Vs will be -- there's v1, and what's v2? What's v2 in the -- how do I make this into an orthonormal basis?

Well, v2 is, in the null space direction.

It's perpendicular to that, so point six and minus point

eight. So those are the Vs that go in here.

Point eight, point six and point six minus point eight.

Okay. And I guess I better finish this guy.

So this guy, all I want is to complete the orthonormal basis -- it'll be coming from there.

It'll be a two over square root of five and a minus one over square root of five.

Let me take square root of five out of that matrix to make it look better.

So one over square root of five times one two two minus one.

Okay.

So there I have -- including the square root of five -- that's an orthogonal matrix, that's an orthogonal matrix, that's a diagonal matrix and its rank is only one.

And now if I do that multiplication, I pray that it comes out right.

The square root of five will cancel into that square root of one twenty five and leave me with the square root of 25, which is five, and five will multiply these numbers and I'll get whole numbers and out will come A.

Okay.

That's like a second example showing how the null space guy -- so this -- this vector and this one were multiplied by this zero.

So they were easy to deal with.

Tthe key ones are the ones in the column space and the row space.

Do you see how I'm getting columns here, diagonal here, rows here, coming together to produce A.

Okay, that's the singular value decomposition.

So, let me think what I want to add to complete this topic.

So that's two examples.

And now let's think what we're really doing.

We're choosing the right basis for the four subspaces of linear algebra.

Let me write this down.

So v1 up to vr is an orthonormal basis for the row space. u1 up to ur is an orthonormal basis for the column space.

And then I just finish those out by v(r+1), the rest up to vn is an orthonormal basis for the null space.

And finally, u(r+1) up to is an orthonormal basis for the null space of A transpose.

Do you see that we finally got the bases right?

They're right because they're orthonormal, and also -- again, Graham Schmidt would have done this in chapter four.

Here we needed eigenvalues, because these bases make the matrix diagonal.

A times V I is a multiple of U I.

So I'll put "and" -- the matrix has been made diagonal.

When we choose these bases, there's no coupling between Vs and no coupling between Us.

Each A -- A times each V is in the direction of the corresponding U.

So it's exactly the right basis for the four fundamental subspaces.

And of course, their dimensions are what we know.

The dimension of the row space is the rank r, and so is the dimension of the column space.

The dimension of the null space is n-r, that's how many vectors we need, and m-r basis vectors for the left null space, the null space of A transpose.

Okay. I'm going to stop there.

I could develop further from the SVD, but we'll see it again in the very last lectures of the course.

So there's the SVD.

Thanks.

Free Downloads

Free Streaming

• VideoLectures.net

Video

• iTunes U (MP4 - 87MB)
• Internet Archive (MP4 - 162MB)

Caption

• English-US (SRT)