Lecture 7: Solving Ax = 0: pivot variables, special solutions

Related Resources

Readings
Table of Contents

• Download English-US transcript (PDF)

OK, here's linear algebra lecture seven. I've been talking about vector spaces and specially the null space of a matrix and the column space of a matrix.

What's in those spaces.

Now I want to actually describe them.

How do you describe all the vectors that are in those spaces?

How do you compute these things?

So this is the, turning the idea, the definition, into an algorithm.

What's the algorithm for solving A x =0? So that's the null space that I'm interested in.

So can I take a particular matrix A and describe the natural algorithm, and I'll execute it for that matrix -- here we go.

So let me take the matrix as an example.

So we're definitely talking rectangular matrices in this

chapter. So I'll make, I'll have four columns.

And three rows.

Two four six eight and three six eight ten. OK. If I just look at those columns, and rows, well, I notice right away that column two is a multiple of column one.

It's in the same direction as column one.

It's not independent.

I'll expect to discover that in the process.

Actually, with rows, I notice that that row plus this row gives the third row.

So the third row is not independent.

So, all that should come out of elimination.

So now what I -- my algorithm is elimination, but extended now to the rectangular case, where we have to continue even if there's zeros in the pivot position, we go on. OK, so let me execute elimination for that matrix.

My goal is always, while I'm doing elimination -- I'm not changing the null space.

That's very important, right?

I'm solving A x equals zero by elimination, and when I do these operations that you already know, when I subtract a multiple of one equation from another equation, I'm not changing the solutions.

So I'm not changing the null space.

Actually, I changing the column space, as you'll see.

So you have to pay attention.

What does elimination leave unchanged?

And the answer is the solutions to the system are not changed because I'm doing the same thing to -- I'm doing a legitimate operations on the equations.

Of course, on the right hand side it's always zero, and I don't plan to write zero all the time. OK, so I'm really just working on the left side, but the right side is, is keeping up always zeros. OK, so what's elimination?

Well, you know where the first pivot is and you know what to do

with it. So can I just take the first step below here?

So that pivot row is fine.

I take two times that row away from this one and I get zero

zero. That's signaling a difficulty.

Two, two twos away from the six leaves me with a two.

Two twos away from the eight leaves me with a four.

And now three of those away from here is zero, again another zero, three twos away from that eight is the two, three twos away from that ten is a four. OK. That's the first stage of elimination.

I've got the first column straight.

So of course I move on to the second column.

I look in that position, I see a zero.

I look below it, hoping for a non-zero that I can do a row exchange.

But it's zero below.

So that's telling me that that column is -- well, what it's really going to be telling me is that that column is a combination of the earlier columns.

It's that second column is dependent on the earlier columns.

But I don't stop to think here.

In that column there's nothing to do.

I go on to the next.

So here's the next pivot.

So there's the first pivot and there's the second pivot, and I just keep this elimination going downwards.

So, so the next step keeps the first row, keeps the second row with its pivot, so I've got my two pivots, and does elimination to clear out the column below that pivot.

So actually you see the multiplier is one.

It subtracts row two from row three and produces a row of zeros. OK. That I would call that matrix U, right?

That's our upper -- well, I can't quite say upper triangular.

Maybe upper -- I don't know -- upper something. It's in this so-called echelon form. The word echelon means, like, staircase form.

It's the, the non-zeros come in that staircase form.

If there was another pivot here, then the staircase would include that.

But here's a case where we have two pivots only. OK, so actually we've already discovered the most important number about this matrix.

The number of pivots is two.

That number we will call the rank of the matrix.

So let me put immediately.

The rank of A -- so I'm telling you what this word rank means in the algorithm.

It's equal to the number of pivots.

And in this case, two. OK, for me that number two is the crucial number. OK, now I go to -- you remember I'm always solving A x equals zero, but now I can solve U x equals zero, right?

Same solution, same null space. OK. So I could stop here -- why don't I stop here and do the back substitution.

So now I have to ask you, how do I describe the solutions?

There are solutions, right, to A x equals zero.

I knew there would be.

I had three equations in four unknowns.

I certainly expected some solutions.

Now I want to see what are they. OK, here's the critical step. I refer to it up here as separating out the pivot variables, the pivot columns, which are these two.

Here I have two pivot columns. Those, obviously, they're the columns with the pivots. So I have two pivot columns.

And I have the other columns, I'll call free. These are free columns, OK. Why do I use those words?

Why do I use that word free?

Because now I want to write, I want to find the solutions to U x equals zero.

Here is the way I do it.

These free columns I can assign any number freely to those -- to the variables x2 and x4, the ones that multiply columns two and four.

So I can assign anything I like to x2 and x4. And then I can solve the equations for x1 and x3. Let me say that again.

If I give -- let me, let me assign.

So, so one particular x is to assign, say, the value one to the, to x2, and zero to x4. Those are -- that was a free choice, but it's a convenient choice. OK. Now I want to solve U x equals zero and find numbers one and three, complete the solution. Can I write down -- let's see.

OK. Shall we just remember what U x equals zero represents?

What are my equations?

That first equation is x1 plus just -- I'm just saying what are these matrices meaning.

That's the first equation.

And the second equation was 2x3 + 4x4=0. Those are my two equations.

OK. Now the point is I can find x1 and x3 by back substitution.

So we're building on what we already know.

The new thing is that there were some free variables that I could give any numbers to.

And I'm systematically going to make a choice like this, Now what is x3? 1 and 0. Let's, let's go backwards, back up.

I look at the last equation. x3 is zero, from the last equation, because, because x4 we've set x4 to zero, and then we get x3 as zero. OK. Now we set x2 to be one, so what is x1? Negative two, right?

So then I have negative two plus two, zero and zero, correctly giving zero.

There is a vector in the null space.

There is a solution to A x equals zero.

In fact, what solution is that?

That simply says that minus two times the first column plus one times the second column is the zero column.

Of course that's right.

Minus two of that column plus one of that, or minus two of that plus one of that.

That solution is -- that, that's just what we saw immediately, that the second column is twice as big as the first column. OK, tell me some more vectors in the null space.

I found one.

Tell me, how to get a bunch more immediately out of that one.

Just take multiples of it.

Any multiple of -- I could multiply this by anything.

I might as well call it, I could say, C, some multiple of this.

So let me -- so X could be any multiple of this one. OK, that, that describes now a line, an infinitely long line in four dimensional space.

But -- which is in the null space.

Is that the whole null space?

No. I've got two free variables here.

I made this choice, one and zero, for the free variables, but I could have made another choice.

Let me make the other choice zero and one.

You see my system. So let me repeat the system.

This is the algorithm that you, you just learned to do. Do elimination.

Decide which are the pivot columns and which are the free columns.

That tells you that, that variables one and three are pivot variables, two and four are free variables.

Then those free variables, you assign them -- you give one of them the value one and the others the value zero -- in this case, we had a one and a zero -- and complete the solution.

And you do -- you give the other one the value one and zero.

And now complete the solution.

So let's complete that solution.

I'm looking for a vector in the null space, and it's absolutely going to be different from this guy, because, because any multiple of that zero is never going to give the one.

So I have somebody new in the null space, and let me finish it

out. What's x3 here?

So we're going by back substitution, looking at this equation.

Now x4 we've changed, we're doing the other possibility, where x2 is zero and x4 is one.

So x3 will happen to be minus two.

And now what do I get for that first equation?

x1 -- let's see.

Two x3s is minus four plus two -- do I get a two there?

Perhaps, yeah.

Two for x1, minus four, and two.

OK. That's in the null space.

What does that say about the columns?

That says that two of this column minus two of this column plus this column gives zero, which it does.

Two of that minus two of that and one of that gives the zero column. OK, now, now I've found another vector in the null space.

Now we're ready to tell me the whole null space.

What are all the solutions to Ax=0? I've got this guy and when I have him, what else is, goes into the null space along with that?

These are my two special solutions.

I call them special -- I just invented that name.

Special solutions.

What's special about them is the special numbers I gave to the free variables, the values zero and one for the free variables.

I could have given the free variables any values and got vectors in the null space.

But this was a good way to be sure I got t- got everybody. OK, so once I have him, I also have any multiple, right?

So I could take any multiple of that and it's in the null space.

And now what else -- I left a little space for what?

What -- a plus sign.

I can take any combination.

Here is a line of vectors in the null space.

A bunch of solutions.

Would you rather I say in the null space or would you rather I say, OK, I'm solving Ax=0? Well, really I'm solving Ux=0. Well, OK, let me put in that crucial plus sign.

I'm taking all the combinations of my two special solutions.

That's my conclusion there. The null space contains, contains exactly all the combinations of the special solutions.

And how many special solutions are there?

There's one for every free variable.

And how many free variables are there?

Oh, I mean, we can see all the whole picture now. If the rank R was two, this is the, this is the number of pivot variables, right, because it counted the pivots.

So how many free variables?

Well, you know it's two, right?

What is it in -- for a matrix that's m rows, n columns, n variables that means, with rank r?

How many free variables have we got left?

If r of the variables are pivot variables, we have n-r -- in this case four minus two -- free variables. Do you see that first of all we get clean answers here?

We get r pivot variables -- so there really were r equations here.

There looked like three equations, but there were really only two independent equations.

And there were n-r variables that we could choose freely, and we gave them those special zero one values, and we got the special solutions. OK. For me -- we could stop at that point.

That gives you a complete algorithm for finding all the solutions to A x equals zero. OK. Again, you do elimination -- going onward when a column, when there's nothing to be done on one column, you just continue.

There's this number r, the number of pivots, is crucial, and it leaves n-r free variables, which you give values zero and one to.

I would like to take one more step.

I would like to clean up this matrix even more.

So now I'm going to go to -- this is in its, this is in echelon form, upper triangular if you like.

I want to go one more step to make it as good as it can be. OK, so now I'm going to speak about the reduced row echelon form. OK, so now I'm going to speak about the matrix R, which is the reduced row echelon form.

So what does that mean?

That means I just -- I can, I can work harder on U.

So let me start, let me suppose I got as far as U, which was good.

Notice how that row of zeros appeared.

I didn't comment on that, but I should have. That row of zeros up here is because row three was a combination of rows one and two, and elimination discovered that fact.

When we get a row of zeros, that's telling us that the -- original row that was there was a combination of other rows, and elimination knocked it out. OK, so we got this far.

Now how can I clean that up further?

I can do, elimination upwards.

I can get zero above the pivots.

So this reduced row echelon form has zeros above and below the pivots.

So let me do that.

So now I'll subtract one of this from the row above.

That will leave a zero and a minus two in there.

And that's good. OK, and I can clean it up even one more step.

I can make the pivots -- the pivots I'm going to make equal to one, because I can divide equation two by the pivot.

That won't change the solutions.

So let me do that.

And then I really -- I'm ready to stop.

One, two, zero, minus two, zero, zero, one, two.

I divided the second equation by two, because now I have a one in the pivot and zeros below. OK. This is my matrix R.

I guess I'm hoping that you could now execute the whole algorithm. Matlab will do it immediately with the command -- reduced row echelon form of A.

So if I input that original matrix A and then I write, then I type that command, press return, that matrix will appear.

That's the reduced row echelon form, and it's got all the information as clear as can be.

What, what information has it got?

Well, of course it immediately tells me the pivot rows, pivot rows, one and two, pivot columns, one and three.

And in fact it's got the identity matrix in there, It's, it's got zeros above and below the pivots, right? and the pivots are one, so it's, so it's got a -- so notice the two by two identity matrix that's sitting in the pivot rows and pivot columns. it's I in the pivot rows and columns.

It's got zero rows below.

Those are always indicating that original rows were, were combinations of other rows.

So we really only had two rows there.

And now it also -- so there's the identity.

Now it's also got its free columns.

And, they're cleaned up as much as possible.

Actually, actually it's now so cleaned up that the special solutions, I can read off -- you remember I went through the steps of computing this -- doing back substitution?

Let me, let me, instead of that system, let me take this improved system.

So I'm going to use these numbers, right.

In these equations, what did I do?

I divided this equation by two and, oh yeah and I had subtracted two of this, which knocked out this guy and made that a minus sign.

Is that what -- I've now written Rx equals zero.

Now I guess I'm hoping everybody in this room understands the solutions to the original A x equals zero, the midway, halfway, U x equals zero, and the final R x equals zero are all the same.

Because going from one of those to another one I didn't mess up. I just multiplied equations and subtracted from other equations, which I'm allowed to do. OK. But my point is that now if I do this free variables and back substitution, it's just, the numbers are there.

When I let x -- so in this guy, I let x2 be one and x4 be zero.

I, I guess, what I seeing here?

Let me, let me sort of separate this out here.

I'm seeing in the pivot, in the pivot columns, if I, if I put the pivot columns here, I'm seeing those.

And I'm -- in the free columns I'm seeing -- what I seeing in the free columns?

A two, zero in that first free column, the x2 column, and a minus two, two in the fourth column, the other free column. And the row of zeros below, which of course have no -- that equation is zero equals zero.

That's satisfied.

Here's my point.

That when I do back substitution, these numbers are exactly what shows up -- oh, their signs get switched.

I was going to say those numbers, two, minus two, zero, two, can I just circle the -- this is the free part of the matrix.

This is the identity part.

This is the free part, maybe I'll call it F.

This, of course, I call I, because it's the identity.

The free part is a, I mean, I'm just doing back substitution.

And those free numbers will show up, with a minus sign, because they pop onto the other side of the equation -- so I see minus two, zero, and I see two, minus two. So that wasn't magic.

It had to happen.

Let me, show you clearly why it happened. OK, so that's -- this is what I'm interested in here.

And now let me, let me just, like, do it, do it for -- let's suppose we've, we've got to -- let's suppose we've got this system already in, in rref form.

So my matrix R is -- what does it look like?

OK, and I'll -- let me pretend that the pivot columns come first and then whatever's in the free columns.

And there might be some zero rows below.

There's a typical -- a pretty typical reduced row echelon form.

You see what's typical. It's got -- this is r by r. This is r pivot rows.

This is r pivot columns.

And here are n-r free columns.

OK. Tell me what are the special solutions?

What are the -- what's x?

If I want to solve R x equals zero -- in fact, let me -- I'm really going to, do the whole -- since these -- this is now block matrices, I might as well do all of the special solutions at once.

So I want to solve R x equals zero, and I'll have some special solutions.

Let me, actually -- can I do them all at once?

I'm going to create a null space matrix, OK. A matrix.

Its, its, its columns are the special -- the columns are the special solutions.

This is, I'm making it sound harder, it's going to be totally easy.

N will be this null space matrix.

I want R N to be the zero matrix.

These columns of N are supposed to multipl- to get multiplied by R and give zero columns.

So what N will do the job?

Let me put -- I'm going to put the identity in the free variable part and then minus F will show up in the pivot variables, just the way it did in that example.

There we had the identity and F.

Here -- in the special solution.

So these columns are -- there's the matrix of special solutions.

And actually, there -- so there's a Matlab command or a teaching code command, NULL -- N equal, so this is the -- produces the null basis, the null space matrix, NULL of A, and there it is. And how does that command actually work?

It uses Matlab to compute R, then it picks out the pivot variables, the free variables, puts, puts ones and zeros in for the free variables, and copies out the pivot variables.

It, it does back substitution, but back substitution for this system is totally simple.

What is this system?

R x equals zero.

So this is R is I F, and x is the pivot variables and the free variables, and it's supposed to give zero.

So what does that mean?

That means that the pivot variables plus F times the free variables give zero.

So let me put F times the free variables on the other side.

I get minus F times the free variables. There's my, equation, as simple as it can be. That's what back substitution comes to when I've reduced and reduced and reduced this system to the, to the best form, OK. And, then if the free variables, I make this special choice of the identity, then the pivot variables are

minus F. OK, can I do, another example?

Could you do another example?

Can I -- let me just take another matrix and, and let's go through this algorithm once more, OK. Here we go.

Here's a blackboard for another matrix, OK. So I'll call the matrix A again, but let me make it -- yeah, how big shall we make it this time?

Why don't I do this? Just for the heck of it.

Let me take the transpose of this A and see what happens to

that. Two four six eight and three six eight ten. Before we do the calculations, tell me what's coming?

How many pivot variables do you expect here?

How many columns are going to have pivots?

How many -- we have three columns in that matrix, but are we going to, are we going to have three pivots?

No, because this third columns is the sum of the first two

columns. I'm totally expecting, totally expecting that these will be pivot columns -- because they're independent, but that this third guy, the third column, which is dependent on the first two, is going to be a free column.

Elimination better discover that.

And elimination will also straighten out the rows, dependent rows and some independent rows.

What's the, what's the row reduced echelon form for this?

Let's just do it, OK. So, so that's the first pivot.

Two times that away from that gives me a row of zeros.

Two times that away from that gives me a zero two two.

And two times that away from that gives me a zero four four. OK, first column is straight.

First variable is a pivot variable.

No problem.

On to the second column.

I look at the second pivot, it's a zero.

I look below it.

There's a two. OK, I do a row exchange.

So this zero is now there.

I now have a perfectly good pivot, and I use it. OK, and I subtract two of that row away from this row.

All right if I do it like that?

I've got to the form U now.

This was my A.

Now there's my U. I can see now -- I have to stop, right?

I would go on to the third column.

I should have tried.

I quit, but without trying.

I shouldn't have done that.

On to the third column, look at the pivot position.

It's got a zero in it.

Look below, all zeros.

Now I'm entitled to stop, OK. So the rank is two again.

What about the null space?

How many special solutions are there this time?

How many special solutions for this matrix?

I've got -- and which are the free variables and which are the pivot variables and so on?

Pivot columns, I've got two pivot columns, and that's no accident.

The number of pivot columns for a matrix A, that's a great fact, that the number of pivot columns for A and A transpose are the same.

And then I have a free column.

There's a free column.

One free column, because the count is three minus two.

Three minus two gives me one free column. OK, so now let me solve, what's in the null space. OK, so how do I -- let's see.

These vectors have length three.

They only have three components.

I'm making too much space for the, to write x.

x has just got three components, and what are they?

I'm looking for the null space. OK, so how do I start?

I give the free variable some convenient value.

And what's that?

I set it to one.

I set the free variable to one.

If I set the free variable to zero and solve for the pivot variables, I'll get all zeros: no progress.

But by setting the free variable to one -- you see w- my two equations now are -- my equations are x1+ 2x2+ 3 x3 is zero, that's my first equation.

And my second equation is now 2x2+2x3 equals zero. And, OK. So if x3 is one, then x2 is minus one.

And if x3 is one and x2 is minus one, then maybe x1 is minus one.

And actually I go back to check now.

I don't, like -- I did a quick calculation mentally.

Can I mentally do a quick check?

That matrix, that solution x says that minus this column minus this column plus this one is the zero column.

And it is.

Minus that minus that plus that is zero.

So that's in the null space.

And now you can tell me what else is in the null space.

What's, what's the whole null space now? I multiply by C, right.

The whole null space is a line.

So that's the description.

You know, if I ask you on a homework or a quiz or the final what -- give me, describe, tell me the null space, find the null space of this matrix, you can take those steps.

And that's the answer I'm looking for.

And I'm looking for that C too, because that's telling me that you're remembering that it's a whole space and not just one vector.

Later I will ask you for a basis for the null space.

Then I just want this vector.

But if I ask for the whole null space, then there's the whole line through that vector. OK, now one more natural thing to do with this example, right, is keep going to the reduced matrix, R.

So can I push onwards to R?

That should be quick, but let's just practice.

Let me keep going to R. OK, so what do I do here?

I subtract -- I clear out above the pivot, so I subtract that from that, that's one zero one is left, right?

When I subtracted this row from this it produced a zero above this pivot.

And now I want that pivot to be a one.

So for the R matrix, I'll divide this equation by two, and of course these zero, zeros are great, they don't change.

There's R.

That's R.

You see what R is?

You see the identity matrix sitting up here?

You see the free part F, the F part here?

And you see the zeros below.

This is I F zero zero.

And what's the x?

The x has the identity -- well, it's only a single number one, but it's the identity matrix in the free, in the free part.

And what does it have in the pivot variables?

What did back substitution give?

It gave minus these guys.

You see that what this is is any multiple of -- this is the identity there, and this is minus F here. This is our null space matrix N for this.

Our, our null space matrix is the guy whose columns are the special solutions.

So their free variables have the special values one and, pivot variables have minus F.

So do you see, though, how the minus F just automatically shows up in the special solutions.

That's it really.

I don't think there's anything more I can say about A x equals zero.

There's more I can say about A x equal b, but that'll be on Friday. OK, so that's, that's the null space.

Thanks.

Free Downloads

Free Streaming

• VideoLectures.net

Video

• iTunes U (MP4 - 93MB)
• Internet Archive (MP4 - 173MB)

Caption

• English-US (SRT)