Lecture 19: Shortest Paths III: All-pairs Shortest Paths, Matrix Multiplication, Floyd-Warshall, Johnson

Related Resources

Lecture Notes (PDF)
Assignments
Exams

• Download English-US transcript (PDF)

-- shortest paths. This is the finale.

Hopefully it was worth waiting for.

Remind you there's a quiz coming up soon, you should be studying for it. There's no problem set due at the same time as the quiz because you should be studying now. It's a take-home exam.

It's required that you come to class on Monday.

Of course, you'll all come, but everyone watching at home should also come next Monday to get the quiz.

It's the required lecture. So, we need a bit of a recap in the trilogy so far. So, the last two lectures, the last two episodes, or about single source shortest paths. So, we wanted to find the shortest path from a source vertex to every other vertex.

And, we saw a few algorithms for this.

Here's some recap. We saw in the unweighted case, that was sort of the easiest where all the edge weights were one. Then we could use breadth first search. And this costs what we call linear time in the graph world, the number of vertices plus the number of edges. The next simplest case, perhaps, is nonnegative edge weights.

And in that case, what algorithm do we use?

Dijkstra, all right, everyone's awake.

Several answers at once, great.

So this takes almost linear time if you use a good heap structure, so, V log V plus E.

And, in the general case, general weights, we would use Bellman-Ford which you saw.

And that costs VE, good, OK, which is quite a bit worse. This is ignoring log factors.

Dijkstra is basically linear time, Bellman-Ford you're quadratic if you have a connected graph.

So, in the sparse case, when E is order V, this is about linear. This is about quadratic.

In the dense case, when E is about V^2, this is quadratic, and this is cubic.

So, Dijkstra and Bellman-Ford are separated by about an order of V factor, which is pretty bad.

OK, but that's the best we know how to do for single source shortest paths, negative edge weights, Bellman-Ford is the best. We also saw in recitation the case of a DAG. And there, what do you do?

Topological sort, yeah.

So, you can do a topological sort to get an ordering on the vertices. That you run Bellman-Ford, one round. This is one way to think of what's going on. You run Bellman-Ford in the order given by the topological sort, which is once, and you get a linear time algorithm.

So, DAG is another case where we know how to do well even with weights. Unweighted, we can also do linear time. But most of the time, though, will be, so you should keep these in mind in the quiz. When you get a shortest path problem, or what you end up determining is the shortest path problem, think about what's the best algorithm you can use in that case? OK, so that's single source shortest paths. And so, in our evolution of the Death Star, initially it was just nonnegative edge weights.

Then we got negative edge weights.

Today, the Death Star challenges us with all pair shortest paths, where we want to know the shortest path weight between every pair of vertices.

OK, so let's get some quick results.

What could we do with this case?

So, for example, suppose I have an unweighted graph. Any suggestions of how I should compute all pair shortest paths? Between every pair of vertices, I want to know the shortest path weight.

BFS, a couple more words? Yeah?

Right, BFS V times. OK, I'll say V times BFS, OK? So, the running time would be V^2 plus V times E, yeah, which is assuming your graph is connected, V times E.

OK, good. That's probably about the best algorithm we know for unweighted graphs.

So, a lot of these are going to sort of be the obvious answer.

You take your single source algorithm, you run it V times.

That's the best you can do, OK, or the best we know how to do. This is not so bad.

This is like one iteration of Bellman-Ford, for comparison. We definitely need at least, like, V^2 time, because the size of the output is V^2, shortest path weight we have to compute.

So, this is not perfect, but pretty good.

And we are not going to improve on that.

So, nonnegative edge weights: the natural thing to do is to run Dijkstra V times, OK, no big surprise.

And the running time of that is, well, V times E again, plus V^2, log V, which is also not too bad.

I mean, it's basically the same as running BFS.

And then, there's the log factor.

If you ignore the log factor, this is the dominant term.

And, I mean, this had an [added?] V^2 as well. So, these are both pretty good.

I mean, this is kind of neat. Essentially, the time it takes to run one Bellman-Ford plus a log factor, you can compute all pair shortest paths if you have nonnegative edge weights. So, I mean, comparing all pairs to signal source, this seems a lot better, except we can only handle nonnegative edge weights.

OK, so now let's think about the general case.

Well, this is the focus of today, and here's where we can actually make an improvement. So the obvious thing is V times Bellman-Ford, which would cost V^2 times E.

And that's pretty pitiful, and we're going to try to improve that to something closer to that nonnegative edge weight bound. So it turns out, here, we can actually make an improvement whereas in these special cases, we really can't do much better.

OK, I don't have a good intuition why, but it's the case. So, we'll cover something like three algorithms today for this problem.

The last one will be the best, but along the way we'll see some nice connections between shortest paths and dynamic programming, which we haven't really seen yet.

We've seen shortest path, and applying greedy algorithms to it, but today will actually do dynamic programming.

The intuition is that with all pair shortest paths, there's more potential subproblem reuse.

We've got to compute the shortest path from x to y for all x and y. Maybe we can reuse those shortest paths in computing other shortest paths.

OK, there's a bit more reusability, let's say.

OK, let me quickly define all pair shortest paths formally, because we're going to change our notation slightly.

It's because we care about all pairs.

So, as usual, the input is directed graph, so, vertices and edges. We're going to say that the vertices are labeled one to n for convenience because with all pairs, we're going to think of things more as an n by n matrix instead of edges in some sense because it doesn't help to think any more in terms of adjacency lists.

And, you have edge weights as usual.

This is what makes it interesting.

Some of them are going to be negative.

So, w maps to every real number, and the target output is a shortest path matrix. So, this is now an n by n matrix. So, n is just the number of vertices of shortest path weights.

So, delta of i, j is the shortest path weight from i to j for all pairs of vertices.

So this, you could represent as an n by n matrix in particular.

OK, so now let's start doing algorithms.

So, we have this very simple algorithm, V times Bellman-Ford, V^2 times E, and just for comparison's sake, I'm going to say, let me rewrite that, V times Bellman-Ford gives us this running time of V^2 E, and I'm going to think about the case where, let's just say the graph is dense, meeting that the number of edges is quadratic, and the number of vertices.

So in that case, this will take V^4 time, which is pretty slow. We'd like to do better.

So, first goal would just be to beat V^4, V hypercubed, I guess. OK, and we are going to use dynamic programming to do that. Or at least that's what the motivation will come from. It will take us a while before we can even beat V^4, which is maybe a bit pathetic, but it takes some clever insights, let's say.

OK, so I'm going to introduce a bit more notation for this graph. So, I'm going to think about the weighted adjacency matrix. So, I don't think we've really seen this in lecture before, although I think it's in the appendix. What that means, so normally adjacency matrix is like one if there's an edge, and zero if there isn't. And this is in a digraph, so you have to be a little bit careful.

Here, these values, the entries in the matrix, are going to be the weights of the edges.

OK, this is this if ij is an edge.

So, if ij is an edge in the graph, and it's going to be infinity if there is no edge. OK, in terms of shortest paths, this is a more useful way to represent the graph.

All right, and so this includes everything that we need from here. And now we just have to think about it as a matrix. Matrices will be a useful tool in a little while. OK, so now I'm going to define some sub problems. And, there's different ways that you could define what's going on in the shortest paths problem. OK, the natural thing is I want to go from vertex i to vertex j. What's the shortest path?

OK, we need to refine the sub problems a little but more than that. Not surprising.

And if you think about my analogy to Bellman-Ford, what Bellman-Ford does is it tries to build longer and longer shortest paths. But here, length is in terms of the number of edges. So, first, it builds shortest paths of length one. We've proven the first round it does that. The second round, it provides all shortest paths of length two, of count two, and so on.

We'd like to do that sort of analogously, and try to reuse things a little bit more. So, I'm going to say d_ij^(m) is the weight of the shortest path from i to j with some restriction involving m. So: shortest path from i to j using at most m edges. OK, for example, if m is zero, then we don't have to really think very hard to find all shortest paths of length zero.

OK, they use zero edges, I should say.

So, Bellman-Ford sort of tells us how to go from m to m plus one. So, let's just figure that out.

So one thing we know from the Bellman-Ford analysis is if we look at d_ij^(m-1), we know that in some sense the longest shortest path of relevance, unless you have negative weight cycle, the longest shortest path of relevance is when m equals n minus one because that's the longest simple path you can have.

So, this should be a shortest path weight from i to j, and it would be no matter what larger value you put in the superscript. This should be delta of i comma j if there's no negative weight cycles.

OK, so this feels good for dynamic programming.

This will give us the answer if we can compute this for all m.

Then we'll have the shortest path weights in particular.

We need a way to detect negative weight cycles, but let's not worry about that too much for now.

There are negative weights, but let's just assume for now there's no negative weight cycles.

OK, and we get a recursion recurrence.

And the base case is when m equals zero.

This is pretty easy. They have the same vertices, the weight of zero, and otherwise it's infinity.

OK, and then the actual recursion is for m.

OK, if I got this right, this is a pretty easy, intuitive recursion for d_ij^(m) is a min of smaller things in terms of n minus one. I'll just show the picture, and then the proof of that claim should be obvious.

So, this is proof by picture. So, we have on the one hand, I over here, and j over here.

We want to know the shortest path from i to j.

And, we want to use, at most, m edges.

So, the idea is, well, you could use m minus one edges to get somewhere. So this is, at most, m minus one edges, some other place, and we'll call it k. So this is a candidate for k.

And then you could take the edge directly from k to j.

So, this costs A_k^j, and this costs DIK m minus one.

OK, and that's a candidate path of length that uses, at most, m edges from I to j. And this is essentially just considering all of them. OK, so there's sort of many paths we are considering. All of these are candidate values of k. We are taking them in over all k as intermediate nodes, whatever.

So there they are. We take the best such path.

That should encompass all shortest paths.

And this is essentially sort of what Bellman-Ford is doing, although not exactly. We also sort of want to think about, well, what if I just go directly with, say, m minus one edges? What if there is no edge here that I want to use, in some sense?

Well, we always think about there being, and the way the A's are defined, there's always this zero weight edge to yourself.

So, you could just take a path that's shorter, go from d i to j, and j is a particular value of k that we might consider, and then take a zero weight edge at the end from A and jj. OK, so this really encompasses everything. So that's a pretty trivial claim. OK, now once we have such a recursion, we get a dynamic program.

I mean, there, this is it in some sense.

It's written recursively. You can write a bottom up.

And I would like to write it bottom up it little bit because while it doesn't look like it, this is a relaxation.

This is yet another relaxation algorithm.

So, I'll give you, so, this is sort of the algorithm. This is not a very interesting algorithm. So, you don't have to write it all down if you don't feel like it.

It's probably not even in the book.

This is just an intermediate step.

So, we loop over all m. That's sort of the outermost thing to do. I want to build longer and longer paths, and this vaguely corresponds to Bellman-Ford, although it's actually worse than Bellman-Ford. But hey, what the heck?

It's a stepping stone. OK, then for all i and j, and then we want to compute this min.

So, we'll just loop over all k, and relax.

And, here's where we're actually computing the min.

And, it's a relaxation, is the point.

This is our good friend, the relaxation step, relaxing edge. Well, it's not, yeah. I guess we're relaxing edge kj, or something, except we don't have the same clear notion. I mean, it's a particular thing that we're relaxing. It's not just a single edge because we don't have a single source anymore.

It's now relative to source I, we are relaxing the edge kj, something like that. But this is clearly a relaxation. We are just making the triangle inequality true if it wasn't before.

The tribal inequality has got to hold between all pairs.

And that's just implementing this min, right?

You're taking d ij. You take the min of what it was before in some sense. That was one of the possibilities we considered when we looked at the zero weight edge. We say, well, or you could go from i to some k in some way that we knew how to before, and then add on the edge, and check whether that's better if it's better, set our current estimate to that. And, you do this for all k.

In particular, you might actually compute something smaller than this min because I didn't put superscripts up here. But that's just making paths even better. OK, so you have to argue that relaxation is always a good thing to do.

So, by not putting superscripts, maybe I do some more relaxation, but more relaxation never hurts us. You can still argue correctness using this claim. So, it's not quite the direct implementation, but there you go, dynamic programming algorithm. The main reason I'll write it down: so you see that it's a relaxation, and you see the running time is n^4, OK, which is certainly no better than Bellman-Ford. Bellman-Ford was n^4 even in the dense case, and it's a little better in the sparse case. So: not doing so great.

But it's a start. OK, it gets our dynamic programming minds thinking. And, we'll get a better dynamic program in a moment. But first, there's actually something useful we can do with this formulation, and I guess I'll ask, but I'll be really impressed if anyone can see. Does this formula look like anything else that you've seen in any context, mathematical or algorithmic? Have you seen that recurrence anywhere else? OK, not exactly as stated, but similar. I'm sure if you thought about it for awhile, you could come up with it.

Any answers? I didn't think you would be very intuitive, but the answer is matrix multiplication. And it may now be obvious to you, or it may not. You have to think with the right quirky mind. Then it's obvious that it's matrix multiplication. Remember, matrix multiplication, we have A, B, and C. They're all n by n matrices.

And, we want to compute C equals A times B.

And what that meant was, well, c_ij was a sum over all k of a_ik times b_kj. All right, that was our definition of matrix multiplication.

And that formula looks kind of like this one.

I mean, notice the subscripts: ik and kj.

Now, the operators are a little different.

Here, we're multiplying the inside things and adding them all together. There, we're adding the inside things and taking them in. But other than that, it's the same. OK, weird, but here we go.

So, the connection to shortest paths is you replace these operators. So, let's take matrix multiplication and replace, what should I do first, plus this thing with min. So, why not just change the operators, replace dot with plus?

This is just a different algebra to work in, where plus actually means min, and dot actually means plus.

So, you have to check that things sort of work out in that context, but if we do that, then we get that c_ij is the min overall k of a_ik plus, a bit messy here, b_kj. And that looks like what we actually want to compute, here, for one value of m, you have to sort of do this m times.

But this conceptually is d_ij^(m), and this is d_ik^(m-1). So, this is looking like a matrix product, which is kind of cool.

So, if we sort of plug in this claim, then, and think about things as matrices, the recurrence gives us, and I'll just write this now at matrix form, that d^(m) is d^(m) minus one, funny product, A.

All right, so these are the weights.

These were the weighted adjacency matrix.

This was the previous d value. This is the new d value.

So, I'll just rewrite that in matrix form with capital letters. OK, I have the circle up things that are using this funny algebra, so, in particular, circled product. OK, so that's kind of nifty.

We know something about computing matrix multiplications. We can do it in n^3 time.

If we were a bit fancier, maybe we could do it in sub-cubic time. So, we could try to sort of use this connection. And, well, think about what we are computing here. We are saying, well, d to the m is the previous one times A.

So, what is d^(m)? Is that some other algebraic notion that we know? Yeah, it's the exponent.

We're taking A, and we want to raise it to the power, m, with this funny notion of product.

So, in other words, d to the m is really just A to the m in a funny way. So, I'll circle it, OK? So, that sounds good.

We also know how to compute powers of things relatively quickly, if you remember how. OK, for this notion, this power notion, to make sense, I should say what A to the zero means.

And so, I need some kind of identity matrix.

And for here, the identity matrix is this one, if I get it right. So, it has zeros along the diagonal, and infinities everywhere else.

OK, that sort of just to match this definition.

d_ij zero should be zeros on the diagonals and infinity everywhere else. But you can check this is actually an identity. If you multiply it with this funny multiplication against any other matrix, you get the matrix back. Nothing changes.

This really is a valid identity matrix.

And, I should mention that for A to the m to make sense, you really knew that your product operation is associative. So, actually A to the m circled makes sense because circled multiplication is associative, and you can check that; not hard because, I mean, min is associative, and addition is associative, and all sorts of good stuff. And, you have some kind of distributivity property. And, this is, in turn, because the real numbers with, and get the right order here, with min as your addition operation, and plus as your multiplication operation is a closed semi-ring. So, if ever you want to know when powers make sense, this is a good rule.

If you have a closed semi-ring, then matrix products on that semi-ring will give you an associative operator, and then, good, you can take products.

OK, that's just some formalism. So now, we have some intuition.

The question is, what's the right.

Algorithm? There are many possible answers, some of which are right, some of which are not.

So, we have this connection to matrix products, and we have a connection to matrix powers.

And, we have algorithms for both.

The question is, what should we do?

So, all we need to do now is to compute A to the funny power, n minus one. n minus one is when we get shortest paths, assuming we have no negative weight cycles. In fact, we could compute a larger power than n minus one. Once you get beyond n minus one, multipling by A doesn't change you anymore.

So, how should we do it? OK, you're not giving any smart answers. I'll give the stupid answer.

You could say, well, I take A.

I multiply it by A. Then I multiply it by A, and I multiply it by A, and I use normal, boring matrix to multiplication.

So, I do, like, n minus two, standard matrix multiplies. So, standard multiply costs, like, n^3. And I'm doing n of them.

So, this gives me an n^4 algorithm, and compute all the shortest pathways in n^4. Woohoo!

OK, no improvement. So, how can I do better?

Right, natural thing to try which sadly does not work, is to use the sub cubic matrix multiply algorithm.

We will, in some sense, get there in a moment with a somewhat simpler problem. But, it's actually not known how to compute shortest paths using fast matrix multiplication like Strassen's system algorithm.

But, good suggestion. OK, you have to think about why it doesn't work, and I'll tell you.

It's not obvious, so it's a perfectly reasonable suggestion. But in this context it doesn't quite work. It will come up in a few moments. The problem is, Strassen requires the notion of subtraction.

And here, addition is min. And, there's no inverse to min.

Once you take the arguments, you can't sort of undo a min.

OK, so there's no notion of subtraction, so it's not known how to pull that off, sadly.

So, what other tricks do we have up our sleeve?

Yeah? Divide and conquer, log n powering, yeah, repeated squaring.

That works. Good, we had a fancy way.

If you had a number n, you sort of looked at the binary number representation of n, and you either squared the number or squared it and added another factor of A.

Here, we don't even have to be smart about it.

OK, we can just compute, we really only have to think about powers of two. What we want to know, and I'm going to need a bigger font here because there's multiple levels of subscripts, A to the circled power, two to the ceiling of log n. Actually, n minus one would be enough. But there you go.

You can write n if you didn't leave yourself enough space like me, n the ceiling, n the circle.

This just means the next power of two after n minus one, two to the ceiling log. So, we don't have to go directly to n minus one. We can go further because anything farther than n minus one is still just the shortest pathways. If you look at the definition, and you know that your paths are simple, which is true if you have no negative weight cycles, then fine, just go farther.

Why not? And so, to compute this, we just do ceiling of log n minus one products, just take A squared, and then take the result and square it; take the result and square it.

So, this is order log n squares.

And, we don't know how to use Strassen, but we can use the boring, standard multiply of n^3, and that gives us n^3 log n running time, OK, which finally is something that beats Bellman-Ford in the dense case.

OK, in the dense case, Bellman-Ford was n^4.

Here we get n^3 log n, finally something better.

In the sparse case, it's about the same, maybe a little worse. E is order V.

Then we're going to get, like, V3 for Bellman-Ford.

Here, we get n^3 log n. OK, after log factors, this is an improvement some of the time.

OK, it's about the same the other times.

Another nifty thing that you get for free out of this, is you can detect negative weight cycles.

So, here's a bit of a puzzle. How would I detect, after I compute this product, A to the power to ceiling log n minus one, how would I know if I found a negative weight cycle?

What would that mean it this matrix of all their shortest paths of, at most, a certain length?

If I found a cycle, what would have to be in that matrix? Yeah?

Right, so I could, for example, take this thing, multiply it by A, see if the matrix changed at all.

Right, that works fine. That's what we do in Bellman-Ford. It's an even simpler thing.

It's already there. You don't have to multiply.

But that's the same running time.

That's a good answer. The diagonal would have a negative value, yeah.

So, this is just a cute thing. Both approaches would work, can detect a negative weight cycle just by looking at the diagonal of the matrix. You just look for a negative value in the diagonal. OK.

So, that's algorithm one, let's say.

I mean, we've seen several that are all bad, but I'll call this number one. OK, we'll see two more.

This is the only one that will, well, I shouldn't say that.

Fine, there we go. So, this is one dynamic program that wasn't so helpful, except it showed us a connection to matrix multiplication, which is interesting. We'll see why it's useful a little bit more. But, it bled to this nasty four nested loops. And, using this trick, we got down to n^3 log n. Let's try, just for n^3.

OK, just get rid of that log. It's annoying.

It makes you a little bit worse than Bellman-Ford, and the sparse case. So, let's just erase one of these nested loops. OK, I want to do that.

OK, obviously that algorithm doesn't work because it's for first decay, and it's not defined, but, you know, I've got enough variables.

Why don't I just define k to the m?

OK, it turns out that works. I'll do it from scratch, but why not? I don't know if that's how Floyd and Warshall came up with their algorithm, but here you go. Here's Floyd-Warshall.

The idea is to define the subproblems a little bit more cleverly so that to compute one of these values, you don't have to take the min of n things.

I just want to take the min of two things.

If I could do that, and I still only have n^3 subproblems, then I would have n^3 time.

So, all right, the running time of dynamic program is number of subproblems times the time to compute the recurrence for one subproblem. So, here's linear times n^3, and we want n^3 times constant. That would be good.

So that's Floyd-Warshall. So, here's the way we're going to redefine c_ij. Or I guess, there it was called d_ij. Good, so we're going to define something new. So, c_ij superscript k is now going to be the weight of the shortest path from I to j as before. Notice I used the superscript k instead of m because I want k and m to be the same thing.

Deep. OK, now, here's the new constraint. I want all intermediate vertices along the path, meeting all vertices except for I and j at the beginning and the end to have a small label.

So, they should be in the set from one up to k.

And this is where we are really using that our vertices are labeled one up to m. So, I'm going to say, well, first think about the shortest paths that don't use any other vertices. That's when k is zero.

Then think about all the shortest paths that maybe they use vertex one. And then think about the shortest paths that maybe use vertex one or vertex two.

Why not? You could define it in this way. It turns out, then when you increase k, you only have to think about one new vertex. Here, we had to take min over all k. Now we know which k to look at.

OK, maybe that made sense. Maybe it's not quite obvious yet. But I'm going to redo this claim, redo a recurrence. So, maybe first I should say some obvious things. So, if I want delta of ij of the shortest pathway, well, just take all the vertices. So, take c_ij superscript n.

That's everything. And this even works, this is true even if you have a negative weight cycle.

Although, again, we're going to sort of ignore negative weight cycles as long as we can detect them.

And, another simple case is if you have, well, c_ij to zero. Let me put that in the claim to be a little bit more consistent here.

So, here's the new claim. If we want to compute c_ij superscript zero, what is it?

Superscript zero means I really shouldn't use any intermediate vertices. So, this has a very simple answer, a three letter answer. So, it's not zero.

It's four letters. What's that?

Nil. No, not working yet.

It has some subscripts, too.

So, the definition would be, what's the shortest path weight from I to j when you're not allowed to use any intermediate vertices? Sorry?

So, yeah, it has a very simple name.

That's the tricky part. All right, so if i equals j, [LAUGHTER] you're clever, right, open bracket i equals j means one, well, OK.

It sort of works, but it's not quite right.

In fact, I want infinity if i does not equal j.

And I want to zero if i equals j, a_ij, good.

I think it's a_ij. It should be, right? Maybe I'm wrong.

Right, a_ij. So it's essentially not what I said. That's the point.

If i does not equal j, you still have to think about a single edge connecting i to j, right?

OK, so that's a bit of a subtlety.

This is only intermediate vertices, so you could still go from i to j via a single edge. That will cost a_ij.

If there is an edge: infinity.

If there isn't one: that is a_ij.

So, OK, that gets us started. And then, we want a recurrence.

And, the recurrence is, well, maybe you get away with all the vertices that you had before.

So, if you want to know paths that you had before, so if you want to know paths that use one up to k, maybe I just use one up to k minus one.

You could try that. Or, you could try using k.

So, either you use k or you don't.

If you don't, it's got to be this.

If you do, then you've got to go to k.

So why not go to k at the end? So, you go from I to k using the previous vertices. Obviously, you don't want to repeat k in there. And then, you go from k to j somehow using vertices that are not k.

This should be pretty intuitive.

Again, I can draw a picture. So, either you never go to k, and that's this wiggly line. You go from i to j using things only one up to k minus one. In other words, here we have to use one up to k.

So, this just means don't use k.

So, that's this thing. Or, you use k somewhere in the middle there. OK, it's got to be one of the two. And in this case, you go from i to k using only smaller vertices, because you don't want to repeat k.

And here, you go from k to j using only smaller labeled vertices. So, every path is one of the two. So, we take the shortest of these two subproblems. That's the answer.

So, now we have a min of two things.

It takes constant time to compute.

So, we get a cubic algorithm. So, let me write it down.

So, this is the Floyd-Warshall algorithm.

I'll write the name again. You give it a matrix A.

That's all it really needs to know.

It codes everything. You copy C to A.

That's the warm up. Right at time zero, C equals A. And then you just have these three loops for every value of k, for every value of i, and for every value of j. You compute that min.

And if you think about it a little bit, that min is a relaxation. Surprise, surprise.

So, that is the Floyd-Warshall algorithm.

And, the running time is clearly n^3, three nested loops, constant time inside. So, we're finally getting something that is never worse than Bellman-Ford.

In the sparse case, it's the same.

And anything denser, the number of edges is super linear. This is strictly better than Bellman-Ford. And, it's better than everything we've seen so far for all pair, shortest paths.

And, this handles negative weights; very simple algorithm, even simpler than the one before.

It's just relaxation within three loops.

What more could you ask for? And we need to check that this is indeed what min we're computing here, except that the superscripts are omitted.

That's, again, a bit of hand waving a bit.

It's OK to omit subscripts because that can only mean that you're doing more relaxation techniques should be.

Doing more relaxations can never hurt you.

In particular, we do all the ones that we have to. Therefore, we find the shortest path weights. And, again, here, we're assuming that there is no negative weight cycles.

It shouldn't be hard to find them, but you have to think about that a little bit. OK, you could run another round of Bellman-Ford, see if it relaxes in a new edges again. For example, I think there's no nifty trick for that version.

And, we're going to cover, that's our second algorithm for all pairs shortest paths. Before we go up to the third algorithm, which is going to be the cleverest of them all, the one Ring to rule them all, to switch trilogies, we're going to take a little bit of a diversion, side story, whatever, and talk about transitive closure briefly. This is just a good thing to know about. And, it relates to the algorithms we've seen so far. So, here's a transitive closure problem. I give you a directed graph, and for all pair vertices, i and j, I want to compute this number. It's one if there's a path from i to j. From i to j, OK, and then zero otherwise. OK, this is sort of like a boring adjacency matrix with no weights, except it's about paths instead of being about edges. OK, so how can I compute this?

That's very simple. How should I compute this?

This gives me a graph in some sense.

This is adjacency matrix of a new graph called the transitive closure of my input graph. So, breadth first search, yeah, good. So, all I need to do is find shortest paths, and if the weights come out infinity, then there's no path. If it's less than infinity, that there's a path. And so here, so you are saying maybe I don't care about the weights, so I can run breadth first search n times, and that will work indeed. So, if we do B times B of S, so it's maybe weird that I'm covering here in the middle, but it's just an interlude. So, we have, then, something like V times E. OK, you can run any of these algorithms. You could take Floyd-Warshall for example. Why not?

OK, then it would just be V^ I mean, you could run in any of these algorithms with weights of one or zero, and just check whether the values are infinity or not.

So, I mean, t_ij equals zero, if and only if the shortest path weight from i to j is infinity.

So, just solve this. This is an easier problem than shortest paths. It is, in fact, strictly easier in a certain sense, because what's going on with transitive closure, and I just want to mention this out of interest because transitive closure is a useful thing to know about. Essentially, what we are doing, let me get this right, is using a different set of operators.

We're using or and and, a logical or and and instead of min and plus, OK, because we want to know, if you think about a relaxation, in some sense, maybe I should think about it in terms of this min.

So, if I want to know, is there a pathway from I to j that uses vertices labeled one through k in the middle?

Well, either there is a path that doesn't use the vertex k, or there is a path that uses k, and then it would have to look like that. OK, so there would have to be a path here, and there would have to be a path there.

So, the min and plus get replaced with or and and.

And if you remember, this used to be plus, and this used to be product in the matrix world.

So, plus is now like or. And, multiply is now like and, which sounds very good, right?

Plus does feel like or, and multiply does feel like and if you live in a zero-one world. So, in fact, this is not quite the field Z mod two, but this is a good, nice, field to work in. This is the Boolean world.

So, I'll just write Boole. Good old Boole knows all about this. It's like his master's thesis, I think, talking about Boolean algebra.

And, this actually means that you can use fast matrix multiply. You can use Strassen's algorithm, and the fancier algorithms, and you can compute the transitive closure in subcubic time.

So, this is sub cubic if the edges are sparse.

But, it's cubic in the worst case if there are lots of edges.

This is cubic. You can actually do better using Strassen. So, I'll just say you can do it. No details here.

I think it should be, so in fact, there is a theorem.

This is probably not in the textbook, but there's a theorem that says transitive closure is just as hard as matrix multiply.

OK, they are equivalent. Their running times are the same. We don't know how long it takes to do a matrix multiply over a field.

It's somewhere between n^2 and n^2.3.

But, whatever the answer is: same for transitive closure.

OK, there's the interlude. And that's where we actually get to use Strassen and friends. Remember, Strassen was n to the log base two of seven algorithm. Remember that, especially on the final. Those are things you should have at the tip of your tongue. OK, the last algorithm we're going to cover is really going to build on what we saw last time: Johnson's algorithm. And, I've lost some of the running times here. But, when we had unweighted graphs, we could do all pairs really fast, just as fast as a single source Bellman-Ford. That's kind of nifty.

We don't know how to improve Bellman-Ford in the single source case. So, we can't really help to get anything better than V times E. And, if you remember running V times Dijkstra, V times Dijkstra was about the same. So, just put this in the recall bubble here: V times Dijkstra would give us V times E plus V^2 log V. And, if you ignore that log factor, this is just VE. OK, so this was really good.

Dijkstra was great. And this was for nonnegative edge weights. So, with negative edge weights, somehow we'd like to get the same running time.

Now, how might I get the same running time?

Well, it would be really nice if I could use Dijkstra.

Of course, Dijkstra doesn't work with negative weights.

So what could I do? What would I hope to do?

What could I hope to? Suppose I want, in the middle of the algorithm, it says run Dijkstra n times.

Then, what should I do to prepare for that?

Make all the weights positive, or nonnegative.

Why not, right? We're being wishful thinking.

That's what we'll do. So, this is called graph re-weighting. And, what's cool is we actually already know how to do it. We just don't know that we know how to do it. But I know that we know that we know how to do it. You don't yet know that we know that I know that we know how to do it.

So, it turns out you can re-weight the vertices.

So, at the end of the last class someone asked me, can you just, like, add the same weight to all the edges? That doesn't work.

Not quite, because different paths have different numbers of edges. What we are going to do is add a particular weight to each vertex.

What does that mean? Well, because we really only have weights on the edges, here's what well do.

We'll re-weight each edge, so, (u,v), let's say, going to go back into graph speak instead of matrix speak, (u,v) instead of I and j, and we'll call this modified weight w_h. h is our function.

It gives us a number for every vertex.

And, it's just going to be the old weight of that edge plus the weight of the start vertex minus the weight of the terminating vertex. I'm sure these have good names.

One of these is the head, and the other is the tail, but I can never remember which. OK, so we've directed edge (u,v). Just add one of them; subtract the other. And, it's a directed edge, so that's a consistent definition.

OK, so that's called re-weighting.

Now, this is actually a theorem.

If you do this, then, let's say, for any vertices, u and v in the graph, for any two vertices, all paths from u to v have the same weight as they did before, well, not quite.

They have the same re-weighting.

So, if you look at all the different paths and you say, well, what's the difference between vh, well, sorry, let's say delta, which is the old shortest paths, and deltas of h, which is the shortest path weights according to this new weight function, then that difference is the same.

So, we'll say that all these paths are re-weighted by the same amounts. OK, this is actually a statement about all paths, not just shortest paths.

There we go. OK, to how many people is this obvious already? A few, yeah, it is. And what's the one word?

OK, it's maybe not that obvious.

All right, shout out the word when you figure it out.

Meanwhile, I'll write out this rather verbose proof.

There's a one word proof, still waiting.

So, let's just take one of these paths that starts at u and ends at v. Take any path.

We're just going to see what its new weight is relative to its old weight. And so, let's just write out w_h of the path, which we define in the usual way as the sum over all edges of the new weight of the edge from v_i to v_i plus one. Do you have the word?

No? Tough puzzle then, OK. So that's the definition of the weight of a path. And, then we know this thing is just w of v_i, v_i plus one.

I'll get it right, plus the weight of the first vertex, plus, sorry, the re-weighting of v_i minus the re-weighting of v_i plus one.

This is all in parentheses that's summed over I.

Now I need the magic word. Telescopes, good.

Now this is obvious: each of these telescopes with an extra previous, except the very beginning and the very end. So, this is the sum of these weights of edges, but then outside the sum, we have plus h of v_1, and minus h of v_k.

OK, those guys don't quite cancel.

We're not looking at a cycle, just a path.

And, this thing is just w of the path, as this is the normal weight of the path. And so the change, the difference between w_h of P and w of P is this thing, which is just h of u minus h of v.

And, the point is that's the same as long as you fix the endpoints, u and v, of the shortest path, you're changing this path weight by the same thing for all paths. This is for any path from u to v, and that proves the theorem. So, the one word here was telescopes. These change in weights telescope over any path. Therefore, if we want to find shortest paths, you just find the shortest paths in this re-weighted version, and then you just change it by this one amount. You subtract off this amount instead of adding it. That will give you the shortest path weight in the original weights.

OK, so this is a tool. We now know how to change weights in the graph. But what we really want is to change weights in the graph so that the weights all come out nonnegative. OK, how do we do that?

Why in the world would there be a function, h, that makes all the edge weights nonnegative?

It doesn't make sense. It turns out we already know.

So, I should write down this consequence.

Let me get this in the right order.

So in particular, the shortest path changes by this amount. And if you want to know this value, you just move the stuff to the other side.

So, we compute deltas of h, then we can compute delta.

That's the consequence here. How many people here pronounce this word corollary? OK, and how many people pronounce it corollary? Yeah, we are alone.

Usually get at least one other student, and they're usually Canadian or British or something.

I think that the accent. So, I always avoid pronouncing his word unless I really think, it's corollary, and get it right. I at least say Z not Zed.

OK, here we go. So, what we want to do is find one of these functions. I mean, let's just write down what we could hope to have. We want to find a re-weighted function, h, the signs of weight to each vertex such that w_h of (u,v) is nonnegative. That would be great for all edges, all (u,v) in E. OK, then we could run Dijkstra.

We could run Dijkstra, get the delta h's, and then just undo the re-weighting, and get what we want. And, that is Johnson's algorithm. The claim is that this is always possible. OK, why should it always be possible? Well, let's look at this constraint. w_h of (u,v) is that.

So, it's w of (u,v) plus h of u minus h of V should be nonnegative. Let me rewrite this a little bit. I'm going to put these guys over here. That would be the right thing, h of v minus h of u is less than or equal to w of (u,v).

Does that look familiar? Did I get it right?

It should be right. Anyone seen that inequality before? Yeah, yes, correct answer.

OK, where? In a previous lecture?

In the previous lecture. What is this called if I replace h with x? Charles knows.

Good, anyone else remember all the way back to episode two?

I know there was a weekend. What's this operator called?

Not subtraction but, I think I heard it, oh man. All right, I'll tell you.

It's a difference constraint, all right?

This is the difference operator.

OK, it's our good friend difference constraints.

So, this is what we want to satisfy.

We have a system of difference constraints.

h of V minus h of u should be, we want to find these.

These are our unknowns. Subject to these constraints, we are given the w's. Now, we know in these difference constraints are satisfiable.

Can someone tell me when these constraints are satisfiable?

We know exactly when for any set of difference constraints.

You've got to remember the math.

Terminology, I can understand.

It's hard to remember words unless you're a linguist, perhaps. So, when is the system of different constraints satisfiable?

All right, you should definitely, very good.

[LAUGHTER] Yes, very good.

Someone brought their lecture notes: when the constraint graph has no negative weight cycles. Good, thank you.

Now, what is the constraint graph?

OK, this has a one letter answer more or less.

I'll accept the one letter answer.

What? A?

A: close. G.

Yeah, I mean, same thing.

Yeah, so the constraint graph is essentially G.

Actually, it is G. The constraint graph is G, good. And, we prove this by adding a new source for text, and connecting that to everyone. But that's sort of beside the point. That was in order to actually satisfy them. But this is our characterization. So, if we assume that there are no negative weight cycles in our graph, which we've been doing all the time, then we know that this thing is satisfiable. Therefore, there is an assignment of this h's. There is a re-weighting that makes all the weights nonnegative.

Then we can run Dijkstra. OK, we're done.

Isn't that cool? And how do we satisfy these constraints? We know how to do that with one run of Bellman-Ford, which costs order VE, which is less than V times Dijkstra.

So, that's it, write down the details somewhere.

So, this is Johnson's algorithm.

This is the fanciest of them all.

It will be our fastest, all pairs shortest path algorithm. So, the claim is, we can find a function, h, from V to R such that the modified weight of every edge is nonnegative for every edge, (u,v), in our graph. And, we do that using Bellman-Ford to solve the difference constraints.

These are exactly the different constraints that we were born to solve that we learned to solve last time.

The graphs here are corresponding exactly if you look back at the definition. Or, Bellman-Ford will tell us that there is a negative weight cycle.

OK, great, so it's not that we really have to assume that there is no negative weight cycle. We'll get to know.

And if your fancy, you can actually figure out the minus infinities from this. But, at this point, I just want to think about the case where there is no negative weight cycle. But if there is, we can find out that it exists, and that just tell the user.

OK, then we'd stop. Otherwise, there is no negative weight cycle. Therefore, there is an assignment that gives is nonnegative edge weights.

So, we just use it. We use it to run Dijkstra.

So, step two is, oh, I should say the running time of all this is V times E. So, we're just running Bellman-Ford on exactly the input graph.

Plus, we add a source, if you recall, to solve a set of difference constraints.

You add a source vertex, S, connected to everyone at weight zero, run Bellman-Ford from there because we don't have a source here. We just have a graph.

We want to know all pairs. So, this, you can use to find whether there is a negative weight cycle anywhere.

Or, we get this magic assignment.

So now, w_h is nonnegative, so we can run Dijkstra on w_h.

We'll say, using w_h, so you compute w_h.

That takes linear time. And, we run Dijkstra for each possible source. I'll write this out explicitly.

We've had this in our minds several times.

But, when we said n times Dijkstra over n times BFS, here it is. We want to compute delta sub h now, of (u,v) for all V, and we do this separately for all u. And so, the running time here is VE plus V^2 log V. This is just V times the running time of Dijkstra, which is E plus V log V.

OK, it happens that this term is the same as this one, which is nice, because that means step one costs us nothing asymptotically. OK, and then, last step is, well, now we know delta h.

We just need to compute delta. So, for each pair of vertices, we'll call it (u,v), we just compute what the original weights would be, so what delta (u,v) is.

And we can do that using this corollary.

It's just delta sub h of (u,v) minus h of u plus h of v.

I got the signs right. Yeah, so this takes V^2 time, also dwarfed by the running time of Dijkstra.

So, the overall running time of Johnson's algorithm is just the running time of step two, running Dijkstra n times -- -- which is pretty cool. When it comes to single source shortest paths, Bellman-Ford is the best thing for general weights. Dijkstra is the best thing for nonnegative weights. But for all pair shortest paths, we can skirt the whole negative weight issue by using this magic we saw from Bellman-Ford.

But now, running Dijkstra n times, which is still the best thing we know how to do, pretty much, for the all pairs nonnegative weights, now we can do it for general weights too, which is a pretty nice combination of all the techniques we've seen.

In the trilogy, and along the way, we saw lots of dynamic programming, which is always good practice. Any questions?

This is the last new content lecture before the quiz.

On Wednesday it will be quiz review, if I recall correctly.

And then it's Thanksgiving, so there's no recitation.

And then the quiz starts on Monday.

So, study up. See you then.

Free Downloads

Free Streaming

• VideoLectures.net

Video

• iTunes U (MP4 - 156MB)
• Internet Archive (MP4 - 307MB)

Caption

• English-US (SRT)