Lecture 12: Searching and Sorting

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

 

PROFESSOR: So, for the last two lectures we've been talking about analyzing algorithms, complexity, orders of growth. How do we estimate the cost of an algorithm as the size of the input grows? And as I've said several times, I'll say at least once more, how do we also turn it the other direction? How do we use thoughts about choices of pieces of algorithm in terms of implications on the cost it's going to take us to compute?

We saw last time a set of examples-- constant algorithms, linear algorithms, logarithmic algorithms, linear algorithms, quadratic algorithms, exponential algorithms. Today, what I'm going to do is fill in one more piece, a log linear algorithm-- something that's really a nice kind of algorithm to have-- and use it to talk about one last class of algorithms that are really valuable, and those are searching and sorting algorithms.

So a search algorithm. Kind of an obvious statement. You use them all the time when you go to Google or Bing or whatever your favorite search mechanism on the web is. It's just a way to find an item or a group of items from a collection. If you think about it, that collection could be either implicit or explicit. So way back at the beginning of the term, we saw an example of a search algorithm when you were looking for square roots. And we saw simple things like exhaustive enumeration. We'd go through all the possibilities. We saw our first version of bisection search there, where you would do approximations. Newton-Raphson-- these are all examples of a search algorithm where the collection is implicit. So all the numbers between some point that some other point.

More common is a search algorithm where the collection is explicit. I don't know. For example, I've got all the data records of students and I want to know how do I find a particular student, so I can record that A plus that everybody in this room is going to get next Tuesday on that exam? That's not a promise. Sorry. But we'll work on it. So could do it implicit, could do it explicit. Today I want to focus on doing search explicitly. And it could be on different kinds of collections, but I'm going to focus-- just as an example-- on search over lists. And to make it a little easier, let's just do search over lists of numbers. But it could obviously be other kinds of elements.

Now you've already seen some of this, right? We did search where we said, we can do linear search. Brute force. Just walk down the list looking at everything till we either find the thing we're looking for or we get to the end of the list. Sometimes also called British Museum algorithm or exhaustive enumeration. I go through everything in the list. Nice news is, the list doesn't have to be sorted. It could be just in arbitrary order. What we saw is that the expected-- sorry, not expected. The worst case behavior is linear. In the worst case, the element's not in the list. I got to look at everything. So it's going to be linear in terms of complexity.

And then we looked at bisection search, where we said the list needs to be sorted. But if it is, we can actually be much more efficient because we can take advantage of the sorting to cut down the size of the problem. And I'll remind you about both of those. There was our simple little linear search. Right? Set a flag that says, I haven't yet found it. And then just loop over the indices into the list. I could have also just looped directly over the list itself, checking to see if the ith member of the list is the thing I'm looking for. If it is, change the flag to true so that when I come out of all of this I'll return the flag-- either false because it was set that way initially or true because I found it.

And of course what we knew is we have to look at everything to see if it's there or not. I could speed this up by just returning true at this point. While that would improve the average case, doesn't improve the worst case. And that's the thing we usually are concerned about, because in the worst case I've got to go through everything. And just to remind you, we said this is order length of the list. To go around this part-- the loop right here-- and inside the loop, it's constant work. I'm doing the same number of things each time. That's order n times order 1. And by our rules, that's just order n. So it's linear in the size of the problem.

OK. We said we could do it on sorted lists. But just again, we'll walk down the list. Again, here I could loop over everything in the list, checking to see if it's the thing I want. Return true. And if I ever get to a point where the element of the list is bigger than the thing I'm looking for, I know it can't be in the rest of the list because all the things to the right are bigger yet. I could just Return false and drop out. In terms of average behavior, this is better because it's going to stop as soon as it gets to a point where it can rule everything else out. But in terms of complexity, it's still order n. Because I still on average have-- not average. In the worst case, I'm still going to be looking n times through the loop before I get to a point where I can decide to bail out of it. So order n.

And then finally-- last piece of recap-- bisection search. Repeat again. The idea here is, take the midpoint of the list. Look at that element. If it's the thing I'm looking for, great. I just won the lottery. If it isn't, decide is the thing I'm looking for bigger or less than that middle point. If it's bigger than that, I only use the upper half of the list. If it's less than that, I only use the lower half of the list. And the characteristic here was, at each step, I'm reducing the size of the problem in half. I'm throwing away half of the remaining list at each step.

And I'll just remind you of that code. I know it's a lot here, but just to remind you. It said, down here if I've got an empty list, it can't be there. I'm going to Return false. Otherwise call this little helper function with the list, the thing for which I'm searching, and the beginning and end point indices into the list. Initially the start and the very end. And this code up here basically says, if those two numbers are the same I'm down to a list of one. Just check to see if it's the thing I'm looking for. Otherwise, pick something halfway in between. And ignore this case for the moment. Basically then check to see, is the thing at that point bigger than e? In which case, I'm in general going to call this only with from the low point to the midpoint. Otherwise I'm going to call this with the midpoint to high. And that was just this idea of, keep cutting down in half the size of the list.

Last piece of the recap-- the thing we wanted you to see here-- is there are the two recursive calls. I'm only going to do one because I'm making a decision. At each step, I'm cutting down the problem by half. And that says the number of steps, the number of times I'm going to iterate through here, will be log in the length of the list. And if that still doesn't make sense to you, it says, I need to know when 1 over 2 to the k-- where k is the number of steps-- is equal to 1. Because in each step, I'm reducing by half. And that's when k is log base 2 of n. So that's why it's log linear.

And so this just reminds you. Again, that recap. Number of calls reduced-- or, sorry. The call gets reduced by a factor or two each time. I'm going to have a log n work going around it. And inside it's a constant amount of work because I'm just passing the pointers, I'm not actually copying the list. And that's a nice state to be.

OK, so-- sounds good. Could just use linear search. It's going to be linear. When you use binary search or bisection search, we can do it in log time. That's great. We assumed the list was sorted, but all right. So that lens basically says, OK. So when does it make sense to sort the list and then do the search? Right? Because if I can sort the list cheaply, then the search is going to be logarithmic. That's really what I would like.

This little expression basically says, let's let sort be the cost of sorting the list. I want to know when that cost plus something that's order log n-- which is what it's going to cost me to do this search. When is that less than something that's order n? Because then it's going to be better to do the sort first than do the search. And so I can just rearrange it. It needs to be, when does the cost of sorting-- when is it last than this expression? Which basically says, when is sorting going to be less expensive than the linear cost?

Crud. Actually, good news for you, right? This is a really short lecture. Because it says it's never true. Ouch. Don't worry. We've got more to go on the lecture. The reason it can't be true-- if you think about it just informally-- is, if I've got a collection of n elements and I want to sort it, I've got to look at each one of those elements at least once. Right? I have to look at them to decide where they go. Oh, that's n elements. So sorting must be at least order n, because I got to look at everything. And in fact as it says there, I'm going to have to use at least linear time to do the sort.

Sounds like we're stuck, but we're not. And the reason is, often when I want to search something I'm going to do multiple searches, but I may only want to sort the list once. In fact, I probably only want to sort the list once. So in that case, I'm spreading out the cost. I'm amortizing the expense of the sort.

And now what I want to know is, if I'm going to do k searches, the cost of those k searches I know is going to be k log n-- because it's log to do the search. And I simply need to know, is the cost of sorting plus this-- can I have something where it's less than k searches just using linear search? And the answer is, yes. There are going to be, for large k's, ways in which we can do the sort where the sort time becomes irrelevant, that the cost is really dominated by this search.

And so what I want to do now is look at-- all right. How could we do the sort reasonably efficiently? It's going to have to be at least linear. We're going to see it's going to be a little more than linear. But if I could do it reasonably, I'm going to be in good shape here. So what I want to do is show you a number of ways in which we can do sorting-- take a list of elements and sort them from, in this case, smaller to higher or increasing order. So here's my goal. I want to efficiently sort a list. I want to see if we can do this as efficiently as possible.

I'm going to start, you might say, with a humorous version of sort. You're all convinced that my humor is non-existent. You're right. But it sets the stage for it. This is a sort. You can look it up. It's called monkey sort, BOGO sort, stupid sort, slow sort, permutation sort, shotgun sort. And here's how it works. Anna has nicely given me a set of numbers on cards here. Here's how you do BOGO sort. I got to do that better. I got to spread them out randomly, like this. Oh good. I'm going to have to-- sorry, Tom. I'm not walking. And now I pick them up, saying, is that less than this? Which is less than-- oh, crud. They're not sorted. All right. I pick them all up and I do it again. A little brain damage, right?

Now it's intended to get your attention. I did. I heard a couple of chuckles. Those are A students, by the way. I heard a couple of chuckles here. We could actually do this exhaustively. Basically it's called permutation sort because you could search through all possible permutations to see if you find something that's sorted. That, by the way-- the complexity of that is something like n factorial, which for large n is n to the nth power. And if n's anything bigger than about 2, don't do it. Right? But it would be a way to think about doing this.

All right. Now, having caught the humorous version of this, how could we do this a little bit better? Oh sorry. I should say, what's the complexity? There's a nice crisp definition of BOGO sort. Its best case is order n, because I just need to check it's sorted. Its average case is n factorial and its worst case, if I'm just doing it randomly, is God knows. Because I could be doing it here forever. So we're going to move on.

Here's a second way to do it called bubble sort. I'm going to do this with a small version of this. I'm going to put out a set. I'll turn these up so you can see them in a second. The idea of bubble sort is, I'm going to start at-- I'm going to call this the front end of the list. And I'm going to walk down, comparing elements pairwise. And I'm always going to move the larger one over. So I start here and I say, 1 is less than 11. I'm OK. 11's bigger than five. I'm going to bubble that up. 11's bigger than 6. I'm going to bubble that up. 11's bigger than 2. I've basically bubbled 11 to the end.

Now I go back here. I say, 1 is less than 5. That's good. 5 is less than 6. That's good. Ah, 6 is bigger than 2. Bubble that. 6 is less than 11. You get the idea-- comparison, comparison, and swap. Comparison, comparison. And now if I go back to this part and do it, you'll notice that's in the right order. That's in the right order. That's in the right order. That's in the right order. I'm done. Small round of applause, please. I was able to sort five elements. Thank you.

The little video is showing the same thing. You can see the idea here. It's called bubble sort because you're literally bubbling things up to the end of the list. It's pretty simple to do. You're just swapping pairs. And as you saw, when I get to the end of the list I go back and do it until I have a pass where I go all the way through the list and I don't do any swaps. And in that case I know I'm done because everything's in order, and I can stop. One of the properties of it is that the largest unsorted element is always at the end after the pass. In other words, after the first one I know that the largest element's at the end. After the second one, the largest thing left is going to be in the next place. And that tells me, among other things, that this is going to take no more than n times through the list to succeed. It might actually take fewer than that.

OK. Again let's look at some code for it. Let's look at its complexity and let's actually run this. So here is a little simple version of bubble sort. I'm going to set a flag up here. I'm going to call its swap initially to false. That's going to let me tell when I'm done, when I've gone through everything in the list without doing a swap. And then I'm going to loop. As long as swap is false-- so the first time through it's going to do that loop. I set swap initially to true, and notice what I then do. I let j range from 1 up to the length of the list, and I look at the jth element and the previous element. If the previous element is bigger, I'm going to flip them. Right there. And that's just doing that swap, what I just did down here.

And if that's the case, I'm going to set the flag to false. Which says, I've done at least one bubble as part of this. Which means when I come out of here and go back around to the loop, it's going to do it again. And it will do it until all of this succeeds without this ever being true, in which case that's true, which makes that false. And it will drop out. OK?

Let's look at an example of this running. It's just to give you a sense of that, assuming I can find the right place here. So there is, again, a version of bubble sort on the side. And I'm going to bring this down to the bottom I've got a little test list there. And I've put a print statement in it. So you can see each time through the loop, what's the form of the list as it starts. And assuming I've done this right-- here you go.

There's the list the first time through. Notice after one pass, 25's at the end of the list-- the biggest element. Exactly what I like. But you can also see a few other things have flipped. Right? Right in there, there have been some other swaps as it bubbled through. And in fact, you can see it's-- well, you can see that idea. You can see 25 moving through. Notice on the next step, a whole bunch of the list is actually in the right order. It's just because I got lucky. All I can guarantee is that the second largest element is the second from the end of the list. But you can see here. Even though the list is, I think, nine long, it only took us four passes through. So this is nice. It says, at most, n times through the list. And at the end, we actually get out something that's in the right form.

OK. So let's go back to this and basically say, what's the complexity? Well that's length n, right? Has to be. I'm going through the entire list. And inside of there is just constant work. Four operations. I'm doing a test. Sorry, five. I'm doing a test. And then depending whether that test is true or not, I'm setting a flag and doing some movement of things around. But it's just constant. I don't care about the five. And there, how many times do I go around the loop? In the worst case, n. All I can guarantee is, after the first pass the biggest thing is here. After the second pass, the second biggest thing is there. After the third pass-- you get the idea.

So I've got order and things inside the loop, and I'm doing that loop n times. And I hope that looks familiar. We've talked about this. Right? This is nested loops. What's this? Quadratic. So it's order n squared, where n is the length of the list. Now as you also saw, on average, it could be less than that. But it's going to be order n squared.

OK. That's one possibility. Here's a second nice, simple, sort algorithm. It's called selection sort. You can kind of think of this as going the other way. Not completely, but going the other way. And when I say going the other way, the idea here is that I'm going to find the smallest element in the list. And I'm going to stick it at the front of the list when I'm done, and simply swap that place with whatever was there. Flip them. I might do a few other flips along the way, depending how I implement this.

Next, pass. I'm just going to look at everything but the first element, because I know that one's done. I'm going to do the same thing. Find the smallest element remaining in the list, put it in the second spot, and keep doing that. What I know is, if I implement this correctly, after i steps the first i elements of the list will be sorted. And everything in the rest of the list has to be bigger than the largest thing in the first part of the list.

OK. So we could build that. Before we do it, I'm going to show you a little video starring Professor Guttag. This is his cameo performance here. But I want to just show you an example of this using not numbers, but people.

[VIDEO PLAYBACK]

- All right. So now we're going to do selection sort. The idea here is that each step we're going to select the shortest person and put them next in line of the sorted group. So we'll bring the leftmost person forward, and we will compare her to everybody else. So one at a time, step forward. You're still the winner. You go back. Please step forward.

PROFESSOR: And watch the number of comparisons that go on, by the way. We're going to come back to that.

- Next. Still the winner. Next. Ah. A new winner. All right. So you can take her place.

PROFESSOR: So here, we're choosing to actually insert into the spot in the Line We could have put her back at the front, but either one will work.

- Now we'll compare. Same old winner. Same winner. No change. It's getting kind of boring. Don't fall, that-- same winner. Please.

PROFESSOR: This is a tough one.

- Oh. Close, but I think you're still shorter. All right. Next. No change, which means you are the first in line. Congratulations.

PROFESSOR: So, smallest element now going to be the first slot.

- Now you step forward, and we'll compare you.

 

PROFESSOR: I would invite you to watch the left hand of the list. Notice how it is slowly building up at each stage to have that portion sorted. And we deliberately admit students to be of different heights, so John can do this demo.

- You are the winner. Take your place in line. Next. It's you. And once again, we have a lovely group of students sorted in height order.

[END PLAYBACK]

[APPLAUSE]

PROFESSOR: And check out-- I want you to remember number of comparisons-- 55. Not that the [INAUDIBLE], but I want you to see a comparison as we go on in a second.

So again, selection sort. This is this idea of, find the smallest element. Put it at the front. I might do a little number of flips, as you can see, here along the way. But this is the same animation of that. So let's first of all convince ourselves it will do the right thing, and then look at some code, and then run the code.

So to convince ourselves that this is going to do the right thing, we could talk about something that we often refer to as a loop invariant. We're going to write a loop, but we're going to walk through this. And the invariant here-- and we want to just demonstrate if it's true at the beginning and it's true at each step. Therefore, by induction as we did earlier, I can conclude it's true always. Is that if I'm given the prefix or the first part of a list from 0 up to i, and a suffix or a second part of the list from i plus 1 up to the end of the overall list-- given that, then I want to assert that the invariant is that the prefix is sorted and no element of the prefix is larger than the smallest element of the suffix. Just what I said earlier. It says, at any stage here-- if this is the amount of sort I've done so far-- I can guarantee, I'm going to claim, this will be sorted. And everything here is bigger than that thing there.

How do I prove it? Well the base case is really easy. In the base case, the prefix is empty. I don't have anything, so it's obviously sorted. And everything in the suffix is bigger than anything in the prefix. So I'm fine. And then I just want to say, as long as I write my code so that this step is true, then I'm going to move the smallest element from the suffix-- the second part of the list-- to the end of the prefix. Since the prefix was sorted, this is now sorted. And everything in the suffix is still going to be bigger than everything in the prefix. And as a consequence, by induction, this is going to give me something that says it's always going to be correct.

So here's code that would do that. Here. I'm just going to set a little thing called the start of suffix, or soft start. Initially it's going to point to the beginning of the list. And then I'm going to run a loop. And as long as I still have things to search in the list, that that pointer doesn't point to the end of the list, what am I going to do? I'm going to loop over everything from that point to the end of the list, comparing it to the thing at that point. If it's less than, I'm going to do a swap because I wanted to move it up. And you can see, by the time I get through this loop I will have found the smallest element in the remainder of the list. And I would have put it at that spot, whatever suffix start points to.

And when I've done all of that, I just change this by one. Having found the smallest element, I've stuck it at spot zero. I'll do the same thing. Having found the next smallest element, I know it's at point one. And I'll just continue around. One of the things you can see here is, as opposed to bubble sort, this one is going to take n times around the loop because I'm only moving this pointer by one So it starts at 0, and then 1, and then 2, all the way up to n minus 1.

You can also see in this particular implementation, while I'm certainly ensuring that the smallest element goes into that spot, I may do a few other flips along the way. I'm going to find something I think is the smallest element, put it there and put that element here. And then when I find another smaller element, I may do that flip. I could have implemented this where I literally search for the smallest element and only move that. Doesn't make any difference in terms of the complexity.

All right. What's the complexity here? Already said this part. I will loop n times, because I start at 0 and then 1. You get the idea. Inside of the loop I'm going to walk down the remainder of the list, which is initially n. And then n minus 1, and then n minus 2 times. But we've seen that before as well. While they get shorter, that complexity is still quadratic. Order n times going through this process. Within the process, order n things that I have to compare. And yes, n gets smaller. But we know that that n term, if you like to dominate. So again, this is quadratic.

OK. Before you believe that all sorting algorithms are quadratic, I want to show you the last one, the one that actually is one of the-- I think-- the prettiest algorithms around, and a great example of a more efficient algorithm. It's called merge sort. Merge sort takes an approach we've seen before. We talked about divide and conquer. Break the problem down into smaller versions of the same problem. And once you've got those solutions, bring the answer back together. For merge sort, that's pretty easy. It says, if I've got a list of 0 or 1 elements, it's sorted. Duh.

OK. If I got a list of more than 1 element, here's my trick. I'm going to split it into two lists. I'm going to sort them. And when I'm done, I'm just going to merge those two lists into one list. And the merge is easy. Because if I've got two lists that are sorted, I just need to look at the first element of each, take the one that's smaller. Add it to my result. And keep doing that until one of the lists is empty. And then just copy the remainder of the other list. You can probably already get a sense of what the cost is going to be here, because this is cutting the problem in half.

Now I've got two pieces. So I need to think about both of them. I want to give you a couple of visualizations of this. Here's the first one. It says, basically, I've got a big unsorted list. I'm going to split it. And I'm going to split it. And I'm going to split it. Until I get down to just lists that are either 0 or 1, which by definition are sorted. And once I'm at that level, then I just have to merge them into a sorted list and then merge them pairwise into a sorted list. And you get the idea.

So it's divide and conquer. The divide is dividing it up into smaller pieces. The conquer is merging them back together. And we have Professor Guttag back for an encore, together with his students. So let's show you an example of merge sort.

[VIDEO PLAYBACK]

- So we're about to demonstrate merge sort. And we're going to sort this rather motley collection of MIT students by height. So the first thing we need to do is, we're going to ask everyone to split into a group of two. So you split a little bit. You two are together. You two are together. You two are together. You two are together. And you are all by yourself. I'm sorry.

PROFESSOR: Poor Anna.

- All right. So now let's take the first group. Take a step down. And what we do is, we sort this group by height, with the shortest on the left. And look at this. We don't have to do anything. Thank you. Feel free to go back up. We then sort the next pair. Please. And it looks to me like we need to switch. All right. Take a step back. Ladies-- OK. Ladies, gentlemen-- also OK. And again, OK.

PROFESSOR: Notice each subgroup is now sorted. Which is great.

- And I think you're in the correct order. Now what we do is, we take these groups and merge the groups. So let's have these two-- going to sort these groups, have them step forward. And now what we're doing is, we're doing a merge of the two sorted groups. So we start by merging them. We'll take the leftmost person in this group and compare her to the first person in this group, and decide. She's still the shortest. Take a step back. Now we're going to look at you and say, you're actually taller than this fellow. So you now step up there. And we're good here. Both of you take a step back.

Now we'll take these two groups and follow the same procedure. We'll merge them. Let's see. We'll compare you-- the first person in this group to the first person in this group. Now it's a little tricky. So let's see, the two of you compare. Let's see, back to back. We have a winner. Step back. And now we need to compare the shortest person in this group to the shortest person in this group. We have a winner. It's you. I'm sorry. And now we just-- we're OK. Please step back.

Now we'll have these two groups come forward. We'll compare the shortest person in this group to the shortest person in that group. I actually need you guys to get back to back here. You are the winner. And it's pretty clear that the shortest person in this group is shorter than the shortest person in that group. So you go there and you step back.

PROFESSOR: Notice the groups. Now all sorted.

- And now we repeat the same process.

 

PROFESSOR: And notice how the whole subgroup now goes up once we know that one group is empty.

 

- And you can see that we have a group of students sorted in order by height.

[END PLAYBACK]

[APPLAUSE]

PROFESSOR: Remember the first number, right? 55, 28. Now it's just numbers but you can see the expectation is, this is going to take less time. And it certainly did there. So again just to demo another way visually. I'm sorting-- sorry. I am splitting down until I get small things, and then just merging them up. I may have to do multiple passes through here, but it's going to be hopefully faster than the other methods we looked at.

I'm going to show you code in a second, and then we're going to run it just to see it. But let me stress one more time just the idea of merging. You can see the idea. I keep splitting down till I got something small enough. And I want to merge them back. The idea of merging-- you've seen it from Professor Guttag. But I just want to highlight why this is going to be efficient. If I've got two lists: list 1 and list 2, the things left there. Process is very simple. I pull out the smallest element of each. I compare them. And I simply put the smallest one into the result, move on in that first list. So the 1 disappears from that left list. And now again I pull up just the smallest element of each one, do the comparison. Smallest one goes to the end of my result. And I drop that element from its list. So I've now taken 1 from list 1 and one from list 2. You get the idea.

The reason I want to give you this visualization-- sorry. Let me do the last step. Once I get to a place where one of the lists is empty, just copy the rest of the list onto the end. You can see already a hint of the code. And that is, that I'm only going to ever look at each element of each sublist once as I do the merge. And that's a nice property. Having had them sorted, I don't need to do lots of interior comparisons. I'm only comparing the ends of the list. I only, therefore, look at each element-- the number of comparisons, rather, I should say. I may look at each element more than once. The number of comparisons is going to be, at most, the number of elements in both lists. And that's going to be a nice Q as we think about how to solve it.

So here's the code to merge, and then we'll write Merge Sort. And I know there's a lot of code here, but we can walk through it and get a good sense of it. I'm going to set up a variable called Result that's going to hold my answer. And I'm going to set up two indices, i and j, that are initially 0. They're pointing to the beginning. And remember, the input here is two lists that we know are sorted-- or should be sorted, or we screwed up in some way. So initially, i and j are both pointing to the beginning of the left and right list. And look at what we do. We say, as long as there's still something in the left list and still something in the right list-- i is less than the length of left, j is less than the length of right.

Do the comparison. If the left wants smaller, add it to the end of result. To the end of result, right? I'm appending it because I want it to be in that sorted order. And increase i. If it's not, add the right one to the end of result and increase j. And I'll just keep doing that until I exhaust one of the lists. And when I do I can basically say, if the right list is empty, I know if I get out of here they can't both be true. In other words, if there's still something in the left list, just put it on the end. Otherwise if the only things left are in the right list, just put them on the end. So I'm just walking down the list, doing the comparison, adding the smallest element to my result. And when I'm done, I just return result.

Complexity we can already begin to see here, right? This says the left and right sublists are ordered, so I'm just moving the indices depending on which one holds the smaller element. And when I get done, I'm just returning the rest of the list. So what's the complexity here? I'm going to do this a little more informally. You could actually do that kind of relationship I did last time. But what am I doing? I'm going through the two lists, but only one time through each of those two lists. I'm only comparing the smallest elements. So as I already said, this says that the number of elements I copy will be everything in the left list and everything in the right list. So that order is just the length of left plus the length of right.

And how many comparisons do I do? The most I have to do is however many are in the longer list. Right? That's the maximum number I need to have. Oh, that's nice. That says, if the lists are of order n-- I'm doing order n copies, because order n plus order n is just 2n, which is order n-- then I'm doing order n comparisons. So it's linear in the length of the lists.

OK. Sounds good. That just does the merge. How do I do merge sort? Well we said it. Break the problem in half. Keep doing it until I get sorted lists. And then grow them back up. So there's merge sort. It says, if the list is either empty or of length 1, just return a copy of the list. It's sorted. Otherwise find the middle point-- there's that integer division-- and split. Split the list everything up to the middle point and do merge sort on that. Split everything in the list from the middle point on. Do merge sort on that. And when I get back those two sorted lists, just merge them.

Again, I hope you can see what the order of growth should be here. Cutting the problem down in half at each step. So the number of times I should have to go through this should be to log n the size of the original list. And you can see why we call it divide and conquer. I'm dividing it down into small pieces until I have a simple solution and then I'm growing that solution back up. So there is the base case, there's the divide, and there's the nice conquer [INAUDIBLE] piece of this.

OK. I'm going to show you an example of that. But let's actually look at some code-- sorry about that. Let's look at some code to do this. And in fact I meant to do this earlier and didn't. I also have a version of bubble sort here. Sorry-- selection sort. I've already done bubble sort. There is selection sort. Let's uncomment this. And let's run both of those and just see the comparison between them. Yeah, sorry-- just make that a little easier to read. There we go.

So we saw a bubble sort. It only went through four times, so less than n times. There's selection sort. And as I said to you, it has to do n passes it because it can only ever guarantee that it gets one element at the beginning. So you can in fact see, in this case, from the first or after the initial input until the end of the first step, it looks like it didn't do anything because it determined eventually that one was in the right spot. And similarly I think there's another one right there where it doesn't do any-- or appears not to do anything. All it's guaranteeing is that the next smallest element is in the right spot. As we get through to the end of it, it in fact ends up in the right place.

And then let's look at merge sort and do one more visualization of this. Again let me remove that. If we run it-- again, I've just put some print statements in there. Here you can see a nice behavior. I start off calling Merge Sort with that, which splits down into doing Merge Sort of this portion. Eventually it's going to come back down there and do the second one. It keeps doing it until it gets down to simple lists that it knows are sorted. And then it merges it. Does the smaller pieces and then merges it. And having now 2 merged things, it can do the next level of merge. So you can see that it gets this nice reduction of problems until it gets down to the smallest size.

So let's just look at one more visualization of that and then get the complexity. So if I start out with this list-- sorry about that. What I need to do is split it. Take the first one, split it. Keep doing that until I get down to a base case where I know what those are and I simply merge them. Pass it back up. Take the second piece. Split it until I get down to base cases. Do the merge, which is nice and linear. Pass that back up. Having done those two pieces, I do one more merge. And I do the same thing.

I want you to see this, because again you can notice how many levels in this tree log. Log in the size. Because at each stage here, I went from a problem of 8 to two problems of 4. Each of those went to two problems of 2, and each of those went to two problems of size 1.

All right. So the last piece is, what's the complexity? Here's a simple way to think about it. At the top level, I start off with n elements. I've got two sorted lists of size n over 2. And to merge them together, I need to do order n work. Because as I said I got to do at least n comparisons where n is the length of the list. And then I've got to do n plus n copies, which is just order n. So I'm doing order n work.

At the second level, it gets a little more complicated. Now I've got problems of size n over 4. But how many of them do I have? 4. Oh, that's nice. Because what do I know about this? I know that I have to copy each element at least once. So not at least once. I will copy each element exactly once. And I'll do comparisons that are equal to the length of the longer list. So I've got four sublists of length n over 4 that says n elements. That's nice. Order n. At each step, the subproblems get smaller but I have more of them. But the total size of the problem is n. So the cost at each step is order n. How many times do I do it? Log n. So this is log n iterations with order n work at each step. And this is a wonderful example of a log linear algorithm. It's n log n, where n is the length of the list.

So what you end up with, then, is-- all right, a joke version, some reasonable ways of doing sort that are quick and easy to implement but are quadratic, and then an elegant way of doing the search that's n log n. And I'll remind you I started by saying, as long as I can make the cost of sorting small enough I can amortize that cost. And if you go back and look at last lecture's notes, you'll see n log n grows pretty slowly. And it's actually a nice thing to do. It makes it reasonable to do the sort. And then I can do the search in order n time.

And here's the last punchline. It's the fastest we can do. I'm going to look at John again. I don't think anybody has found a faster sort algorithm. Right? This is the best one can do. Unless you do-- sorry, the best worst case. I'm sorry. John is absolute right. There are better average cases. Again, our concern is worst case. So this is as good as we're going to do in terms of a worst case algorithm. So there you now have sorting algorithms and searching algorithms, and you've now seen-- excuse me, sorry-- constant, log, linear, log linear, quadratic, and exponential algorithms. I'll remind you, we want things as high up in that hierarchy as possible.

All right. I have six minutes left. Some of you are going to leave us. We're going to miss you, but that's OK. I'm sure we'll see later on. For those of you hanging around, this isn't a bad time just to step back and say, so what have we seen? And I want to do this just very quickly. I'm sorry. And I'll remind you, we started by in some sense giving you a little bit of a contract of things we were going to show you. And I would simply suggest to you, what have we done? We've given you a sense of how to represent knowledge with data structures, tuples, lists, dictionaries, more complicated structures. We've shown you some good computational metaphors, iteration, and loops. Recursion has a great way of breaking problems down into simpler versions of the same problem. And there really are metaphors. There are ways of thinking about problems.

We've given you abstraction, the idea of capture a computation, bury it in a procedure. You now have a contract. You don't need to know what happens inside the procedure as long as it delivers the answer it says it would. Or another way of saying it, you can delegate it to somebody and trust that you're going to get what you like out of it. We've seen classes and methods as a wonderful way to modularize systems, to capture combinations of data and things that operate on them in a nice, elegant way. And we just spent a week and a half talking about classes of algorithms and their complexity.

If you step up a level, what we hope you've gotten out of this are a couple of things. You've begun to learn computational modes of thinking. How do I tackle a problem and divide and conquer? How do I think about recursion as a tool in dealing with something? You've begun to-- begun, I will use that word deliberately-- to master the art of computational problem solving. How can you take a problem and turn it into an algorithm? And especially, you've begun to have the ability to make the computer do what you want it to. To say, if I've got a problem from biology or chemistry or math or physics or chemical engineering or mechanical engineering, how do I take that problem and say, here's how I would design an algorithm to give me a simulation and a way of evaluating what it does.

And so what we hope we've done is, we've started you down the path to being able to think and act like a computer scientist. All right. Don't panic. That doesn't mean you stare at people's shoes when you talk to them. Not all computer scientists do that, just faculty. Sorry, John. So what do computer scientists do? And this is actually meant to be serious. And I put up two of my famous historical figures of computer scientists. They do think computationally. They think about abstractions, about algorithms, about automated execution. So the three A's of computational thinking. And in the same way that traditionally you had the three R's of reading, writing, and arithmetic, computational thinking we hope is becoming a fundamental that every well-educated person is going to need.

And that says, you think about the right abstraction. When you have a problem in your [INAUDIBLE] what's the right abstraction? How do I pull apart the pieces? How do I think about that in terms of decomposing things into a relationship that I can use to solve problems? How do I automate? How do I mechanize that abstraction? How do I use what I know happens inside of the machine to write a sequence of steps in a language I'm using to capture that process? And then finally, how do I turn that into an algorithm? And that not only means I need a language for describing those automated processes, and if you like allowing the abstraction of details, but frankly also a way to communicate. If you have to think crisply about how do I describe an algorithm, it's actually giving you a way to crystallize or clarify your thinking about a problem. This is not to say you should talk to your friends in Python. I don't recommend it. But it does say you should use that thinking as a way of capturing your ideas of what you're going to do.

And that leads, then, to this idea of, how difficult is a problem? How best can I solve it? We've shown you these complexity classes and we've hinted at the idea that in fact some problems are inherently more difficult than others. That's something I hope you come back to as you go along. And especially we want you to start thinking recursively. We want you to think about how do I take a hard problem, break it up into simpler versions of the same problem, and then construct the solution. And that shows up lots of places. Right? Recursion is in all sorts of wonderful places. So just to give you an example, I could say to you recursively, "This lecture will end when I'm done talking about this lecture, which will end when I'm done talking about this lecture, which will end when I'm done--"

All right. You don't like infinite recursion. Good luck on the exam.