Lecture 5: Linear-time Sorting: Lower Bounds, Counting Sort, Radix Sort

Related Resources

Lecture Notes (PDF)
Assignments
Exams

• Download English-US transcript (PDF)

Today we're going to talk about sorting, which may not come as such a big surprise. We talked about sorting for a while, but we're going to talk about it at a somewhat higher level and question some of the assumptions that we've been making so far. And we're going to ask the question how fast can we sort? A pretty natural question.

You may think you know the answer.

Perhaps you do. Any suggestions on what the answer to this question might be?

There are several possible answers.

Many of them are partially correct.

Let's hear any kinds of answers you'd like and start waking up this fresh morning. Sorry?

Theta n log n. That's a good answer.

That's often correct. Any other suggestions?

N squared. That's correct if all you're allowed to do is swap adjacent elements.

Good. That was close.

I will see if I can make every answer correct.

Usually n squared is not the right answer, but in some models it is. Yeah?

Theta n is also sometimes the right answer.

The real answer is "it depends".

That's the point of today's lecture.

It depends on what we call the computational model, what you're allowed to do. And, in particular here, with sorting, what we care about is the order of the elements, how are you allowed to manipulate the elements, what are you allowed to do with them and find out their order. The model is what you can do with the elements.

Now, we've seen several sorting algorithms.

Do you want to shout some out? I think we've seen four, but maybe you know even more algorithms.

Quicksort. Keep going.

Heapsort. Merge sort.

You can remember all the way back to Lecture 1.

Any others? Insertion sort.

All right. You're on top of it today.

I don't know exactly why, but these two are single words and these two are two words. That's the style.

What is the running time of quicksort?

This is a bit tricky. N log n in the average case.

Or, if we randomize quicksort, randomized quicksort runs in n log n expected for any input sequence.

Let's say n lg n randomized. That's theta.

And the worst-case with plain old quicksort where you just pick the first element as the partition element.

That's n^2. Heapsort, what's the running time there? n lg n always.

Merge sort, I hope you can remember that as well, n lg n. And insertion sort?

n^2. All of these algorithms run no faster than n lg n, so we might ask, can we do better than n lg n?

And that is a question, in some sense, we will answer both yes and no to today.

But all of these algorithms have something in common in terms of the model of what you're allowed to do with the elements. Any guesses on what that model might be? Yeah?

You compare pairs of elements, exactly.

That is indeed the model used by all four of these algorithms.

And in that model n lg n is the best you can do.

We have so far just looked at what are called comparison sorting algorithms or "comparison sorts".

And this is a model for the sorting problem of what you're allowed to do. Here all you can do is use comparisons meaning less than, greater than, less than or equal to, greater than or equal to, equals to determine the relative order of elements.

This is a restriction on algorithms.

It is, in some sense, stating what kinds of elements we're dealing with. They are elements that we can This is a three digit number. somehow compare. They have a total order, some are less, some are bigger.

But is also restricts the algorithm.

You could say, well, I'm sorting integers, but still I'm only allowed to do comparisons with them.

I'm not allowed to multiply the integers or do other weird things. That's the comparison sorting model. And this lecture, in some sense, follows the standard mathematical progression where you have a theorem, Then we get some 4s, then you have a proof, then you have a counter example. It's always a good way to have a math lecture. We're going to prove the theorem that no comparison sorting algorithm runs better than n lg n. Comparisons.

State the theorem, prove that, and then we'll give a counter example in the sense that if you go outside the comparison sorting model you can do better, you can get linear time in some cases, better than n lg n.

So, that is what we're doing today.

But first we're going to stick to this comparison model and try to understand why we need n lg n comparisons if that's all we're allowed to do. And for that we're going to look at something called decision trees, which in some sense is another model of what you're allowed to do in an algorithm, but it's more general than the comparison model. And let's try and example to get some intuition. Suppose we want to sort three elements. This is not very challenging, but we'll get to draw the decision tree that corresponds to sorting three elements. Here is one solution I claim.

This is, in a certain sense, an algorithm, but it's drawn as a tree instead of pseudocode.

What this tree means is that each node you're making a comparison. This says compare a_1 versus a_2. If a_1 is smaller than a_2 you go this way, if it is bigger than a_2 you go this way, and then you proceed. When you get down to a leaf, this is the answer. Remember, the sorting problem is you're trying to find a permutation of the inputs that puts it in sorted order. Let's try it with some sequence of numbers, say 9, 4 and 6.

We want to sort 9, 4 and 6, so first we compare the first element with the second element.

9 is bigger than 4 so we go down this way.

Then we compare the first element with the third element, that's 9 versus 6. 9 is bigger than 6, so we go this way. And then we compare the second element with the third element, 4 is less than 6 and, so we go this way. And the claim is that this is the correct permutation of the elements.

You take a_2, which is 4, then you take a_3, which is 6, and then you take a_1, which is 9, so indeed that works out. And if I wrote this down right, this is a sorting algorithm in the decision tree model.

In general, let me just say the rules of this game.

In general, we have n elements we want to sort.

And I only drew the n = 3 case because these trees get very big very quickly. Each internal node, so every non-leaf node, has a label of the form i : j where i and j are between 1 and n.

And this means that we compare a_i with a_j.

And we have two subtrees from every such node.

We have the left subtree which tells you what the algorithm does, what subsequent comparisons it makes if it comes out less than.

And we have to be a little bit careful because it could also come out equal. What we will do is the left subtree corresponds to less than or equal to and the right subtree corresponds to strictly greater than.

That is a little bit more precise than what we were doing here. Here all the elements were distinct so no problem. But, in general, we care about the equality case too to be general.

So, that was the internal nodes.

And then each leaf node gives you a permutation.

So, in order to be the answer to that sorting problem, that permutation better have the property that it orders the elements. This is from the first lecture when we defined the sorting problem.

Some permutation on n things such that a_pi(1) is less than or equal to a_pi(2) and so on.

So, that is the definition of a decision tree.

Any binary tree with these kinds of labels satisfies all these properties. That is, in some sense, a sorting algorithm. It's a sorting algorithm in the decision tree model. Now, as you might expect, this is really not too different than the comparison model. If I give you a comparison sorting algorithm, we have these four, quicksort, heapsort, merge sort and insertion sort.

All of them can be translated into the decision tree model.

It's sort of a graphical representation of what the algorithm does. It's not a terribly useful one for writing down an algorithm. Any guesses why?

Why do we not draw these pictures as a definition of quicksort or a definition of merge sort?

It depends on the size of the input, that's a good point.

This tree is specific to the value of n, so it is, in some sense, not as generic.

Now, we could try to write down a construction for an arbitrary value of n of one of these decision trees and that would give us sort of a real algorithm that works for any input size.

But even then this is not a terribly convenient representation for writing down an algorithm.

Well, let's write down a transformation that converts a comparison sorting algorithm to a decision tree and then maybe you will see why. This is not a useless model, obviously, I wouldn't be telling you otherwise.

It will be very powerful for proving that we cannot do better than n lg n, but as writing down an algorithm, if you were going to implement something, this tree is not so useful. Even if you had a decision tree computer, whatever that is. But let's prove this theorem that decision trees, in some sense, model comparison sorting algorithms, which we call just comparison sorts.

This is a transformation. And we're going to build one tree for each value of n. The decision trees depend on n.

The algorithm hopefully, well, it depends on n, but it works for all values of n.

And we're just going to think of the algorithm as splitting into two forks, the left subtree and the right subtree whenever it makes a comparison.

If we take a comparison sort like merge sort.

And it does lots of stuff. It does index arithmetic, it does recursion, whatever.

But at some point it makes a comparison and then we say, OK, there are two halves of the algorithm.

There is what the algorithm would do if the comparison came out less than or equal to and what the algorithm would do if the comparison came out greater than.

So, you can build a tree in this way.

In some sense, what this tree is doing is listing all possible executions of this algorithm considering what would happen for all possible values of those comparisons.

We will call these all possible instruction traces.

If you write down all the instructions that are executed by this algorithm, for all possible input arrays, a_1 to a_n, see what all the comparisons, how they could come and what the algorithm does, in the end you will get a tree.

Now, how big will that tree be roughly?

As a function of n. Yeah?

Right. If it's got to be able to sort every possible list of length n, at the leaves I have to have all the permutations of those elements.

That is a lot. There are a lot of permeations on n elements. There's n factorial of them.

N factorial is exponential, it's really big.

So, this tree is huge. It's going to be exponential on the input size n. That is why we don't write algorithms down normally as a decision tree, even though in some cases maybe we could.

It's not a very compact representation.

These algorithms, you write them down in pseudocode, they have constant length.

It's a very succinct representation of this algorithm. Here the length depends on n and it depends exponentially on n, which is not useful if you wanted to implement the algorithm because writing down the algorithm would take a long time.

But, nonetheless, we can use this as a tool to analyze these comparison sorting algorithms.

We have all of these. Any algorithm can be transformed in this way into a decision tree.

And now we have this observation that the number of leaves in this decision tree has to be really big.

Let me talk about leaves in a second.

Before we get to leaves, let's talk about the depth of the tree.

This decision tree represents all possible executions of the algorithm. If I look at a particular execution, which corresponds to some root to leaf path in the tree, the running time or the number of comparisons made by that execution is just the length of the path.

And, therefore, the worst-case running time, over all possible inputs of length n, is going to be -- n - 1? Could be.

Depends on the decision tree. But, as a function of the decision tree? The longest path, right, which is called the height of the tree.

So, this is what we want to measure.

We want to claim that the height of the tree has to be at least n lg n with an omega in front.

That is what we'll prove.

And the only thing we're going to use is that the number of leaves in that tree has to be big, has to be n factorial.

This is a lower bound on decision tree sorting.

And the lower bound says that if you have any decision tree that sorts n elements then its height has to be at least n lg n up to constant factors.

So, that is the theorem. Now we're going to prove the theorem. And we're going to use that the number of leaves in that tree must be at least n factorial.

Because there are n factorial permutations of the inputs.

All of them could happen. And so, for this algorithm to be correct, it has detect every one of those permutations in some way. Now, it may do it very quickly.

We better only need n lg n comparisons because we know that's possible. The depth of the tree may not be too big, but it has to have a huge number of leaves down there. It has to branch enough to get n factorial leaves because it has to give the right answer in possible inputs. This is, in some sense, counting the number of possible inputs that we have to distinguish. This is the number of leaves.

What we care about is the height of the tree.

Let's call the height of the tree h.

Now, if I have a tree of height h, how many leaves could it have? What's the maximum number of leaves it could have?

2^h, exactly. Because this is binary tree, comparison trees always have a branching factor of 2, the number of leaves has to be at most 2^h, if I have a height h tree. Now, this gives me a relation.

The number of leaves has to be greater than or equal to n factorial and the number of leaves has to be less than or equal to 2^h. Therefore, n factorial is less than or equal to 2^h, if I got that right.

Now, again, we care about h in terms of n factorial, so we solve this by taking logs.

And I am also going to flip sides.

Now h is at least log base 2, because there is a 2 over here, of n factorial. There is a property that I'm using here in order to derive this inequality from this inequality. This is a technical aside, but it's important that you realize there is a technical issue here.

The general principle I'm applying is I have some inequality, I do the same thing to both sides, and hopefully that inequality should still be true.

But, in order for that to be the case, I need a property about that operation that I'm performing.

It has to be a monotonic transformation.

Here what I'm using is that log is a monotonically increasing function. That is important.

If I multiply both sides by -1, which is a decreasing function, the inequality would have to get flipped.

The fact that the inequality is not flipping here, I need to know that log is monotonically increasing.

If you see log that's true. We need to be careful here.

Now we need some approximation of n factorial in order to figure out what its log is. Does anyone know a good approximation for n factorial? Not necessarily the equation but the name. Stirling's formula.

Good. You all remember Stirling.

And I just need the highest order term, which I believe is that. N factorial is at least (n/e)^n. So, that's all we need here.

Now I can use properties of logs to bring the n outside.

This is n lg (n/e).

And then lg (n/e) I can simplify.

That is just lg n - lg e. So, this is n(lg n - lg e).

Lg e is a constant, so it's really tiny compared to this lg n which is growing within.

This is Omega(n lg n). All we care about is the leading term. It is actually Theta(n lg n), but because we have it greater than or equal to all we care about is the omega. A theta here wouldn't give us anything stronger. Of course, not all algorithms have n lg n running time or make n lg n comparisons.

Some of them do, some of them are worse, but this proves that all of them require a height of at least n lg n. There you see proof, once you observe the fact about the number of leaves, and if you remember Stirling's formula.

So, you should know this proof. You can show that all sorts of problems require n lg n time with this kind of technique, provided you're in some kind of a decision tree model.

That's important. We really need that our algorithm can be phrased as a decision tree.

And, in particular, we know from this transformation that all comparison sorts can be represented as the decision tree.

But there are some sorting algorithms which cannot be represented as a decision tree. And we will turn to that momentarily. But before we get there I phrased this theorem as a lower bound on decision tree sorting.

But, of course, we also get a lower bound on comparison sorting. And, in particular, it tells us that merge sort and heapsort are asymptotically optimal. Their dependence on n, in terms of asymptotic notation, so ignoring constant factors, these algorithms are optimal in terms of growth of n, but this is only in the comparison model.

So, among comparison sorting algorithms, which these are, they are asymptotically optimal.

They use the minimum number of comparisons up to constant factors. In fact, their whole running time is dominated by the number of comparisons.

It's all Theta(n lg n). So, this is good news.

And I should probably mention a little bit about what happens with randomized algorithms. What I've described here really only applies, in some sense, to deterministic algorithms. Does anyone see what would change with randomized algorithms or where I've assumed that I've had a deterministic comparison sort?

This is a bit subtle. And I only noticed it reading the notes this morning, oh, wait.

I will give you a hint. It's over here, the right-hand side of the world.

If I have a deterministic algorithm, what the algorithm does is completely determinate at each step.

As long as I know all the comparisons that it made up to some point, it's determinate what that algorithm will do.

But, if I have a randomized algorithm, it also depends on the outcomes of some coin flips. Any suggestions of what breaks over here? There is more than one tree, exactly. So, we had this assumption that we only have one tree for each n.

In fact, what we get is a probability distribution over trees. For each value of n, if you take all the possible executions of that algorithm, all the instruction traces, well, now, in addition to branching on comparisons, we also branch on whether a coin flip came out heads or tails, or however we're generating random numbers it came out with some value between 1 and n. So, we get a probability distribution over trees. This lower bound still applies, though. Because, no matter what tree we get, I don't really care. I get at least one tree for each n. And this proof applies to every tree. So, no matter what tree you get, if it is a correct tree it has to have height Omega(n lg n). This lower bound applies even for randomized algorithms. You cannot get better than n lg n, because no matter what tree it comes up with, no matter how those coin flips come out, this argument still applies. Every tree that comes out has to be correct, so this is really at least one tree.

And that will now work. We also get the fact that randomized quicksort is asymptotically optimal in expectation.

But, in order to say that randomized quicksort is asymptotically optimal, we need to know that all randomized algorithms require Omega(n lg n) comparisons.

Now we know that so all is well.

That is the comparison model. Any questions before we go on?

Good. The next topic is to burst outside of the comparison model and try to sort in linear time.

It is pretty clear that, as long as you don't have some kind of a parallel algorithm or something really fancy, you cannot sort any better than linear time because you've at least got to look at the data. No matter what you're doing with the data, you've got to look at it, otherwise you're not sorting it correctly.

So, linear time is the best we could hope for.

N lg n is pretty close. How could we sort in linear time? Well, we're going to need some more powerful assumption. And this is the counter example. We're going to have to move outside the comparison model and do something else with our elements. And what we're going to do is assume that they're integers in a particular range, and we will use that to sort in linear time.

We're going to see two algorithms for sorting faster than n lg n. The first one is pretty simple, and we will use it in the second algorithm.

It's called counting sort. The input to counting sort is an array, as usual, but we're going to assume what those array elements look like. Each A[i] is an integer from the range of 1 to k. This is a pretty strong assumption. And the running time is actually going to depend on k. If k is small it is going to be a good algorithm. If k is big it's going to be a really bad algorithm, worse than n lg n.

Our goal is to output some sorted version of this array.

Let's call this sorting of A. It's going to be easier to write down the output directly instead of writing down permutation for this algorithm. And then we have some auxiliary storage. I'm about to write down the pseudocode, which is why I'm declaring all my variables here.

And the auxiliary storage will have length k, which is the range on my input values.

Let's see the algorithm.

This is counting sort.

And it takes a little while to write down but it's pretty straightforward.

First we do some initialization.

Then we do some counting.

Then we do some summing.

And then we actually write the output.

Is that algorithm perfectly clear to everyone?

No one. Good. This should illustrate how obscure pseudocode can be.

And when you're solving your problem sets, you should keep in mind that it's really hard to understand an algorithm just given pseudocode like this.

You need some kind of English description of what's going on because, while you could work through and figure out what this means, it could take half an hour to an hour.

And that's not a good way of expressing yourself.

And so what I will give you now is the English description, but we will refer back to this to understand.

This is sort of our bible of what the algorithm is supposed to do. Let me go over it briefly.

The first step is just some initialization.

The C[i]'s are going to count some things, count occurrences of values. And so first we set them to zero. Then, for every value we see A[j], we're going to increment the counter for that value A[j].

Then the C[i]s will give me the number of elements equal to a particular value i. Then I'm going to take prefix sums, which will make it so that C[i] gives me the number of keys, the number of elements less than or equal to [i] instead of equals. And then, finally, it turns out that's enough to put all the elements in the right place. This I will call distribution.

This is the distribution step. And it's probably the least obvious of all the steps. And let's do an example to make it more obvious what's going on.

Let's take an array A = [4, 1, 3, 4, 3].

And then I want some array C. And let me add some indices here so we can see what the algorithm is really doing.

Here it turns out that all of my numbers are in the range 1 to 4, so k = 4. My array C has four values.

Initially, I set them all to zero.

That's easy. And now I want to count through everything. And let me not cheat here.

I'm in the second step, so to speak.

And I look for each element in order.

I look at the C[i] value. The first element is 4, so I look at C4. That is 0.

I increment it to 1. Then I look at element 1.

That's 0. I increment it to 1.

Then I look at 3 and that's here.

It is also 0. I increment it to 1.

Not so exciting so far. Now I see 4, which I've seen before, how exciting.

I had value 1 in here, I increment it to 2.

Then I see value 3, which also had a value of 1.

I increment that to 2. The result is [1, 0, 2, 2]. That's what my array C looks like at this point in the algorithm.

Now I do a relatively simple transformation of taking prefix sums. I want to know, instead of these individual values, the sum of this prefix, the sum of this prefix, the sum of this prefix and the sum of this prefix. I will call that C prime just so we don't get too lost in all these different versions of C.

This is just 1. And 1 plus 0 is 1.

1 plus 2 is 3. 3 plus 2 is 5.

So, these are sort of the running totals.

There are five elements total, there are three elements less than or equal to 3, there is one element less than or equal to 2, and so on.

Now, the fun part, the distribution.

And this is where we get our array B.

B better have the same size, every element better appear here somewhere and they should come out in sorted order.

Let's just run the algorithm. j is going to start at the end of the array and work its way down to 1, the beginning of the array. And what we do is we pick up the last element of A, A[n].

We look at the counter. We look at the C vector for that value. Here the value is 3, and this is the third column, so that has number 3.

And the claim is that's where it belongs in B.

You take this number 3, you put it in index 3 of the array B. And then you decrement the counter. I'm going to replace 3 here with 2. And the idea is these numbers tell you where those values should go.

Anything of value 1 should go at position 1.

Anything with value 3 should go at position 3 or less.

This is going to be the last place that a 3 should go.

And then anything with value 4 should go at position 5 or less, definitely should go at the end of the array because 4 is the largest value. And this counter will work out perfectly because these counts have left enough space in each section of the array. Effectively, this part is reserved for ones, there are no twos, this part is reserved for threes, and this part is reserved for fours. You can check if that's really what this array means. Let's finish running the algorithm. That was the last element.

I won't cross it off, but we've sort of done that.

Now I look at the next to last element.

That's a 4. Fours go in position 5.

So, I put my 4 here in position 5 and I decrement that counter.

Next I look at another 3. Threes now go in position 2, so that goes there. And then I decrement that counter. I won't actually use that counter anymore, but let's decrement it because that's what the algorithm says. I look at the previous element.

That's a 1. Ones go in position 1, so I put it here and decrement that counter.

And finally I have another 4. And fours go in position 4 now, position 4 is here, and I decrement that counter.

So, that's counting sort. And you'll notice that all the elements appear and they appear in order, so that's the algorithm. Now, what's the running time of counting sort? kn is an upper bound.

It's a little bit better than that.

Actually, quite a bit better. This requires some summing.

Let's go back to the top of the algorithm.

How much time does this step take?

k. How much time does this step take? n.

How much time does this step take?

k. Each of these operations in the for loops is taking constant time, so it is how many iterations of that for loop are there?

And, finally, this step takes n.

So, the total running time of counting sort is k + n.

And this is a great algorithm if k is relatively small, like at most n. If k is big like n^2 or 2^n or whatever, this is not such a good algorithm, but if k = O(n) this is great. And we get our linear time sorting algorithm. Not only do we need the assumption that our numbers are integers, but we need that the range of the integers is pretty small for this algorithm to work. If all the numbers are between 1 and order n then we get a linear time algorithm.

But as soon as they're up to n lg n we're toast.

We're back to n lg n sorting. It's not so great.

So, you could write a combination algorithm that says, well, if k is bigger than n lg n, then I will just use merge sort. And if it's less than n lg n I'll use counting sort. And that would work, but we can do better than that. How's the time?

It is worth noting that we've beaten our bound, but only assuming that we're outside the comparison model.

We haven't really contradicted the original theorem, we're just changing the model. And it's always good to question what you're allowed to do in any problem scenario.

In, say, some practical scenarios, this would be great if the numbers you're dealing with are, say, a byte long. Then k is only 2^8, which is 256. You need this auxiliary array of size 256, and this is really fast.

256 + n, no matter how big n is it's linear in n.

If you know your numbers are small, it's great.

But if you're numbers are bigger, say you still know they're integers but they fit in like 32 bit words, then life is not so easy. Because k is then 2^32, which is 4.2 billion or so, which is pretty big.

And you would need this auxiliary array of 4.2 billion words, which is probably like 16 gigabytes.

So, you just need to initialize that array before you can even get started. Unless n is like much, much more than 4 billion and you have 16 gigabytes of storage just to throw away, which I don't even have any machines with 16 gigabytes of RAM, this is not such a great algorithm. Just to get a feel, it's good, the numbers are really small.

What we're going to do next is come up with a fancier algorithm that uses this as a subroutine on small numbers and combines this algorithm to handle larger numbers.

That algorithm is called radix sort.

But we need one important property of counting sort before we can go there.

And that important property is stability.

A stable sorting algorithm preserves the order of equal elements, let's say the relative order.

This is a bit subtle because usually we think of elements just as numbers. And, yeah, we had a couple threes and we had a couple fours.

It turns out, if you look at the order of those threes and the order of those fours, we kept them in order. Because we took the last three and we put it here. Then we took the next to the last three and we put it to the left of that where O is decrementing our counter and moving from the end of the array to the beginning of the array. No matter how we do that, the orders of those threes are preserved, the orders of the fours are preserved. This may seem like a relatively simple thing, but if you look at the other four sorting algorithms we've seen, not all of them are stable. So, this is an exercise.

Exercise is figure out which other sorting algorithms that we've seen are stable and which are not.

I encourage you to work that out because this is the sort of thing that we ask on quizzes. But for now all we need is that counting sort is stable. And I won't prove this, but it should be pretty obvious from the algorithm.

Now we get to talk about radix sort.

Radix sort is going to work for a much larger range of numbers in linear time. Still it has to have an assumption about how big those numbers are, but it will be a much more lax assumption. Now, to increase suspense even further, I am going to tell you some history about radix sort.

This is one of the oldest sorting algorithms.

It's probably the oldest implemented sorting algorithm.

It was implemented around 1890. This is Herman Hollerith.

Let's say around 1890. Has anyone heard of Hollerith before? A couple people.

Not too many. He is sort of an important guy.

He was a lecturer at MIT at some point.

He developed an early version of punch cards.

Punch card technology. This is before my time so I even have to look at my notes to remember.

Oh, yeah, they're called punch cards.

You may have seen them. If not they're in the PowerPoint lecture notes. There's this big grid.

These days, if you've used a modern punch card recently, they are 80 characters wide and, I don't know, I think it's something like 16, I don't remember exactly.

And then you punch little holes here.

You have this magic machine. It's like a typewriter.

You press a letter and that corresponds to some character.

Maybe it will punch out a hole here, punch out a hole here.

You can see the website if you want to know exactly how this works for historical reasons. You don't see these too often anymore, but this is in particular the reason why most terminals are 80 characters wide because that was how things were. Hollerith actually didn't develop these punch cards exactly, although eventually he did. In the beginning, in 1890, the big deal was the US Census.

If you watched the news, I guess like a year or two ago, the US Census was a big deal because it's really expensive to collect all this data from everyone.

And the Constitution says you've got to collect data about everyone every ten years. And it was getting hard.

In particular, in 1880, they did the census.

And it took them almost ten years to complete the census.

The population kept going up, and ten years to do a ten-year census, that's going to start getting expensive when they overlap with each other. So, for 1890 they wanted to do something fancier. And Hollerith said, OK, I'm going to build a machine that you take in the data. It was a modified punch card where you would mark out particular squares depending on your status, whether you were single or married or whatever.

All the things they wanted to know on the census they would encode in binary onto this card. And then he built a machine that would sort these cards so you could do counting.

And, in some sense, these are numbers.

And the numbers aren't too big, but they're big enough that counting sort wouldn't work. I mean if there were a hundred numbers here, 2^100 is pretty overwhelming, so we cannot use counting sort. The first idea was the wrong idea. I'm going to think of these as numbers. Let's say each of these columns is one number. And so there's sort of the most significant number out here and there is the least significant number out here. The first idea was you sort by the most significant digit first.

That's not such a great algorithm, because if you sort by the most significant digit you get a bunch of buckets each with a pile of cards. And this was a physical device.

It wasn't exactly an electronically controlled computer. It was a human that would push down some kind of reader. It would see which holes in the first column are punched. And then it would open a physical bin in which the person would sort of swipe it and it would just fall into the right bin.

It was a semi-automated. I mean the computer was the human plus the machine, but never mind.

This was the procedure. You sorted it into bins.

Then you had to go through and sort each bin by the second digit. And pretty soon the number of bins gets pretty big. And if you don't have too many digits this is OK, but it's not the right thing to do. The right idea, which is what Hollerith came up with after that, was to sort by the least significant digit first.

And you should also do that using a stable sorting algorithm. Now, Hollerith probably didn't call it a stable sorting algorithm at the time, but we will. And this won Hollerith lots of money and good things. He founded this tabulating machine company in 1911, and that merged with several other companies to form something you may have heard of called IBM in 1924. That may be the context in which you've heard of Hollerith, or if you've done punch cards before. The whole idea is that we're doing a digit by digit sort. I should have mentioned that at the beginning. And we're going to do it from least significant to most significant.

It turns out that works. And to see that let's do an example. I think I'm going to need a whole two boards ideally. First we'll see an example.

Then we'll prove the theorem. The proof is actually pretty darn easy. But, nonetheless, it's rather counterintuitive this works if you haven't seen it before. Certainly, the first time I saw it, it was quite a surprise. The nice thing also about this algorithm is there are no bins. It's all one big bin at all times. Let's take some numbers.

I'm spacing out the digits so we can see them a little bit better. 657, 839, 436, 720 and 355.

I'm assuming here we're using decimal numbers.

Why not? Hopefully this are not yet sorted. We'd like to sort them.

The first thing we do is take the least significant digit, sort by the least significant digit.

And whenever we have equal elements like these two nines, we preserve their relative order.

So, 329 is going to remain above 839.

It doesn't matter here because we're doing the first sort, but in general we're always using a stable sorting algorithm. When we sort by this column, first we get the zero, so that's 720, then we get 5, Then we get 6, Stop me if I make a mistake. Then we get the 7s, and we preserve the order. Here it happens to be the right order, but it may not be at this point.

We haven't even looked at the other digits.

Then we get 9s, there are two 9s, 329 and 839. All right so far?

Good. Now we sort by the middle digit, the next least significant.

And we start out with what looks like the 2s.

There is a 2 up here and a 2 down here.

Of course, we write the first 2 first, 720, then 329.

Then we have the 3s, so we have 436 and 839.

Then we have a bunch of 5s it looks like.

Have I missed anyone so far? No.

Good. We have three 5s, 355, 457 and 657. I like to check that I haven't lost any elements. We have seven here, seven here and seven elements here.

Good. Finally, we sort by the last digit. One thing to notice, by the way, is before we sorted by the last digit -- Currently these numbers don't resemble sorted order at all.

But if you look at everything beyond the digit we haven't yet sorted, so these two digits, that's nice and sorted, 20, 29, 36, 39, 55, 57, 57.

Pretty cool. Let's finish it off.

We stably sort by the first digit.

And the smallest number we get is a 3, so we get 329 and then 436 and 457, then we get a 6, 657, then a 7, and then we have an 8.

And check. I still have seven elements.

Good. I haven't lost anyone.

And, indeed, they're now in sorted order.

And you can start to see why this is working.

When I have equal elements here, I have already sorted the suffix. Let's write down a proof of that. What is nice about this algorithm is we're not partitioning into bins.

We always keep the huge batch of elements in one big pile, but we're just going through it multiple times.

In general, we sort of need to go through it multiple times.

Hopefully not too many times. But let's first argue correctness. To analyze the running time is a little bit tricky here because it depends how you partition into digits. Correctness is easy.

We just induct on the digit position that we're currently sorting, so let's call that t. And we can assume by induction that it's sorted beyond digit t. This is our induction hypothesis. We assume that we're sorted on the low-order t - 1 digits. And then the next thing we do is sort on the t-th digit. We just need to check that things work. And we restore the induction hypothesis for t instead of t - When we sort on the t-th digit there are two cases.

If we look at any two elements, we want to know whether they're put in the right order. If two elements are the same, let's say they have the same t-th digit -- This is the tricky case. If they have the same t-th digit then their order should not be changed.

So, by stability, we know that they remain in the same order because stability is supposed to preserve things that have the same key that we're sorting on.

And then, by the induction hypothesis, we know that that keeps them in sorted order because induction hypothesis says that they used to be sorted.

Adding on this value in the front that's the same in both doesn't change anything so they remain sorted.

And if they have differing t-th digits -- -- then this sorting step will put them in the right order.

Because that's what sorting does.

This is the most significant digit, so you've got to order them by the t-th digit if they differ.

The rest are irrelevant. So, proof here of correctness is very simple once you know the algorithm.

Any questions before we go on? Good.

We're going to use counting sort.

We could use any sorting algorithm we want for individual digits, but the only algorithm that we know that runs in less than n lg n time is counting sort.

So, we better use that one to sort of bootstrap and get an even faster and more general algorithm.

I just erased the running time. Counting sort runs in order k + n time. We need to remember that.

And the range of the numbers is 1 to k or 0 to k - 1.

When we sort by a particular digit, we shouldn't use n lg n algorithm because then this thing will take n lg n for one round and it's going to have multiple rounds.

That's going to be worse than n lg n.

We're going to use counting sort for each round.

We use counting sort for each digit.

And we know the running time of counting sort here is order k + n . But I don't want to assume that my integers are split into digits for me.

That's sort of giving away too much flexibility.

Because if I have some number written in whatever form it is, probably written in binary, I can cluster together some of those bits and call that a digit.

Let's think of our numbers as binary.

Suppose we have n integers. And they're in some range.

And we want to know how big a range they can be.

Let's say, a sort of practical way of thinking, you know, we're in a binary world, each integer is b bits long. So, in other words, the range is from 0 to 2b - 1. I will assume that my numbers are non-negative. It doesn't make much difference if they're negative, too.

I want to know how big a b I can handle, but I don't want to split into bits as my digits because then I would have b digits and I would have to do b rounds of this algorithm.

The number of rounds of this algorithm is the number of digits that I have. And each one costs me, let's hope, for linear time. And, indeed, if I use a single bit, k = 2.

And so this is order n. But then the running time would be order n per round. And there are b digits, if I consider them to be bits, order n times b time.

And even if b is something small like log n, if I have log n bits, then these are numbers between 0 and n - 1. I already know how to sort numbers between 0 and n - 1 in linear time.

Here I'm spending n lg n time, so that's no good.

Instead, what we're going to do is take a bunch of bits and call that a digit, the most bits we can handle with counting sort. The notation will be I split each integer into b/r digits. Each r bits long.

In other words, I think of my number as being in base 2^r. And I happen to be writing it down in binary, but I cluster together r bits and I get a bunch of digits in base 2^r.

And then there are b/ r digits. This b/r is the number of rounds. And this base -- This is the maximum value I have in one of these digits.

It's between 0 and 2^r. This is, in some sense, k for a run of counting sort.

What is the running time? Well, I have b/r rounds.

It's b/r times the running time for a round.

Which I have n numbers and my value of k is 2^r.

This is the running time of counting sort, n + k, this is the number of rounds, so this is b/r (n+2^r).

And I am free to choose r however I want.

What I would like to do is minimize this run time over my choices of r. Any suggestions on how I might find the minimum running time over all choices of r?

Techniques, not necessarily solutions.

We're not used to this because it's asymptomatic, but forget the big O here. How do I minimize a function with respect to one variable? Take the derivative, yeah. I can take the derivative of this function by r, differentiate by r, set the derivative equal to 0, and that should be a critical point in this function. It turns out this function is unimodal in r and you will find the minimum.

We could do that. I'm not going to do it because it takes a little bit more work. You should try it at home.

It will give you the exact minimum, which is good if you know what this constant is. Differentiate with respect to r and set to 0. I am going to do it a little bit more intuitively, in other words less precisely, but I will still get the right answer.

And definitely I will get an upper bound because I can choose r to be whatever I want. It turns out this will be the right answer. Let's just think about growth in terms of r. There are essentially two terms here. I have b/r(n) and I have b/r(2^r). Now, b/r(n) would like r to be as big as possible. The bigger r is the number of rounds goes down. This number in front of n, this coefficient in front of n goes down, so I would like r to be big. So, b/r(n) wants r big.

However, r cannot be too big. This is saying I want digits that have a lot of bits in them. It cannot be too big because there's 2^r term out here. If this happens to be bigger than n then this will dominate in terms of growth of r.

This is going to be b times 2 to the r over r.

2 the r is much, much bigger than r, so it's going to grow much faster is what I mean.

And so I really don't want r to be too big for this other term.

So, that is b/4(2^r) wants r small.

Provided that this term is bigger or equal to this term then I can set r pretty big for that term.

What I want is the n to dominate the 2^r.

Provided I have that then I can set r as large as I want.

Let's say I want to choose r to be maximum subject to this condition that n is greater than or equal to 2^r.

This is an upper bound to 2^r, and upper bound on r.

In other words, I want r = lg n.

This turns out to be the right answer up to constant factors.

There we go. And definitely choosing r to be lg n will give me an upper bound on the best running time I could get because I can choose it to be whatever I want.

If you differentiate you will indeed get the same answer.

This was not quite a formal argument but close, because the big O is all about what grows fastest.

If we plug in r = lg n we get bn/lg n.

The n and the 2^r are equal, that's a factor of 2, 2 times n, not a big deal. It comes out into the O.

We have bn/lg n which is r. We have to think about what this means and translate it in terms of range.

b was the number of bits in our number, which corresponds to the range of the number. I've got 20 minutes under so far in lecture so I can go 20 minutes over, right? No, I'm kidding.

Almost done. Let's say that our numbers, are integers are in the range, we have 0 to 2^b, I'm going to say that it's range 0 to nd.

This should be a -1 here. If I have numbers that are between 0 and n^d - 1 where d is a constant or d is some parameter, so this is a polynomial in n, then you work out this running time.

It is order dn. This is the way to think about it because now we can compare to counting sort.

Counting sort could handle 0 up to some constant times d in linear time. Now I can handle 0 up to n to some constant power in linear time.

This is if d = order 1 then we get a linear time sorting algorithm. And that is cool as long as d is at most lg n. As long as your numbers are at most n lg n then we have something that beats our n lg n sorting algorithms. And this is pretty nice.

Whenever you know that your numbers are order log end bits long we are happy, and you get some smooth tradeoff there. For example, if we have our 32 bit numbers and we split into let's say eight bit chunks then we'll only have to do four rounds each linear time and we have just 256 working space.

We were doing four rounds for 32 bit numbers.

If you use n lg n algorithm, you're going to be doing lg n rounds through your numbers. n is like 2000, and that's at least 11 rounds for example.

You would think this algorithm is going to be much faster for small numbers. Unfortunately, counting sort is not very good on a cache.

In practice, rating sort is not that fast an algorithm unless your numbers are really small.

Something like quicksort can do better.

It's sort of shame, but theoretically this is very beautiful. And there are contexts where this is really the right way to sort things.

I will mention finally that if you have arbitrary integers that are one word length long. Here we're assuming that there are b bits in a word and we have some depends indirectly on b here. But, in general, if you have a bunch of integers and they're one word length long, and you can manipulate a word in constant time, then the best algorithm we know for sorting runs in n times square root of lg lg n time expected.

It is a randomized algorithm. We're not going to cover that algorithm in this class. It's rather complicated.

I didn't even cover it in Advanced Algorithms when I taught it. If you want something easier, you can get n times square root of lg lg n time worst-case.

And that paper is almost readable.

I have taught that in Advanced Algorithms.

If you're interested in this kind of stuff, take Advanced Algorithms next fall.

It's one of the follow-ons to this class.

These are much more complicated algorithms, but it gives you some sense. You can even break out of the dependence on b, as long as you know that b is at most a word. And I will stop there unless there are any questions. Then see you Wednesday.

Free Downloads

Free Streaming

• VideoLectures.net

Video

• iTunes U (MP4 - 161MB)
• Internet Archive (MP4 - 316MB)

Caption

• English-US (SRT)