Lecture 10: Understanding Program Efficiency, Part 1

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MITOpenCourseWare@ocw.mit.edu.

PROFESSOR: OK, folks. Welcome back. Hope you had a nice long weekend with no classes. You got caught up on all those problem sets that have been sneaking up on you. You enjoyed watching the Patriots and Tom Brady come back. Oh, sorry, I'm showing my local bias.

Before we talk about today's topic, I want to take a second to set the stage. And I want you to stop and think about what you've seen so far in this course. We're coming up on the end of the first section of the course. And you've already seen a lot. You've certainly learned about fundamentals of computation. You've seen different kinds of data structures, both mutable and immutable, so tuples and lists, dictionaries, different ways of pulling things together.

You've seen a range of algorithms from simple linear code to loops, fors and whiles. You've seen iterative algorithms. You've seen recursive algorithms. You've seen classes of algorithms. Divide and conquer. Greedy algorithms. Bisection search. A range of things. And then most recently, you start pulling things together with classes-- a way to group together data that belongs together along with methods or procedures that are designed to manipulate that data.

So you've had actually a fairly good coverage already of a lot of the fundamentals of computation. And you're starting to get geared up to be able to tackle a pretty interesting range of problems. Today and Monday, we're going to take a little bit of a different look at computation. Because now that you've got the tools to start building up your own personal armamentarium of tools, we'd like to ask a couple of important questions. The primary one of which is how efficient are my algorithms?

And by efficiency, we'll see it refers both to space and time, but primarily to time. And we'd like to know both how fast are my algorithms going to run and how could I reason about past performance. And that's what we're going to do with today's topics. We're going to talk about orders of growth. We'll define what that means in a few minutes. We're going to talk about what's called the big O notation. And we're going to begin to explore different classes of algorithms.

Before we do that though, let's talk about why. And I want to suggest to you there are two reasons this is important to be considering. First question is how could we reason about an algorithm something you write in order to predict how much time is it going to need to solve a problem of a particular size? I might be testing my code on small scale examples.

And I want to know if I'd run it on a really big one, how long is it going to take? Can I predict that? Can I make guesses about how much time I'm going to need to solve this problem? Especially if it's in a real world circumstance where time is going to be crucial.

Equally important is going the other direction. We want you to begin to reason about the algorithms you write to be able to say how do certain choices in a design of an algorithm influence how much time it's going to take. If I choose to do this recursively, is that going to be different than iteratively? If I choose to do this with a particular kind of structure in my algorithm, what does that say about the amount of time I'm going to need?

And you're going to see there's a nice association between classes of algorithms and the interior structure of them. And in particular, we want to ask some fundamental questions. Are there fundamental limits to how much time it's going to take to solve a particular problem, no matter what kind of algorithm I design around this? And we'll see that there are some nice challenges about that.

So that's what we're going to do over the next two days. Before we do though, let's maybe ask the obvious question-- why should we care? Could be on a quiz, might matter to you. Better choice is because it actually makes a difference. And I say that because it may not be as obvious to you as it was in an earlier generation.

So people with my gray hair or what's left of my gray hair like to tell stories. I'll make it short. But I started programming 41 years ago-- no, sorry, 45 years ago-- on punch cards. You don't know what those are unless you've been to a museum on a machine that filled a half a room and that took about five minutes to execute what you can do in a fraction of a second on your phone. Right. This is to tell you're living in a great time, not independent of what's going to happen on November 8th.

All right. We'll stay away from those topics as well, won't we? My point is yeah, I tell old stories. I'm an old guy. But you might argue look, computers are getting so much faster. Does it really matter? And I want to say to you-- maybe it's obvious to you-- yes, absolutely it does. Because in conjunction with us getting faster computers, we're increasing the sizes of the problems. The data sets we want to analyze are getting massive.

And I'll give you an example. I just pulled this off of Google, of course. In 2014-- I don't have more recent numbers-- Google served-- I think I have that number right-- 30 trillion pages on the web. It's either 30 trillionaire or 30 quadrillion. I can't count that many zeros there. It covered 100 million gigabytes of data. And I suggest to you if you want to find a piece of information on the web, can you write a simple little search algorithm that's going to sequentially go through all the pages and find anything in any reasonable amount of time? Probably not. Right? It's just growing way too fast.

This, by the way, is of course, why Google makes a lot of money off of their map reduced algorithm for searching the web, written by the way, or co-written by an MIT grad and the parent of a current MIT student. So there's a nice hook in there, not that Google pays MIT royalties for that wonderful thing, by the way. All right. Bad jokes aside, searching Google-- ton of time. Searching a genomics data set-- ton of time. The data sets are growing so fast.

You're working for the US government. You want to track terrorists using image surveillance from around the world, growing incredibly rapidly. Pick a problem. The data sets grow so quickly that even if the computers speed up, you still need to think about how to come up with efficient ways to solve those problems. So I want to suggest to you while sometimes simple solutions are great, they are the easy ones to rate-- too write. Sorry.

At times, you need to be more sophisticated. Therefore, we want to reason about how do we measure efficiency and how do we relate algorithm design choices to the cost that's going to be associated with it? OK. Even when we do that, we've got a choice to make. Because we could talk about both efficiency in terms of time or in terms of space, meaning how much storage do I have inside the computer?

And the reason that's relevant is there's actually in many cases a trade-off between those two. And you've actually seen an example, which you may or may not remember. You may recall when we introduced dictionaries, I showed you a variation where you could compute Fibonacci using the dictionary to keep track of intermediate values. And we'll see in next week that it actually tremendously reduces the time complexity.

That's called a trade-off, in the sense that sometimes I can pre-compute portions of the answer, store them away, so that when I've tried to a bigger version of the answer I can just look up those portions. So there's going to be a trade-off here. We're going to focus, for purposes of this lecture and the next one, on time efficiency. How much time is it going to take our algorithms to solve a problem?

OK. What are the challenges in doing that before we look at the actual tools? And in fact, this is going to lead into the tools. The first one is even if I've decided on an algorithm, there are lots of ways to implement that. A while loop and a for loop might have slightly different behavior. I could choose to do it with temporary variables or using direct substitution. There's lots of little choices. So an algorithm could be implemented many different ways. How do I measure the actual efficiency of the algorithm?

Second one is I might, for a given problem, have different choices of algorithm. A recursive solution versus an iterative one. Using divide and conquer versus straightforward search. We're going to see some examples of that. So I've got to somehow separate those pieces out. And in particular, I'd like to separate out the choice of implementation from the choice of algorithm. I want to measure how hard is the algorithm, not can I come up with a slightly more efficient way of coming up with an implementation.

So here are three ways I might do it. And we're going to look at each one of them very briefly. The obvious one is we could be scientists-- time it. Write the code, run a bunch of test case, run a timer, use that to try and come up with a way of estimating efficiency. We'll see some challenges with that.

Slightly more abstractly, we could count operations. We could say here are the set of fundamental operations-- mathematical operations, comparisons, setting values, retrieving values. And simply say how many of those operations do I use in my algorithm as a function of the size of the input? And that could be used to give us a sense of efficiency.

We're going to see both of those are flawed somewhat more in the first case than the second one. And so we're going to abstract that second one to a more abstract notion of something we call an order of growth. And I'll come back to that in a couple of minutes. This is the one we're going to focus on. It's one that computer scientists use. It leads to what we call complexity classes. So order of growth or big O notation is a way of abstractly describing the behavior of an algorithm, and especially the equivalences of different algorithms.

But let's look at those. Timing. Python provides a timer for you. You could import the time module. And then you can call, as you can see right down here. I might have defined a really simple little function-- convert Celsius to Fahrenheit. And in particular, I could invoke the clock method from the time module. And what that does is it gives me a number as the number of some fractions of a second currently there.

Having done that I could call the function. And then I could call the clock again, and take the difference to tell me how much time it took to execute this. It's going to be a tiny amount of time. And then I could certainly print out some statistics. I could do that over a large number of runs-- different sizes of the input-- and come up with a sense of how much time does it take.

Here's the problem with that. Not a bad idea. But again, my goal is to evaluate algorithms. Do different algorithms have different amounts of time associated with them? The good news is is that if I measure running time, it will certainly vary as the algorithm changes. Just what I want to measure. Sorry.

But one of the problems is that it will also vary as a function of the implementation. Right? If I use a loop that's got a couple of more steps inside of it in one algorithm than another, it's going to change the time. And I don't really care about that difference. So I'm confounding or conflating implementation influence on time with algorithm influence on time. Not so good.

Worse, timing will depend on the computer. My Mac here is pretty old. Well, at least for computer versions. It's about five years old. I'm sure some of you have much more recent Macs or other kinds of machines. Your speeds may be different from mine. That's not going to help me in trying to measure this.

And even if I could measure it on small sized problems, it doesn't necessarily predict what happens when I go to a really large sized problems, because of issues like the time it takes to get things out of memory and bring them back in to the computer. So what it says is that timing does vary based on what I'd like to measure, but it varies on a lot of other factors. And it's really not all that valuable.

OK. Got rid of the first one. Let's abstract that. By abstract, I'm going to make the following assumption. I'm going to identify a set of primitive operations. Kind of get to say what they are, but the obvious one is to say what does the machine do for me automatically. That would be things like arithmetic or mathematical operations, multiplication, division, subtraction, comparisons, something equal to another thing, something greater than, something less than, assignments, set a name to a value, and retrieval from memory. I'm going to assume that all of these operations take about the same amount of time inside my machine.

Nice thing here is then it doesn't matter which machine I'm using. I'm measuring how long does the algorithm take by counting how many operations of this type are done inside of the algorithm. And I'm going to use that count to come up with a number of operations executed as a function of the size of the input. And if I'm lucky, that'll give me a sense of what's the efficiency of the algorithm.

So this one's pretty boring. It's got three steps. Right? A multiplication, a division, and an addition-- four, if you count the return. But if I had a little thing here that added up the integers from 0 up to x, I've got a little loop inside here. And I could count operations.

So in this case, it's just, as I said, three operations. Here, I've got one operation. I'm doing an assignment. And then inside here, in essence, there's one operation to set i to a value from that iterator. Initially, it's going to be 0. And then it's going to be 1. And you get the idea.

And here, that's actually two operations. It's nice Python shorthand. But what is that operation? It says take the value of total and the value of i, add them together-- it's one operation-- and then set that value, or rather, set the name total to that new value. So a second operation. So you can see in here, I've got three operations. And what else do I have?

Well, I'm going to go through this loop x times. Right? I do it for i equals 0. And therefore, i equal 1, and so on. So I'm going to run through that loop x times. And if I put that together, I get a nice little expression. 1 plus 3x. Actually, I probably cheated here. I shouldn't say cheated. I probably should have counted the return as one more operation, so that would be 1 plus 3x plus 1, or 3x plus 2 operations.

Why should you care? It's a little closer to what I'd like. Because now I've got an expression that tells me something about how much time is this going to take as I change the size of the problem. If x is equal to 10, it's going to take me 32 operations. If x is equal to 100, 302 operations. If x is equal to 1,000, 3,002 operations. And if I wanted the actual time, I'd just multiply that by whatever that constant amount of time is for each operation. I've got a good estimate of that.

Sounds pretty good. Not quite what we want, but it's close. So if I was counting operations, what could I say about it? First of all, it certainly depends on the algorithm. That's great. Number of operations is going to directly relate to the algorithm I'm trying to measure, which is what I'm after.

Unfortunately, it still depends a little bit on the implementation. Let me show you what I mean by that by backing up for a second. Suppose I were to change this for loop to a while loop. I'll set i equal to 0 outside of the loop. And then while i is less than x plus 1, I'll do the things inside of that. That would actually add one more operation inside the loop, because I both have to set the value of i and I have to test the value of i, as well as doing the other operations down here.

And so rather than getting 3x plus 1, I would get 4x plus 1. Eh. As the government says, what's the difference between three and for when you're talking about really big numbers? Problem is in terms of counting, it does depend. And I want to get rid of that in a second, so it still depends a little bit on the implementation. I remind you, I wanted to measure impact of the algorithm.

But the other good news is the count is independent of which computer I run on. As long as all my computers come with the same set of basic operations, I don't care what the time of my computer is versus yours to do those operations on measuring how much time it would take. And I should say, by the way, one of the reasons I want to do it is last to know is it going to take 37.42 femtoseconds or not, but rather to say if this algorithm has a particular behavior, if I double the size of the input, does that double the amount of time I need? Does that quadruple the amount of time I need? Does it increase it by a factor of 10?

And here, what matters isn't the speed of the computer. It's the number of operations. The last one I'm not going to really worry about. But we'd have to really think about what are the operations we want to count.

I made an assumption that the amount of time it takes to retrieve something from memory is the same as the amount of time it takes to do a numerical computation. That may not be accurate. But this one could probably be dealt with by just agreeing on what are the common operations and then doing the measurement. So this is closer. Excuse me.

And certainly, that count varies for different inputs. And we can use it to come up with a relationship between the inputs and the count. And for the most part, it reflects the algorithm, not the implementation. But it's still got that last piece left there, so I need to get rid of the last piece.

So what can we say here? Timing and counting do evaluate or reflect implementations? I don't want that. Timing also evaluates the machines. What I want to do is just evaluate the algorithm. And especially, I want to understand how does it scale?

I'm going to say what I said a few minutes ago again. If I were to take an algorithm, and I say I know what its complexity is, my question is if I double the size of the input, what does that say to the speed? Because that's going to tell me something about the algorithm. I want to say what happens when I scale it? And in particular, I want to relate that to the size of the input.

So here's what we're going to do. We're going to introduce orders of growth. It's a wonderful tool in computer science. And what we're going to focus on is that idea of counting operations. But we're not going to worry about small variations, whether it's three or four steps inside of the loop. We're going to show that that doesn't matter. And if you think about my statement of does it double in terms of size or speed or not-- or I'm sorry-- time or not, whether it goes from three to six or four to eight, it's still a doubling. So I don't care about those pieces inside.

I'm going to focus on what happens when the size of the problem gets arbitrarily large. I don't care about counting things from 0 up to x when x is 10 or 20. What happens when it's a million? 100 million? What's the asymptotic behavior of this? And I want to relate that time needed against the size of the input, so I can make that comparison I suggested.

OK. So to do that, we've got to do a couple of things. We have to decide what are we going to measure? And then we have to think about how do we count without worrying about implementation details. So we're going to express efficiency in terms of size of input. And usually, this is going to be obvious. If I've got a procedure that takes one argument that's an integer, the size of the integer is the thing I'm going to measure things in. If I double the size of that integer, what happens to the computation?

If I'm computing something over a list, typically the length of the list is going to be the thing I'm going to use to characterize the size of the problem. If I've got-- and we'll see this in a second-- a function that takes more than one argument, I get to decide what's the parameter I want to use. If I'm searching to see is this element in that list, typically, I'm going to worry about what's the size of the list, not what's the size of the element. But we have to specify what is that we're measuring. And we're going to see examples of that in just a second.

OK. So now, we start thinking about that sounds great. Certainly fun computing something numeric. Sum of integers from 0 up to x. It's kind of obvious x is the size of my problem. How many steps does it take? I can count that. But in some cases, the amount of time the code takes is going to depend on the input.

So let's take this little piece of code here. And I do hope by now, even though we flash up code, you're already beginning to recognize what does it do. Not the least of which, by the clever name that we chose. But this is obviously just a little function. It runs through a loop-- sorry, a for loop that takes i for each element in a list L. And it's checking to see is i equal to the element I've provided. And when it is, I'm going to return true. If I get all the way through the loop and I didn't find it, I'm going to return false. It's just saying is e in my input list L?

How many steps is this going to take? Well, we can certainly count the number of steps in the loop. Right? We've got a set i. We've got to compare i and potentially we've got to return. So there's at most three steps inside the loop. But depends on how lucky I'm feeling. Right?

If e happens to be the first element in the list-- it goes through the loop once-- I'm done. Great. I'm not always that lucky. If e is not in the list, then it will go through this entire loop until it gets all the way through the elements of L before saying false. So this-- sort of a best case scenario. This is the worst case scenario.

Again, if I'm assigned and say well, let's run some trials. Let's do a bunch of examples and see how many steps does it go through. And that would be the average case. On average, I'm likely to look at half the elements in the list before I find it. Right? If I'm lucky, it's early on. If I'm not so lucky, it's later on.

Which one do I use? Well, we're going to focus on this one. Because that gives you an upper bound on the amount of time it's going to take. What happens in the worst case scenario? We will find at times it's valuable to look at the average case to give us a rough sense of what's going to happen on average. But usually, when we talk about complexity, we're going to focus on the worst case behavior.

So to say it in a little bit different way, let's go back to my example. Suppose you gave it a list L of some length. Length of L, you can call that len if you like. Then my best case would be the minimum running type. And in this case, it will be for the first element in the list. And notice in that case, the number of steps I take would be independent of the length of L. That's great. It doesn't matter how long the list is. If I'm always going to find the first element, I'm done.

The average case would be the average over the number of steps I take, depending on the length of the list. It's going to grow linearly with the length of the list. It's a good practical measure. But the one I want to focus on will be the worst case. And here, the amount of time as we're going to see in a couple of slides, is linear in the size of the problem. Meaning if I double the length of the list in the worst case, it's going to take me twice as much time to find that it's not there. If I increase the length in the list by a factor of 10, in the worst case, it's going to take me 10 times as much time as it did in the earlier case to find out that the problem's not there. And that linear relationship is what I want to capture.

So I'm going to focus on that. What's the worst case behavior? And we're about ready to start talking about orders of growth, but here then is what orders of growth are going to provide for me. I want to evaluate efficiency, particularly when the input is very large. What happens when I really scale this up? I want to express the growth of the program's runtime as that input grows. Not the exact runtime, but that notion of if I doubled it, how much longer does it take? What's the relationship between increasing the size of the input and the increase in the amount of time it takes to solve it?

We're going to put an upper bound on that growth. And if you haven't seen this in math, it basically says I want to come up with a description that is at least as big as-- sorry-- as big as or bigger than the actual amount of time it's going to take. And I'm going to not worry about being precise. We're going to talk about the order of rather than exact. I don't need to know to the femtosecond how long this is going to take, or to exactly one operation how long this is going to take.

But I want to say things like this is going to grow linearly. I double the size of the input, it doubles the amount of time. Or this is going to grow quadratically. I double the size of the input, it's going to take four times as much time to solve it. Or if I'm really lucky, this is going to have constant growth. No matter how I change the input, it's not going to take any more time.

To do that, we're going to look at the largest factors in the runtime. Which piece of the program takes the most time? And so in orders of growth, we are going to look for as tight as possible an upper bound on the growth as a function of the size of the input in the worst case. Nice long definition. Almost ready to look at some examples. So here's the notation we're going to use. It's called Big O notation. I have to admit-- and John's not here today to remind me the history-- I think it comes because we used Omicron-- God knows why. Sounds like something from Futurama. But we used Omicron as our symbol to define this.

I'm having such good luck with bad jokes today. You're not even wincing when I throw those things out. But that's OK. It's called Big O notation. We're going to use it. We're going to describe the rules of it. Is this the tradition of it? It describes the worst case, because it's often the bottleneck we're after. And as we said, it's going to express the growth of the program relative to the input size.

OK. Let's see how we go from counting operations to getting to orders of growth. Then we're going to define some examples of ordered growth. And we're going to start looking at algorithms. Here's a piece of code you've seen before. Again, hopefully, you recognize or can see fairly quickly what it's doing. Computing factorials the iterative way. Basically, remember n factorial is n times n minus 1 times n minus 2 all the way down to 1. Hopefully, assuming n is a non-negative integer.

Here, we're going to set up an internal variable called answer. And then we're just going to run over a loop. As long as n is bigger than 1, we're going to multiply answer by n, store it back into answer, decrease n by 1. We'll keep doing that until we get out of the loop. And we're going to return answer. We'll start by counting steps. And that's, by the way, just to remind you that in fact, there are two steps here.

So what do I have? I've got one step up there. Set answer to one. I'm setting up n-- sorry, I'm not setting up n. I'm going to test n. And then I'm going to do two steps here, because I got a multiply answer by n and then set it to answer. And now similarly, we've got two steps there because I'm subtracting 1 from n and then setting it to n.

So I've got 2 plus 4 plus the test, which is 5. I've got 1 outside here. I got 1 outside there. And I'm going to go through this loop n times. So I would suggest that if I count the number of steps, it's 1 plus 5n plus 1. Sort of what we did before. 5n plus 2 is the total number of steps that I use here.

But now, I'm interested in what's the worst case behavior? Well, in this case, it is the worst case behavior because it doesn't have decisions anywhere in here. But I just want to know what's the asymptotic complexity? And I'm going to say-- oh, sorry-- that is to say I could do this different ways. I could have done this with two steps like that. That would have made it not just 1 plus 5n plus 1. It would have made it 1 plus 6n plus 1 because I've got an extra step.

I put that up because I want to remind you I don't care about implementation differences. And so I want to know what captures both of those behaviors. And in Big O notation, I say that's order n. Grows linearly. So I'm going to keep doing this to you until you really do wince at me. If I were to double the size of n, whether I use this version or that version, the amount of time the number of steps is basically going to double.

Now you say, wait a minute. 5n plus 2-- if n is 10 that's 52. And if n is 20, that's 102. That's not quite doubling it. And you're right. But remember, we really care about this in the asymptotic case. When n gets really big, those extra little pieces don't matter. And so what we're going to do is we're going to ignore the additive constants and we're going to ignore the multiplicative constants when we talk about orders of growth.

So what does o of n measure? Well, we're just summarizing here. We want to describe how much time is needed to compute or how does the amount of time, rather, needed to computer problem growth as the size of the problem itself grows. So we want an expression that counts that asymptotic behavior. And we're going to focus as a consequence on the term that grows most rapidly.

So here are some examples. And I know if you're following along, you can already see the answers here. But we're going to do this to simply give you a sense of that. If I'm counting operations and I come up with an expression that has n squared plus 2n plus 2 operations, that expression I say is order n squared. The 2 and the 2n don't matter. And think about what happens if you make n really big. n squared is much more dominant than the other terms. We say that's order n squared.

Even this expression we say is order n squared. So in this case, for lower values of n, this term is going to be the big one in terms of number of steps. I have no idea how I wrote such an inefficient algorithm that it took 100,000 steps to do something. But if I had that expression for smaller values of n, this matters a lot. This is a really big number.

But when I'm interested in the growth, then that's the term that dominates. And you see the idea or begin to see the idea here that when I have-- sorry, let me go back there-- when I have expressions, if it's a polynomial expression, it's the highest order term. It's the term that captures the complexity. Both of these are quadratic. This term is order n, because n grows faster than log of n.

This funky looking term, even though that looks like the big number there and it is a big number, that expression we see is order n log n. Because again, if I plot out as how this changes as I make n really large, this term eventually takes over as the dominant term. What about that one? What's the big term there?

How many people think it's n to the 30th? Show of hands. How many people think it's 3 to the n? Show of hands. Thank you. You're following along. You're also paying attention. How many people think I should stop asking questions? No show of hands. All right. But you're right. Exponentials are much worse than powers.

Even something like this-- again, it's going to take a big value of n before it gets there, but it does get there. And that, by the way, is important, because we're going to see later on in the term that there are some problems where it's believed that all of the solutions are exponential. And that's a pain, because it says it's always going to be expensive to compute. So that's how we're going to reason about these things.

And to see it visually, here are the differences between those different classes. Something that's constant-- the amount of time doesn't change as I change the size of the input. Something that linear grows as a straight line, as you would expect. Nice behavior. Quadratic starts to grow more quickly. The log is always better than linear because it slows down as we increase the size. n log n or log linear is a funky term, but we're going to see it's a very common complexity for really valuable algorithms in computer science. And it has a nice behavior, sort of between the linear and the quadratic. And exponential blows up.

Just to remind you of that-- well, sorry-- let me show you how we're going to do the reasoning about this. So here's how we're going to reason about it. We've already seen some code where I started working through this process of counting operations. Here are the tools I want you to use. Given a piece of code, you're going to reason about each chunk of code separately. If you've got sequential pieces of code, then the rules are called the law of addition for order of growth is that the order of growth of the combination is the combination of the order of the growth. Say that quickly 10 times.

But let's look at an example of that. Here are two loops. You've already seen examples of how to reason about those loops. For this one, it's linear in the size of n. I'm going to go through the loop n times doing a constant amount of things each time around. But what I just showed, that's order n. This one-- again, I'm doing just a constant number of things inside the loop-- but notice, that it's n squared. So that's order n squared. n times n.

The combination is I have to do this work and then that work. So I write that as saying that is order of n plus order of n squared. But by this up here, that is the same as saying what's the order of growth of n plus n squared. Oh yeah. We just saw that. Says it's n squared. So addition or the law of addition let's be reasonable about the fact that this will be an order n squared algorithm.

Second one I'm going to use is called the law of multiplication. And this says when I have nested statements or nested loops, I need to reason about those. And in that case, what I want to argue-- or not argue-- state is that the order of growth here is a multiplication. That is, when I have nested things, I figure out what's the order of growth of the inner part, what's the order growth of the outer part, and I'm going to multiply-- bleh, try again-- I'm going to multiply together those orders of growth, get the overall order of growth. If you think about it, it makes sense.

Look at my little example here. It's a trivial little example. But I'm looping for i from 0 up to n. For every value of i, I'm looping for j from 0 up to n. And then I'm printing out A. I'm the Fonz. I'm saying heyyy a lot. Oh, come on. At least throw something, I mean, when it's that bad. Right? Want to make sure you're still awake. OK. You get the idea.

But what I want to show you here is notice the order of growth. That's order n. Right? I'm doing that n times. But I'm doing that for each value of i. The outer piece here loops also n times. For each value of i, I'm doing order n things. So I'm doing order of n times order of n steps. And by that law, that is the same order of n times n or n squared. So this is a quadratic expression.

You're going to see that a lot. Nested loops typically have that kind of behavior. Not always, but typically have that kind of behavior. So what you're going to see is there's a set of complexity classes. And we're about to start filling these in.

Order one is constant. Says amount of time it takes doesn't depend on the size of the problem. These are really rare that you get. They tend to be trivial pieces of code, but they're valuable. Log n reflects logarithmic runtime. You can sort of read the rest of them. These are the kinds of things that we're going to deal with. We are going to see examples here, here, and here. And later on, we're going to come back and see these, which are really nice examples to have.

Just to remind you why these orders of growth matter-- sorry, that's just reminding you what they look like. We've already done that. Here is the difference between constant log, linear, log linear squared, and exponential. When n is equal to 10, 100, 1,000 or a million. I know you know this, but I want to drive home the difference. Something that's constant is wonderful, no matter how big the problem is. Takes the same amount of time.

Something that is log is pretty nice. Increase the size of the problem by 10, it increases by a factor of 2. From another 10, it only increases by a factor of another 50%. It only increases a little bit. That's a gorgeous kind of problem to have. Linear-- not so bad. I go from 10 to 100 to 1,000 to a million.

You can see log linear is not bad either. Right? A factor of 10 increase here is only a factor of 20 increase there. A factor of 10 increase there is only a factor of 30 increase there. So log linear doesn't grow that badly. But look at the difference between n squared and 2 to the n. I actually did think of printing this out. By the way, Python will compute this. But it was taken pages and pages and pages. I didn't want to do it. You get the point. Exponential-- always much worse. Always much worse than a quadratic or a power expression. And you really see the difference here.

All right. The reason I put this up is as you design algorithms, your goal is to be as high up in this listing as you can. The closer you are to the top of this list, the better off you are. If you have a solution that's down here, bring a sleeping bag and some coffee. You're going to be there for a while. Right? You really want to try and avoid that if you can.

So now what we want to do, both for the rest of today in the last 15 minutes and then next week, is start identifying common algorithms and what is their complexity. As I said to you way back at the beginning of this lecture, which I'm sure you remember, it's not just to be able to identify the complexity. I want you to see how choices algorithm design are going to lead to particular kinds of consequences in terms of what this is going to cost you. That's your goal here.

All right. We've already seen some examples. I'm going to do one more here. But simple iterative loop algorithms are typically linear. Here's another version of searching. Imagine I'll have an unsorted list. Arbitrary order. Here's another way of doing the linear search. Looks a little bit faster.

I'm going to set a flag initially to false. And then I'm going to loop for i from 0 up to the length of L. I'm going to use that to index into the list, pull out each element of the list in turn, and check to see is it the thing I'm looking for. As soon as I find it, I'm going to send-- sorry-- set the flag to true. OK? So that when I return out of the loop, I can just return found. And if I found it to be true, if I never found it, found will still be false and I'll return it.

We could count the operations here, but you've already seen examples of doing that. This is linear, because I'm looping n times if n is the length of the list over there. And the number of things I do inside the loop is constant. Now, you might say, wait a minute. This is really brain damaged, or if you're being more politically correct, computationally challenged. OK? In the sense of once I've found it, why bother looking at the rest of the list?

So in fact, I could just return true right here. Does that change the order of growth of this algorithm? No. Changes the average time. I'm going to stop faster. But remember the order of growth captures what's the worst case behavior. And the worst case behavior is the elements not in the list I got to look at everything. So this will be an example of a linear algorithm. And you can see, I'm looping length of L times over the loop inside of there. It's taking the order one to test it. So it's order n.

And if I were to actually count it, there's the expression. It's 1 plus 4n plus 1, which is 4n plus 2, which by my rule says I don't care about the additive constant. I only care about the dominant term. And I don't care about that multiplicative constant. It's order n. An example of a template you're going to see a lot.

Now, order n where n is the length of the list and I need to specify that. That's the thing I'm after. If you think about it, I cheated. Sorry-- I never cheat. I'm tenure. I never cheat. I just mislead you badly. Not a chance. How do I know that accessing an element of the list takes constant time? I made an assumption about that. And this is a reasonable thing to ask about-- both what am I assuming about the constant operations and how do I know that's actually true?

Well, it gives me a chance to point out something that Python does very effectively. Not all languages do. But think about a list. Suppose I've got a list that's all integers. I'm going to need some amount of memory to represent each integer. So if a byte is 8 bits, I might reserve 4 bytes or 32 bits to cover any reasonable sized integer. When I represent a list, I could simply have each of them in turn. So what do I need to know?

I'm going to allocate out a particular length-- say 4 bytes, 32 bits, 32 sequential elements of memory to represent each integer. And then I just need to know where's the first part of the list, what's the address and memory of the first part of the list. And to get to the i-th element, I take that base plus 4 bytes times i. And I can go straight to this point without having to walk down the list. That's nice. OK? It says, in fact, I can get to any element of memory-- I'm sorry-- any element of the list in constant time.

OK. Now, what if the things I'm representing aren't integers? They're arbitrary things and they take up a big chunk of space. Well, if the list is heterogeneous, we use a nice technique called indirection. And that simply says we, again, have a list. We know the address of this point. We know the address there for the i-th element of this list.

But inside of here, we don't store the actual value. We store a pointer to where it is in memory. Just what these things are indicating. So they can be arbitrary size. But again, I can get to any element in constant time, which is exactly what I want. So that's great.

OK. Now, suppose I tell you that the list is sorted. It's in increasing order. I can be more clever about my algorithm. Because now, as I loop through it, I can say if it's the thing I'm looking for, just return true. If the element of the list is bigger than the thing I'm looking for, I'm done. I don't need to look at the rest of the list, because I know it can't be there because it's ordered or sorted. I can just return false. If I get all the way through the loop, I can return false.

So I only have to look until I get to a point where the thing in the list is bigger than what I'm looking for. It's the order of growth here. Again, the average time behavior will be faster. But the order of growth is I've got to do order of length of the list to go through the loop, order of one to do the test, and in the worst case, again, I still have to go through the entire list. So the order of growth here is the same. It is, again, linear in the length of the list, even though the runtime will be different depending whether it's sorted or not.

I want you to hold on to that idea, because we're going to come back to the sorted list next week to see that there actually are much more efficient ways to use the fact that a list is sorted to do the search. But both of these versions same order growth, order n. OK. So lurching through a list-- right, sorry-- searching through a list in sequence is linear because of that loop. There are other things that have a similar flavor. And I'm going to do these quickly to get to the last example.

Imagine I give you a string of characters that are all soon to be composed of decimal digits. I just want to add them all up. This is also linear, because there's the loop. I'm going to loop over the characters in the string. I'm going to cast them into integers, add them in, and return the value. This is linear in the length of the input s. Notice the pattern. That loop-- that in-iterative loop-- it's got that linear behavior, because inside of the loop constant number of things that I'm executing.

We already looked at fact iter. Same idea. There's the loop I'm going to do that n times inside the loop a constant amount of things. So looping around it is order n. There's the actual expression. But again, the pattern I want you to see here is that this is order n. OK.

Last example for today. I know you're all secretly looking at your watches. Standard loops, typically linear. What about nested loops? What about loops that have loops inside of them? How long do they take? I want to show you a couple of examples. And mostly, I want to show you how to reason about them. Suppose I gave you two lists composed of integers, and I want to know is the first list a subset of the second list. Codes in the handbook, by the way, if you want to go run it.

But basically, the simple idea would be I'm going to loop over every element in the first list. And for each one of those, I want to say is it in the second list? So I'll use the same kind of trick. I'll set up a flag that's initially false. And then I'm going to loop over everything in the second list. And if that thing is equal to the thing I'm looking for, I'll set match to true and break out of the loop-- the inner loop. If I get all the way through the second list and I haven't found the thing I'm looking for, when I break out or come out of this loop, matched in that case, will still be false and all return false.

But if up here, I found something that matched, match would be true. I break out of it. It's not false. Therefore, a return true. I want you look at the code. You should be able to look at this and realize what it's doing. For each element in the first list, I walk through the second list to say is that element there. And if it is, I return true. If that's true for all of the elements in the first list, I return true overall.

OK. Order of growth. Outer loop-- this loop I'm going to execute the length of L1 times. Right? I've got to walk down that first list. If I call that n, it's going to take that n times over the outer loop. But what about n here? All of the earlier examples, we had a constant number of operations inside of the loop. Here, we don't. We've got another loop that's looping over in principle all the elements of the second list.

So in each iteration is going to execute the inner loop up to length of L2 times, where inside of this inner loop there is a constant number of operations. Ah, nice. That's the multiplicative law of orders of growth. It says if this is order length L1. And we're going to do that then order length of L2 times, the order of growth is a product. And the most common or the worst case behavior is going to be when the lists are of the same length and none of the elements of L1 are in L2. And in that case, we're going to get something that's order n squared quadratic, where n is the length of the list in terms of number of operations.

I don't really care about subsets. I've got one more example. We could similarly do intersection. If I wanted to say what is the intersection of two lists? What elements are on both list 1 and list 2? Same basic idea. Here, I've got a pair of nested loops. I'm looping over everything in L1. For that, I'm looping over everything in L2. And if they are the same, I'm going to put that into a temporary variable.

Once I've done that, I need to clean things up. So I'm going to write another loop that sets up an internal variable and then runs through everything in the list I accumulated, making sure that it's not already there. And as long as it isn't, I'm going to put it in the result and return it. I did it quickly. You can look through it. You'll see it does the right thing.

What I want it to see is what's the order of growth. I need to look at this piece. Then I need to look at that piece. This piece-- well, it's order length L1 to do the outer loop. For each version of e1, I've got to do order of length L2 things inside to accumulate them. So that's quadratic.

What about the second loop? Well, this one is a little more subtle. I'm only looping over temp, which is at most going to be length L1 long. But I'm checking to see is that element in a list? And it depends on the implementation. But typically, that's going to take up to the length of the list to do it. I got to look to see is it there or not. And so that inner loop if we assume the lists are the same size is also going to take potentially up to length L1 steps. And so this is, again, quadratic. It's actually two quadratics-- one for the first nested loop, one for the second one, because there's an implicit second loop right there. But overall, it's quadratic.

So what you see in general-- this is a really dumb way to compute n squared. When you have nested loops, typically, it's going to be quadratic behavior. And so what we've done then is we've started to build up examples. We've now seen simple looping mechanisms, simple iterative mechanisms, nested loops. They tend to naturally give rise to linear and quadratic complexity. And next time, we're going to start looking at more interesting classes. And we'll see you next time.