Session 5: Concept Selection and Tradespace Exploration

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK, so let's situate ourselves where we are. So we're starting to really make a lot of progress here, moving in the V model. Today's topic is session five: tradespace exploration, concept selection, and PDR-- preliminary design review.

So there's a lot to talk about, and I'm going to go relatively quickly through this. So, first I want to talk about decision analysis. Fundamentally, when you come up for PDR, you have to make a big decision which is, what concept are you going for? What is your system architecture? That's a big decision. You don't have all the details yet. The design is not fully done, but you've chosen your key architecture. That's a big decision, and there's a whole branch of research and science called decision analysis. So I want to tell you about that.

Talk about some of the issues in concept selection, and then give you some tools for doing that in a kind of organized way. There's a couple of-- what I would say, simple methods Pugh matrix and multi-attribute utility-- that are relatively, I think, straightforward. And then there's a bit more advanced concept called non-dominance-- Pareto frontiers, getting into multi-objective optimization.

And so, at a minimum, when you choose your concept, you want to choose a non-dominated concept. And I'll explain what that means. And then we'll close with discussing what is a PDR? What do you do with the PDR? What's the purpose of it?

So, here is another way to explain essentially the flow of things as we come up to PDR. So the way you can think of it as, you start with a need, right? And your requirements. That's-- we've talked about that before. And then you have this creative phase, which we talked about last time, right? You come up with a lot of ideas, a lot of concepts, and the funnel that you see here is going up. So each of these shapes here-- the circle, the triangle-- they're meant to represent fundamentally different concepts, different architecture, different way of satisfying the requirements.

And so there's a lot of different things you can do, and the funnel opens up. This is sort of the size of options or alternatives you're considering. And then you have to filter this down. In the end, you can only build one system, right? You have to choose one architecture. And this filtering down happens typically in two steps. One is what we call concept screening, which is more qualitative. And then the actual concept selection is more quantitative, based on models, based on data, based on metrics. And then you arrive at PDR.

So in this sort of simplified view, we've chosen the square as our concept. And we've made that decision by looking at these different metrics. So you can see that on this graphic here, we have metric one, metric two-- you can see the triangle is dominated, right? The triangle doesn't do as well if we want to maximize metric one and two, so it wouldn't make sense to choose the triangle. But the circle and the square-- circle does better on metric two, but less well on metric one. The square does better on metric one, not quite as well-- and they crossover at some point.

So, whatever-- how you weigh this in decision making, the decision is made. We're going to go for the square. That's our concept. And then you move to the actual system design, post-PDR, which is all the details. So we're expanding the box now, the square, and designing all the details within-- the structure, the avionics, the software, the controls. And that's design modeling and optimization.

And so here I'm showing you metric three, which is maybe another metric, and then design variable x1, which is some pretty detailed decision that you make within that concept, and you're looking for the optimal, in this case, minimizing metric three. It's this point here. And then that gives you your-- and I put this deliberately in quotes-- optimal design. That's what you show at the CDR, and the critical design review, and that's what you go and build.

So that's roughly the flow of decision making. Any question about that? That general flow? So what we'll talk about today is essentially this portion here, concept screening and concept selection. So, since you make a decision, what is decision analysis?

So decision analysis, in general, is methods and tools for ranking and choosing among competing alternatives or courses of action. And I just want to make-- the word alternative. A lot of people use the word option and alternative sort of as synonyms. They're not. OK? They're not. Options and alternatives are not the same thing. An alternative means if I choose A, I can't choose B and C. But the fact that you've chosen A, or the square, or whatever your concept is, means that you cannot choose the other ones.

An option means, well given that I've chosen A, I could also add something on top of A, right? It's kind of like if you were buying a new car, are you going to get the winter package or not? That's an option, but you've chosen a particular vehicle. So, that's important. So alternatives are mutually exclusive choices or courses of action. So you need the alternatives, those are the components-- four components of a decision. You need criteria, which means how are you going to evaluate and compare the alternatives?

Then you need value judgments. You now apply the criteria to the alternatives to compare them, and then finally you need a decision maker, which is either an individual or a group with preferences to then make the choice. Those are the four ingredients of decision analysis. And if we show this somewhat graphically-- so here's our alternatives. We can do A, B, or C. The criteria in this case are cost, performance, and risk. Then we basically evaluate, in this case, option or alternative A, I should say, for cost, for performance, for risk. And this gives us some score or number, and then we need to figure out a way to combine this to rank the alternatives.

And when it comes to ranking, there's a distinction between what's known as an ordinal scale and a cardinal scale. So in an ordinal ranking all you really care about and do is say, what's the best choice, the second best choice, and the third choice. You ordered the alternative. But you don't know in an ordinal ranking or ordinal scale, whether one and two are really close, and then there's a big gap between two and three. You don't know that, and you don't really care. All you want is the rank order.

When you put things on a cardinal scale-- so this is a continuous line-- you actually get, in a sense, both the order but you also see how close the alternatives are. So that's a cardinal scale as opposed to an ordinal scale. OK? So-- yes? Please.

AUDIENCE: How do you get a discrete value for something like risk or something like that? And especially in an ordinal scale, if you have two things that are really close to each other, but the risk is kind of a blurry region, how do you--

PROFESSOR: So then they would be equivalent in terms of that criterion. And you can actually have ties, right? You can have alternatives that are tied. And I'll talk about-- this is one of the issues in concept selection is, how do you do tie breaking? Good question. Any questions at EPFL? I know this is very abstract. But fundamentally, that's what we do. Any questions over there? Is it clear?

AUDIENCE: Yes.

PROFESSOR: OK. Good. So let's keep moving. So what are the issues? I think that's-- so multiple criteria, how do you deal with them? We usually have multiple criteria and how do we-- and eventually, when you make a decision, when you pick a concept, you're somehow combining these criteria into a single measure, whether you're doing that explicitly or not. So how do you deal with these multiple criteria? We'll talk about that.

Here's the point that you brought up. What if there's ties? How do you break them? Group decision making versus individual decision making-- who gets to really make the decision? Right? And this relates to the stakeholder analysis. Who has the power to make the decision in the end?

And then uncertainty-- did we choose the right criteria? Did we evaluate each option properly? And then the big one is are the the best alternatives represented? So you can choose among A, B, and C, but maybe there's this much, much better architecture concept D out there, but it's not in the decision set. Because it didn't-- nobody thought of it, or some other thing happened, and it didn't come up in concept generation. Right? So there's a lot of uncertainties when you're doing this. So you have to be aware of it.

So let me talk about two-- again, quote, end quote-- "simple methods" for doing concept selection. The first one is called Pugh matrix, and then I'll talk about utility analysis. So the Pugh matrix, essentially, is a very discrete method. It uses essentially plus, zero, and minus to score alternatives relative to a datum. A datum is a reference decision, a reference architecture. And this is named after Stuart Pugh, who is a British professor of design, essentially. And it's a very simple method. And it's used extensively. It does have some pitfalls, as well, but it's very well known. So Pugh matrix

And then the second one is called utility analysis. So it essentially maps all the criteria to a dimensionless utility, this is between zero and one. So zero is a completely useless system. It has zero utility. And one is a system that has perfect utility. It can't really be improved. You could try to improve it but your satisfaction would not increase, OK? And this has very deep mathematical theory, particularly the Von Neumann and Morgenstern utility that was basically developed in the mid-twentieth century.

Anybody seen the movie A Beautiful Mind? Yeah? What is is A Beautiful Mind about?

AUDIENCE: It's about John-- mathematician John Nash who's a mathematician who won-- I forget exactly what prize he had won.

PROFESSOR: He won the Nobel Prize.

AUDIENCE: Was it the Nobel Prize? OK.

AUDIENCE: Founded game theory.

PROFESSOR: For what? For game theory, right. And so one of Nash's contributions was the idea of a Nash equilibrium. So we have multiple decision makers in gaming, and the question is-- you have your current strategy, but is there a better move you could make? And if you could make a better move that improves your satisfaction, or your utility, but by doing this, you decrease somebody else's utility, you may not go there. So if you reach a Nash equilibrium, that means that everybody believes that this is the best they can achieve.

And how do you measure that? Well Nash actually developed his Nash equilibrium and game theory building upon theories of Von Neumann and Morgenstern. So there's a very deep theory behind it. I'm not going to go very deep into it, but so you know that there's a lot behind utility analysis.

OK, so let me just talk you through the basic steps for doing a Pugh matrix analysis. And I'll show you a very simple example. First step, you choose or develop the criteria for comparison. So, what's important? How are we going to decide this?

Selection. And obviously this should be based on a set of system requirements and goals. Now, there are two flavors of requirements. Remember, what-- let's ask here at EPFL. You remember the two flavors of requirements? What were the two flavors of requirements?

AUDIENCE: [INAUDIBLE]

PROFESSOR: I mean, there's actually six types, right? There's functional requirements. There is constraints. There's interface requirements, and so forth. What I'm trying to get at is there are requirements that use shall statements and then there's requirements that use should statements, right? So the shall statements are what? What do the shell statements imply?

AUDIENCE: So basically, for the shall, it's compulsory, and for should, it's a goal of the system.

PROFESSOR: That's right. Exactly. So, in some sense, the shall state-- the criteria associated with shall statements don't necessarily make the best criteria for the Pugh matrix, because by definition, the concepts that you're evaluating should all satisfy the shall statements. Because if they don't, they're essentially infeasible, right? And you shouldn't select among them in the first place.

So architectures, or concepts, or alternatives that violate shall requirements do not satisfy the must have requirements. And you shouldn't-- so you shouldn't use criteria that are associated with hard constraints. That's what I'm trying to say.

Now the requirements that are associated with should statements, those are good criteria for this Pugh matrix, because the degree to which you satisfy this, the goals are somewhat variable. Therefore, you can then compare amongst alternatives. OK?

So that's number one. Number two-- thank you, that was good-- select the alternatives to be compared. So that those are coming out of concept generation and one little bit tricky aspect here is when you're putting together the alternatives, the concepts to be selected from, they should be represented at a similar level of detail of abstraction. It is not good to have one concept that's really detailed, and you know a lot about it, and often it's kind of an existing system or something you've done before, and then there's this other concept that is very fuzzy.

Why shouldn't you compare something very detailed with something very, very poorly defined? Why do you think that's not a good practice? What do you guys think here? Yes.

AUDIENCE: Is it like-- it's not fair, because--

PROFESSOR: That's right. It's a fairness issue, and because the concepts that are poorly defined often, you're too optimistic about them. That's sort of the typical-- this looks really good, very promising, but it looks very promising because it's kind of ill-defined. So that's important. Try to represent all the concepts at about the same level of detail when you make the comparison. That's number two.

Number three-- you actually go through and generate the scores. And the key thing here-- this is maybe the most important thing about the Pugh matrix is you always use a datum. The datum is a reference-- one of the alternatives that you're going to compare is your reference. So what you do is you always compare every other alternative in the set against the datum, and you say for each criterion is it better? Is it about the same, or is it worse than the reference, the datum?

So you don't compare all the alternatives pairwise against each other. You always compare against the datum. Then you compute the total scores and I'll show you this in the example, and then there's variations like-- rather than better, equal, or worse, you could go to a five-point. Much better, a little better, or about the same. So there are some variations on the scoring, but the classic method just uses better, the same, or worse.

So here's some--

AUDIENCE: Excuse me, just a question.

PROFESSOR: Yeah, go ahead.

AUDIENCE: Just about the number three, about generic scores and user datum-- by doing so, don't you exclude a solution that could have, under the generic picture, be better then others?

PROFESSOR: What do you mean by general picture?

AUDIENCE: You should take all the solution that you have, and at the end, one performs slightly better than the other, but you couldn't spot it by using the datum?

PROFESSOR: Yeah. So it's a very good question. And so this is a question of resolution right? How good is your resolution? The main purpose of the Pugh matrix is not for making the final choice right away. The main purpose of the Pugh matrix is actually to eliminate concepts that are-- you can tell already that they're not competitive. And you don't typically do Pugh matrix one time and then say that's the best one. You use it, essentially, to eliminate purely inferior solutions. Does that make sense?

AUDIENCE: I'm really sorry but the video cut just during your explanation.

PROFESSOR: OK, so let me repeat this. So the purpose of the Pugh matrix is not to select the final concept as a result of doing the Pugh matrix, but to identify inferior concepts to eliminate them from the set. OK?

AUDIENCE: OK.

PROFESSOR: All right. So here's a kind of generic example to explain how this works. So here's our Pugh matrix, our evaluation matrix. On the rows, we have our criteria. It could be cost, performance, resilience, effectiveness, and so those are called A, B, C, D, E, and F. So we have six criteria.

And then the column-- each column represents a concept, a design, an architecture. And so you know we're designing-- I guess we're designing beams here, or levers, or something like that. So this is a solid cylinder. This is a cylinder that's hollow. This is a square, it's cross-sectioned. This is a triangular cross-section. You get the picture, right? There they're qualitatively different.

And then you note which one is our is our reference here? Which concept? Number seven, right? If you look at seven this sort of shallow triangle. Seven, it says datum. You see that? Which means that this is our baseline, our reference, our datum.

And then what you do is, as you fill this in-- and you can do it two ways. You can do-- pick, say, concept one, the solid cylinder, and then compare it against seven for all criteria. So A, so concept one is better than concept seven in criterion A. So it gets a plus. It's also better in B. It gets a plus.

It's worse in C and D, so this is a minus compared to seven, and so forth. So the two ways of filling it in is you do concept by concept, column-wise, or you can do it row- wise.

And there's some arguments in favor of one or the other. But essentially, you fill in this matrix by always comparing against number seven. Alexis, what's your question?

AUDIENCE: Yeah, just a question. Shouldn't we weight the criteria? I mean, for example, if the criteria A is more important, shall we put more weight on it?

PROFESSOR: No. No. At this point, in the Pugh matrix, the criteria are unweighted. This is very important. You don't say B is twice as important as C. They're unweighted when you first do this. Eventually, when you do the final selection, you will probably weigh them. But when you first do this, there's no weighting. They're equal. OK? Yes? Go ahead.

AUDIENCE: Are there any rules in picking your datum or do you just pick arbitrarily?

PROFESSOR: So I'll get into that. It should be a design that you think is competitive, that [INAUDIBLE] maybe you know it, maybe it's an existing sort of system, there's data on it, a known quantity. But if you don't have if you don't have any information, if they're all equal, then it's a random choice. OK?

So, then once you've filled in this matrix-- so S here is for same, right? About the same. Then you essentially sum all the pluses, you sum all the minuses, and then you have the net result, unweighted. Right? So, here you can essentially compare that, you can see that concept one is better in three and worse in three than the datum. You can see that concept two is inferior in three criteria, better in two, and the same in one. But we're not actually summing. We're not actually summing at this point, because we're not weighting these, OK?

But it gives you a basis of understanding. OK so let's do a quick partner exercise. What do you see as the main advantages and potential disadvantages or pitfalls of this method? So turn to your partner, discuss this for a few minutes, and then we'll see what you came up with.

[MICROPHONE NOISE]

AUDIENCE: [INAUDIBLE]

PROFESSOR: All right. Good. So let's see advantages. Let's get a couple here, and then we'll sort of go back and forth between here and EPFL. So what are-- what's good about this method? Yes, go ahead.

AUDIENCE: So, Nate and I were talking that it is a relatively simple decision making process. So instead of just going with some sort of gut feeling, or not looking at everything, you can get stuff down on paper and actually see these criteria and how they relate to each other.

PROFESSOR: Yeah. So simplicity. What else? Go ahead, Sam.

AUDIENCE: It allows you to look at a large array of concepts very quickly.

PROFESSOR: OK large samples. Sorry. Quick, right? You said quick. . OK. What advantages, EPFL, what did you guys come up with? So simple, it allows us to look at a lot of concepts. It's quick. What else?

AUDIENCE: [INAUDIBLE] we got [INAUDIBLE] the same things. This method is simple, and it's pretty fast and pretty straightforward, too. So this is a good way to get rid of some of the concepts pretty fast.

PROFESSOR: OK. Anything else that we didn't [INAUDIBLE]? There's one more that I would like to put on this list. Veronica, go ahead.

AUDIENCE: [INAUDIBLE]

PROFESSOR: Qualitative.

AUDIENCE: [INAUDIBLE] I kind of like [INAUDIBLE] improvements, rather than this is better because the requirement is--

AUDIENCE: Please put on the microphone.

PROFESSOR: I think-- can you just quickly repeat that?

AUDIENCE: Just that it's qualitative.

PROFESSOR: And that you see that as an advantage, actually.

AUDIENCE: I think you can see it both ways.

PROFESSOR: Yeah. So I think what you implied with is it stimulates the discussion, right? Stimulates debate.

AUDIENCE: There may be another criterion on the positive side.

PROFESSOR: Go ahead.

AUDIENCE: And it would be that it's extremely easy to explain to [INAUDIBLE], to [INAUDIBLE], to whoever your customer is, because-- and they can relate to it. If the criteria are mostly physical or scientific, I mean, larger, stronger, lighter, cheaper, it's probably very intuitive, yes, as I say.

PROFESSOR: OK, great. Now, downsides. What are the downsides of the method? Go ahead. Make sure you--

AUDIENCE: I think it definitely depends on which you pick as a datum. Like, if you pick the clearly best one as the datum, then it's not going to give you much information. Or if you pick the worst one as the datum, then all of them will be better. And you kind of don't really gain much information from that. So you have to like iterate with different datums.

PROFESSOR: Yes. There's actually research on this. There's some research-- this matrix, this Pugh matrix method, by the way, has been studied scientifically quite a bit. There's quite a literature on it. And the point that you brought up was studied. They would basically give the same criteria and the same alternatives to different groups of people, but then give them a different datum. And it's a little tricky-- is the result different because it's a different group dynamic? Or is it-- but this has been done statistically, and the choice of datum has an impact on the outcome. Very good.

That's actually a pretty subtle point. That was excellent. What are some other-- yes, please go ahead.

AUDIENCE: While it's very easy to implement and anyone can do it, it is very subjective. So if you have somebody that doesn't have a whole lot of experience in a certain area, they may get a completely different answer than somebody who is an expert looking at the same type of criteria.

PROFESSOR: Subjectivity-- which basically, repeatability-- may be low, right? You want to have robustness of the methods such that if you repeated this multiple times or you gave it to different groups of people, you want to have some confidence that the results will come up similarly. Very good. Let's see at EPFL, any downsides? So datum dependent, subjective-- what else?

AUDIENCE: So what we are thinking is that the criteria depend, from group to group. It doesn't seem to be clearly defined what you need. Although, there are some that are straightforward and our other point--

AUDIENCE: Datum dependence? That's what we--

AUDIENCE: Yeah, datum dependence. If it was a very easy solution, or if a very bad solution as a reference, then all the others look good. And finally, if you have a-- if you judge on the overall concept, and maybe you discard one or two solutions, but inside those concepts you have ideas that you could keep or use, then they might be lost.

PROFESSOR: OK, so loss of sub-concepts. OK. Good. Anything else here? Mike.

AUDIENCE: We were talking about how it's very easy to tune your metrics to be-- like to be biased.

PROFESSOR: I see. Such gaming, right? Gaming-- gaming the-- basically producing a matrix to give you the answer you want, that you already had preordained, sort of. Right? That essentially what you're saying. . OK. Very good. So these are all-- go ahead.

AUDIENCE: Just sort of one last disadvantage of the method that I think is a bit interesting, because I think the way of grading everything with plus or minus can be [INAUDIBLE], because you can imagine a system, for instance, that is very, very good in one criteria, and then pretty bad in all the others, and it could be a good concept that is disregarded nonetheless.

PROFESSOR: What's your name?

AUDIENCE: Bastian.

PROFESSOR: Pasqual?

AUDIENCE: Sorry?

PROFESSOR: Pasqual. Is your name Pasqual?

AUDIENCE: Bastian.

PROFESSOR: [INAUDIBLE]?

AUDIENCE: Bastian.

PROFESSOR: OK. Keep that point until-- we're going to talk about exactly that point when it comes to non-dominance, OK? So please reserve-- this is great. Just, we'll come back to that in a few minutes, OK? All right, so this is great. I think you really got it. This is exact-- this is a very good list.

So, you may say well, OK this is good for beams and you know very simple things. So, I want to show you an example of an application of the Pugh matrix to what I would argue is a fairly complex architectural decision. And what I'll talk to you about is a thesis that was done here at MIT in 2003, so a while ago, 12 years ago, by Brian Smith. He was an STM student, and was one of the easiest students ever to advise. Very knowledgeable in nuclear propulsion and nuclear power.

He's now the branch chief for nuclear power propulsion at NASA Glenn research center in Ohio. And so, at the time, NASA had a mission that was kind of high priority called JIMO-- Jupiter icy moons mission. And when you go out to Jupiter, and you want to not just do a fly by but you actually want to go to different orbits, you want to do high power imaging, you need a lot of power.

You're far from the sun. RTG's only give you 100 watts or so, per RTG. So nuclear power is pretty much the way to go. But there's a lot of different choices of different nuclear reactor architectures that would potentially-- and then couple that with the propulsion system.

So that was the challenge was sifting through the large space of possibility for nuclear electric power, nuclear exploration class electric power and propulsion systems. And you know, here's the way the study was set up is the expansion phase, the filtering, and then the screening phase. So here's the architectural space the way it was defined.

So nuclear electric power and propulsion is what we're after. And there's different ways to generate these alternatives. So the most important is the design vector, the type of reactor, the operating temperature, the power conversion scheme, the heat exchange, and then the fuel type. And then there were requirements. So this is the power range, the delivery timeline, the fact that you should be able to do it in a single launch, and then the operational lifetime of the system. So those are the shall requirements. Every architecture had to meet these requirements otherwise it would not be considered.

And then there were some things that were assumed as constants. They're shown here. There was a policy vector in terms of funding profiles, international partnerships, at what altitude you could insert the system into orbit, and then influence on future missions. And the objective vector is the criteria for the decision.

Technology-- TRL is Technology Readiness Level. How much infrastructure do you need? Complexity. Strategic value to the nation. This is a little fuzzy, but basically what this means is, could you use it for other applications than exploring Jupiter? The schedule. The launch packaging. Power. Specific mass, which is watts per kilogram. The lifetime.

So it's interesting you have lifetime here and here. So there's a minimum lifetime, right, and if you satisfy this, the extra life you get can actually be used as a decision criteria. How does it interact with the payload? Do you need a lot of shielding? And then adaptability? Can you adapt it for different missions? Yeah?

AUDIENCE: In the previous slide, it listed both possible-- on the previous slide, we had both possible and feasible on there as different kind of criteria. What do you have for a difference for those?

PROFESSOR: So essentially, possible is, this is sort of the combinatorial space. And then, applying the hard constraints based on the requirements vector gets you from possible to feasible. They satisfy your must have requirements, your shall requirements, and then you use, in screening, you use the objective vector-- these criteria-- you use to go from here to here.

So let me show you what this looks like, and I'll post this thesis, if you're interested. It's really-- the whole thesis is about doing this process. So here we have-- each column represents a different reactor architecture. So you can see in the legend what these things mean.

So for the reactor type, we have a liquid metal, gas cooled, or heat pipe type reactor. These are three types of nuclear fission reactors. For fuel, we have two types of fuel, if I remember correctly. We have UO2 or UN. For the temperature at which the reactor would be run, we have medium or high. For the conversion-- this is the thermodynamics. How do you get the heat out of the reactor? We have a Brayton or a Rankine cycle, or a thermoelectric reactor, which is basically directly converting the heat to electricity without a working fluid. And then, D is the heat exchange architecture, which is either direct or indirect.

And so you can see that each of these columns represents a different kind of architecture, and they're colored here by-- the dark is a direct heat pipe with liquid metal reactor, and so forth. So that's essentially-- what you're seeing here is the filtered set that is deemed to be feasible. Potentially feasible.

And then the actual screening is done based on this. So this is the actual Pugh matrix here. So we have the concept combinations, which are the columns. The datum here is the one that's shaded in gray. So it's a liquid metal reactor with a thermal-- it's a liquid metal thermoelectric reactor with indirect heat exchange, UN fuel type and a high operating temperature. And why was this chosen?

Well if you look here, it says SP100 reference. So the US actually has launched-- at least, officially, as far as we know-- one nuclear reactor into space, and that's the SP100. You can look it up. There's a history behind it. And that's the architecture of the SP100. So it was launched. We had data on it. It's a known quantity.

The Russians, it turns out-- Russia has launched a lot of nuclear reactors over the years. But this is sort of the US baseline, if you want to call it that. And then all of these other combinations are compared against the SP100, the reference reactor.

And so, you can see here that the comparison is drawn again. So zero means it's about-- so for TRL, it means it's about equally mature. A plus means it's actually more mature. A negative means it's less mature than the SP100. And at the end of this, you can actually look at the total pluses and zeros and minuses, and then in this case, what Brian did, he did calculate a net score. That's an unweighted score.

Basically, if you give one point-- so let's look at the first one. So it has four pluses, right? It has four pluses, and it has five equal to the SP100, to the datum, and has two minus. So it's worse than two, better in four criteria, and the same in five. So the net score would be plus two, because it's better in two criteria, net. But it's unweighted.

And so in order to figure out what-- based on this net score, so a zero net score would say it's about equal to the datum, but it maybe in other, different criteria. But essentially it's about the same. If it's a plus two, then it is potentially better than the SP100. But you don't know that 100%, because you haven't weighted the criteria. It's really used for screening.

So the ones that are circled are better, essentially, than the datum. And the ones that are-- this is-- I'm sorry, the rank. This is the rank among the set. So if you have a one here, you're in the rank one. And there's two architectures, the first two are rank one.

And then we have this architecture, which is ranked two. The datum itself, plus two other architectures, are rank three. And then you see that there's some-- like, for example, this architecture here is rank eight, right? So it-- why is that? Because it's not better in any criteria than the datum. It's the same in six. And it's worse in five, right? So that gives you an indication that this particular concept probably is not very attractive, and you would eliminate that. OK?

So that's essentially an application of the Pugh matrix to a pretty real-world, complex problem. So what the Pugh matrix is for is essentially structured-- structuring and representing the evaluation procedure. It serves as a common visual. It provides some amount of discipline, and helps break down self-sealing behavior, meaning-- what that means is people defending their concepts without sort of seeing the bigger picture. Encourages teamwork. It helps you eliminate weaker ideas, retain a set of stronger concepts. And then this is interesting-- divergence. It helps to identify opportunities for combination.

There's also a way to use this Pugh matrix methods multiple times. And the way you would do this is you would say, OK so these are the stronger concepts here, right? The ones that do equally well or maybe even a little better than the datum. What are the strengths and weaknesses of each concept and can we hybridize them? Can we create some hybrid concepts?

And so you eliminate the weaker ones. The surviving concepts are kept, and then maybe hybridized with each other. So you expand and create more concepts again. Maybe a little bit more detailed. And then you apply the Pugh matrix again. So it's iterative application of the Pugh matrix, where you eliminate, you keep strong concepts, you hybridize them, and then you repeat it two or three times.

And there's actually research on this, as well. Professor Dan Frye, who is here at MIT in mechanical engineering, has done research on that and shown that that can actually lead to very good outcomes. So this sort of gradual-- so then, you don't have just one expansion and contraction of your concept space, but it sort of goes multiple times. But eventually it does converge. So this is a more refined application of the method.

What Pugh matrix is not good for is automatic decision-making, right? Completely controlling the process. So, you know, the idea of automating the Pugh matrix-- it should not be automatic. And it's not really done-- it's not really good for trade studies.

Challenges-- [LAUGHS] so here's some quotes. "People who have a lot of experience exhibit an impatience. Get on with it. This procedure holds us back." Right? "Strong-willed individuals who have a lot of experience and whose initial concepts have not emerged in the final selection commence a defense based on emotion, experience, and bluster." Right?

So there's this social dynamics that gets unfolded here. So, therefore, it is recommended to do Pugh matrix with a facilitator. Somebody who-- similar to brainstorming, right, where you have a facilitator. So you have a facilitator for Pugh matrix. So somebody who controls the flow and pace of the session, records the results, tries to maintain some discipline. Always compare against the datum.

I've seen Pugh matrix sessions where you start with the datum, and then by the third or fourth concept, people are starting to do pairwise comparisons, and the datum is sort of lost. That should not happen, right? Preventing tangents, but encourage clarification. What do we really mean by criteria? Clarification of the concepts, and then opportunities for divergence in hybrids.

So the critique of the method is-- I think a lot of these things you've already mentioned. The ranking depends on the choice of datum. The weighting. You know, there's no real weighting in the classic method. However, you can implement a weighted version of the Pugh method, but then you have to have the whole discussion about the weightings, a priori.

Next, I'm going to talk about multi-attribute utility. And so Pugh matrix and multi-attribute utility could give you a different rank order of alternatives. So how do you reconcile that? The most important criteria may be intangible and missing from the list.

And my personal opinion on this is that the Pugh matrix is useful and simple to use. It stimulates discussion about the criteria, the alternatives, but it shouldn't be your only means of concept selection. But it's usually one of the first things you should do because you learn a lot from doing it. OK?

Any questions about matrix before I move on? Yes, Sam?

AUDIENCE: Should it always be done with a group that's working together and discussing the concepts? Or could you also have people individually filling them out and then combining their results?

PROFESSOR: So yeah, usually you do it as a group because in the discussion with the group, there's a lot of clarification that happens. Now if you do it separately, and you firewall people from each other and have people do this individually and then try to combine the results later, that's actually called-- that's closer to a method called the Delphi method. Which I'm not going to talk about here.

But the Delphi method means you're asking these kinds of questions to experts. They give you the answers, and then you anonymize the answers and reflect the combined answers back to the group. And then you eliminate the social dynamics, and you can see whether was my assessment an outlier, am I close to the center of the group? So the Delphi method has a lot of benefits, but it's not usually-- you don't have the social dynamic in the same way, which is sort of essential to make this work. Good question though. Any questions at EPFL about Pugh matrix before I move on?

AUDIENCE: No, it's clear.

PROFESSOR: OK. Go ahead.

AUDIENCE: It is useful to almost have an independent group do this, as opposed to everyone generates their ideas and then everyone's trying to fill out the matrix so that their idea wins.

PROFESSOR: Usually not, because if you give it to an independent group, they'd know much less about the problem. They probably weren't involved in generating the alternatives, they weren't involved in selecting the criteria. They're probably not as knowledgeable. And so there is an advantage of doing it within the team. But then, you know, before you actually go and pull the trigger on the final-- that's the PDR. You get vetted by independent people. That's what happens at the PDR. OK.

AUDIENCE: One point here is that you can invite external people to join your discussion. So you have your design team. And it's very good to have the experts to come, sit in the meeting, and then give their opinion, and with the Pugh matrix, as it's pretty straightforward and simple, you probably shouldn't then bias the result, but you could probably enrich it.

PROFESSOR: That's a great point. I agree with that, [INAUDIBLE]. Absolutely. Yeah.

OK. So let's switch gears to utility theory or multi-attribute utility theory. So what's that all about? So like I said, utility is a very deeply rooted concept in economics.

And so it's defined as, "Utility is a measure of relative happiness, or satisfaction, or gratification gained by consuming different bundles of goods and services." This is sort of economic-- very-- this is how economists talk. So it's-- the idea is you want to choose a concept or alternative that will maximize your happiness, your satisfaction, your gratification.

And whenever you buy something, when you go to Anna's Taqueria, or wherever you guys have lunch, or at EPFL you go to the Rowlock center, and you pick from the menu, you're actually doing just that, right?

You have-- how much money do I have in my pocket? How hungry am I? Am I a vegetarian or not? That would filter out some menu items right away, right? And you pick, every day. You pick your lunch. Well, you're doing exactly this. You don't really know that you're doing it, but in your mind, as you're picking from the menu, and then making your choice, you're maximizing your utility at that moment. So we do this every day.

Here, in this class, we talk about designing complex systems. So we need to do it a little bit more deliberately, but this is not just some abstract thing. We do this every day.

And so the idea is, essentially, we have this Consumption Set X, which are our alternatives-- mutually exclusive alternatives-- and then we map that to the real scale. We rank each member of the Consumption Set. So we map the alternatives on this utility scale. In this case, the cardinal scale between zero and one.

So, the way this is done, in order to do the mapping from the criteria to utility, you need these mapping functions. And we call these utility functions. They're called utility functions. So basically we have-- I'm going to use J here for-- J means, essentially, your attribute, your objective. And then U is your utility.

And so there are different shapes of utility functions. And different scholars call them differently. So this is Cook. He says, this is smaller is better, or larger is better. And you see they're essentially monotonically increasing or decreasing curves. Messac, another scholar in multi objective design, calls this class 1S, class 2S. So what would be an example of a smaller is better utility function? What would be an example of that? Go ahead.

AUDIENCE: An example would be the weight of a launch vehicle. It's decreasing.

PROFESSOR: The launch mass?

AUDIENCE: Yes.

PROFESSOR: OK. Launch mass. OK. What about EPFL? Smaller is better.

AUDIENCE: Cost.

PROFESSOR: Cost, right. So, larger is better?

AUDIENCE: Revenue.

PROFESSOR: Revenue. Maybe range, endurance, reliability, right? Then there's-- the next one is strictly concave or strictly convex, which is nominal is better. So the idea there is there's a sweet spot, right? You want the performance or the attribute to be a pretty specific value, and if you deviate from that on the up or down side then the utility decreases. So what would be an example of that?

AUDIENCE: Size of a meal.

PROFESSOR: Size of a meal, OK? Well, some restaurants may beg to differ, right? All you can eat. [LAUGHS]

AUDIENCE: [INAUDIBLE] what their objective is.

PROFESSOR: Yes. All right. Very good. Yes, Sam?

AUDIENCE: [INAUDIBLE] has to interface with something else [INAUDIBLE] you're not [INAUDIBLE] it [INAUDIBLE].

PROFESSOR: Yup, so if there is a very specific interface condition. Yeah. OK, go ahead at EPFL. Nominal is better.

AUDIENCE: Like, for example, ambient temperature.

PROFESSOR: Yeah. So temperature-- ambient temperature, right? Humans are pretty-- we have a fairly narrow range where we say, this is good. Now we can put on a sweater or-- but it's a fairly narrow range.

So, great example. Then the next one is range is better, which is concave or convex. So this is the idea that as long as you're within this interval, within the interval itself, you're sort of indifferent. Right? But then when you drop outside the interval, then utility decreases or increases. And then the last one is very exotic-- non-monotonic utility functions that have multiple peaks. They exist in theory, but in practice, you almost never see them. I can't give you a good example of multi-modal one. But in theory they do exist.

AUDIENCE: Well, there is one, and it's clearly the landing sites on a planet.

PROFESSOR: OK.

AUDIENCE: Because you have the one injection trajectory, and then you have the primary sites of the primary elipse, then the secondary. And it's non-monotonic, mostly [INAUDIBLE]. It doesn't go up, it goes down. So you have ideal. Then you are out of it. Then you would land on rocks. And then you have the next plateau, and maybe a third one, on one pass by.

PROFESSOR: That's an interesting-- that's an interesting comment. I'm trying to figure out what's the attribute-- what's the engineering attribute that goes with that?

AUDIENCE: Well, it's the tolerance on the injection. I know that, at JPL-- by the way, thanks for organizing the [INAUDIBLE]-- they made this presentation, also, of the landing of the rovers. And they have the primary ellipse where they try to land-- well, try-- it depends on the entry. And then they have secondary-- if they miss the entry, then they have very bad option. And then, maybe a little bit later, they maybe can land on the next plateau, or in the next crater, which is useful. So they have this non-monotonic, but degrading, function. The first time is best. The second would still work. But in between, there's a part that doesn't go.

PROFESSOR: OK. Got to think about that one. But that's an interesting-- that's a pretty interesting example. So, the main point here is that in order to calculate utility, you need a translation function between the engineering or financial attributes of each alternative and utility.

If it's not-- if it was directly that, you'd just have a linear, right? It would be a 45 degree line, right? Or negative 45. But typically the mapping from your attribute values to the utility is non-linear. And that's what this says.

So then the challenge is well, how do you get these utility functions? Where do they come from? And the answer is you've got to do interviews, you have to survey people-- the decision-makers, the stakeholders we talked about. They're the ones who have these utility functions in their minds, even if they're not-- So you've got to make those explicit.

So here's an example. You may have three different customers for your system. User attribute, some performance attribute. Customer one is shown here. They need a minimum amount of performance, but once you reach this threshold, after that there's no more utility. But then customer two and customer three are different. They want-- their utility increases gradually, and in fact, customer three doesn't see any much utility until you hit this much higher level.

So one of the challenges, then, in designing these utility functions when you have multiple customers is to solicit this-- and there's interview and survey techniques for doing that-- and then combining those. And then the other challenge in combining them is that, remember at the system level, utility is still between zero and one, right?

So if you had, for example, two attributes, and they both give you perfect utility, and you add them together you'd get a utility of what? You have two attributes. Two. But that can't be because the total system utility can never be better than one. So you have to normalize. As you combine the utilities, you have to normalize them, and actually weight them as well.

So here's the equation when you have two utilities. You basically have the utility of the system just from J1-- your first attribute-- times the utility of J2. This is the mixed term, right, the combined term. Plus the utility of just J1 plus the utility of just J2. Then you have this K factor here-- this capital K factor-- which re-normalizes everything to zero and one. And if you have more than two, then it becomes a matrix calculation.

So it's a little tricky for how to do this properly, but it's well-known how to do this. So the steps in multi-attribute utility analysis are to identify your objectives and attributes, you develop an interview questionnaire, you administer that questionnaire. You then develop your aggregate utility functions, you determine the utility of your alternatives, and then you analyze the results.

And there is one word of caution I want to give you, which is that utility is essentially the surrogate of value, but value often we express in dollars, in monetary terms. But utility is unit-less.

So let me just comment about utility maximization. It's very common and generally well accepted. It's a non-linear combination of your criteria-- your decision criteria. The downside of it is the physical meaning is often lost, because the engineering or customer focused attributes all get combined.

So you say this system has a utility 0.83, and this one has a utility 0.75. So one is better than the other, but immediately you want to say well why is that? What does that mean? So then you have to backtrack and reverse engineer how those utilities were calculated.

You need to obtain a mathematical representation for all utility functions, and-- this is a probably not just US-M centric comment, this is probably everywhere in the world more or less-- but the utility function can vary drastically depending on the decision maker. This is a big issue in government programs.

So you guys, it's uniform day, I guess, today. So how long is a typical tour of duty of a program manager in the Pentagon for a big program? What would you say?

AUDIENCE: It's usually three to four years.

PROFESSOR: So three to four years, like it's written here. And what does that that mean in practice for these utility functions and for programs? Sorry I'm putting you on the spot here.

AUDIENCE: Well, you don't usually have a lot of overlap between the PM. So, as one goes out, another one's coming in, so you're losing a lot of experience, gaining somebody who doesn't have much experience. So there's a learning curve here.

PROFESSOR: Right. That's right. What I'm getting at is-- so there's definitely that effect-- but the priorities may be different. You know, whereas the prior program manager really valued a lot performance or quick response, and the next program manager-- because maybe the context has changed-- is very, very cost conscious. Really wants a system that is very, very affordable, and is willing to sacrifice performance for that.

So the way you can think of this is the shape of the utility curves for those decision makers has now shifted, and because of that the architecture or the choice of concept that you would go for may be different.

AUDIENCE: This happens to NASA all the time, right?

PROFESSOR: Right. And so this is not just a DOD issue. And in commercial world. You know, a new CEO comes in, new CTO comes in, and these utility curves actually shift. So you've got to be to be aware of this.

So one of the big topics there is-- particularly now with NASA is-- how do you choose an architecture, a concept, that is robust to changing utilities and decision makers. So maybe you don't go for the super duper best, most exciting, most capable concept, but you go for the one that's least likely to be overturned or disrupted with the next administration.

And this is a discussion-- active discussion-- right now at NASA headquarters. You know, what are the investments you can make that are going to provide utility, even if the utility function of the future decision makers changes? It's a big, big topic.

The other thing is, of course, this requires your formulation of preferences. Those K factors, those weightings, a priori-- before you've actually scored the alternatives. All right. So the example I want to talk you through is essentially a space tug. So we're going to do trade space exploration. There's a trade space, a design space, of alternatives-- conceptual alternatives.

And we're going to try to understand that using utility theory. And let me explain to you what we mean by space tug. So a space tug is essentially a satellite that has the ability to change its orbital elements-- there are six orbital elements, semi-major axis, eccentricity, inclination, right ascension of the ascending node, and so forth-- of a target satellite by a predefined amount without degrading its functionality in the process.

So here's a picture of the earth. There's two orbits. The space tug is here in the black orbit, and then our target satellite is in the orange orbit. And so the typical process is the space tug waits in its parking orbit-- the black orbit. It gets tasked. It transfers to the other orbit, searches for the target, identifies it rendezvous and approach, docking and capture, does an orbital transfer and then releases the target satellite at the new orbit, verifies its status and then either goes directly to the next target, or returns to the parking orbit.

And as you can imagine, depending on the plane changes that are required here this can be quite expensive. It's less expensive in geosynchronous orbit, and there are some capabilities that we think heard about in the public that US and maybe other countries have as well to do this.

But there's also applications-- commercial applications for it. So, for example, for space debris removal and things like that. So that's what we mean by space tug.

So what are the attributes? What are the-- so this is based on a paper from 2003 called Understanding the Orbital Transfer Vehicle Trade Space. And so here's three attributes that are combined into utility: total delta V capability-- delta V is the change in velocity. This essentially tells you where can it go? You calculate that from essentially the rocket equation.

The second one is response time. How fast can it get there after it has been tasked? And there there's a big distinction between-- it's almost binary-- between electric propulsion, which is very efficient but slow, and then chemical propulsion. The third is the mass of the observation or grappling equipment. And so this tells you what it can do when it gets there. So the size of target satellites that can be actually interacted with.

Those are the three attributes that define utility, and we combine those into a utility between zero and one. And then in this case, cost or a surrogate for cost is kept separately. This is the vehicle wet and dry mass. These are the cost drivers, and this is calculated from some simple scaling relationships.

And so what we're interested in is looking at the trade space of utility versus cost of the system. So how is utility defined for a space tug? So we have response time, which is bad in electric systems. Total utility is a weighted sum, and then we estimate the cost from wet and dry mass.

So the delta V utility is shown here. This is in terms of meters per second. So two, four, six, eight, 10 kilometers per second. These are pretty large delta Vs. And what's interesting is this is a larger is better, but it's a step curve. It's not a smooth curve, it's a step curve. Why is that?

Because as you reach certain delta Vs, it enables operating just in GEO or a LEO-GEO transfer, or a LEO-GEO return. Right? So you can use the space tug more than once, the more delta V you have. And it's a step curve because as you hit this value of delta V, you can now operate in a different regime.

The capability. The payload mass utility is essentially discretized between low, medium, high, and extreme. So low is for small satellites. Medium satellites up to one metric ton, up to three metric tons, and then more than five metric tons, which are the big satellites in geosynchronous orbit. And then the weighting factors are we're going to capability 30%, delta V 60%, and the time responsiveness of the system 10%. So time responsiveness is not that important compared to the other two.

So once we have this-- I'm not going to go through all the details of the calculations. That's in that paper, which I'll post. You get this. So this is a cloud of points. Each point here represents a particular alternative or architecture, and we have the cost of the system in millions of dollars. So this is not cheap. All right? This is a billion dollars, two billion, three billion. And then we have this utility, this dimensionless utility. And, of course, what we're particularly interested in is this lower right corner: high utility lower cost systems.

And what we did here is to identify some particularly interesting architecture, as they're shown here in this space. So let me just explain two of them. One of them, here-- and we gave them names that are recognizable. So this point here, it's below 0.4, so it's a rather lower utility system, but it's also relatively affordable. We call it the bi-prop low-earth orbit tender.

So it's intended for use in low-earth orbit and it has a dry mass of about 680 kilograms, a wet mass of 1,400. So the difference is propellant-- about 800 kilograms of propellant-- and has reasonable size and mass fraction. And it's fairly responsive.

Another alternative is what we call the electric geocruiser. Sounds cool, doesn't it? This is basically a space tug that operates only in geosynchronous orbit. So you launch into geosynchronous orbit, but once it's there it can do a lot. But it's electric, so it's kind of slow. Seven hundred eleven kilograms dry mass, 1,100 kilograms wet mass. And it includes return of the tug to a safe orbit. And this is sort of a versatile space tug in the GEO belt.

So that's the idea. You calculate utility, shown on the x-axis, you have the cost of the system. You have this cloud of points. And you start understanding what are the interesting architectures in that trade space. This is the same trade space. But now what we've done here is we've shown the choice of propulsion system is critical. And what you can see here, it's really almost impossible to get to a utility of one. It's very, very hard.

And the reason for this is the rocket equation. You see how these-- so the blue are the bi-propellant architectures. The purple are the cryogenic, so these are [? lux ?] hydrogen propulsion. The yellow ones are the electric propulsion system, and then we also have nuclear propulsion, here, which clearly is challenging and has policy implications. So you can see the nuclear propulsion gets you close to much higher utility, but it's also much, much more expensive. But in all cases, as you're trying to get more and more utility, at some point you hit the wall because of the rocket equation, which is very non-linear.

You know, more fuel-- in order to carry more fuel, you need more dry mass, more fuel and dry mass requires more fuel to push, and the system blows up on you. And you can see this in these curves. So you get some physical intuition in terms of the shape of the space. Yeah?

AUDIENCE: Is there a reason between this layering of cost?

PROFESSOR: Yeah. They're essentially, if I remember correctly, the capability of the system. Remember there's this small, medium, large, you know the size of the grappling equipment? I think that's the tiering that you see.

AUDIENCE: That doesn't change the utility?

PROFESSOR: Well, it depends on the weighting, right? So as you change the weighting among the attributes, that space can get scrambled. This space is valid for that particular weighting. So any questions about-- this is utility theory, multi-attribute utility theory, applied to trade space exploration. Any questions? Go ahead. EPFL, do you have a question?

OK. So now what you see here is implicitly, is there's nothing here in the lower right corner. There is no system that has perfect utility and is low cost. Right? There's an empty space. We'd love to be here, but there's nothing there. So the best we can do is get close to that. And that's what I want to talk about next, which is this concept of non-dominance.

So what do we mean by non-dominance? What is a Pareto frontier and what is multi-objective optimization? The key point about this is that when you do non-dominance filtering, when you look for Pareto frontiers, you do not need to express your preferences ahead of time. Right?

So when you do you a weighted Pugh matrix or when you do a utility theory, you had to define ahead of time, before you did the scoring, what's more important-- fuel efficiency is twice as important to me as cost. Right? The weighting, the preferences had to be expressed a priori-- before you did the scoring. And there's arguments that that's not the best thing to do, but you can do it.

Here in non-dominance and multi-objective optimization, eventually you have to express your preferences, but you do it after you've scored the options. That's a really important distinction.

So let me give you a little history here on this. And what's really interesting is there's a Lausanne connection. OK? There's a connection with Lausanne. So the word Pareto frontier is named after a an economist, who actually started as an engineer, Vilfredo Pareto. He was born in Paris in 1848 and then graduated from the University of Turin in 1870. Very much in engineering, civil engineering, his thesis title was The Fundamental Principles of Equilibrium and Solid Bodies. Something we take-- that's pretty basic now. Forces and torques in equilibrium on a body.

Now he worked as a civil engineer in Florence, and then he got interested in philosophy, politics, and economics, and he started to think about this in terms of how does this concept of equilibrium apply to economics? And in 1893, he actually became a professor at the University of Lausanne, which is located right next to EPFL, and started applying this to economics and societal theory.

And then he did-- some of his work was a bit controversial, but the one that I want to talk about here is this the idea of the Pareto optimum. And I will-- let me just read this quote to you. "The optimum allocation of the resources of a society is not attained, so long as it is possible to make at least one individual better off in his or her own estimation, while keeping others as well off as before in their own estimation."

So what it means is if you can make an investment or make a change in society that makes a particular individual or group of individuals better off in their own estimation, without having to take the resources from another group and make them less well-off, then you've not reached an optimal allocation of resources. You know, if there can be a win-win-- if everybody can be better off by making certain investments, you haven't yet found the optimal strategy.

Only when-- the only way to make somebody better off in their own estimation than somebody else is by taking-- and this is a big political debate still today-- by taking from one group and redistributing resources to the other. When you're in that situation, then you are at the Pareto optimal point. OK?

So that's the key idea underlying this. So what this means mathematically is that an optimal solution -- X*-- is optimal if and only if-- for a feasible solution-- so, first of all the solution has to be feasible. And this is when you have multiple criteria. We're trying to do vector optimization, and x must be what we call an efficient solution, and it's efficient only if its objective vector-- J of X-- is non- dominated.

And what this means is that if you're looking at a point, it's only efficient when it is not possible to move from that point to another point without degrading at least one of the other objectives. So if you can move from a particular design point to another design point, and all the objectives get better, you're not yet efficient. You're not yet at a Pareto optimal point. And this gets us to the notion of dominance.

So dominance-- let's say we have two designs, two alternatives-- J1 and J2. Those are their objective vectors. It means that-- this is for maximization. We're trying to maximize. It means J1 dominates J2 weakly if J1 is better or equal J2, and at least in one of the objectives, I, strictly better than J2. So you could have two alternatives. This gets back to Pugh matrix and the question-- who asked about the ties? You did, right? It's so long ago. It's like, wait a minute, it's like an hour ago, you know?

So we have two alternatives, OK? They're all equal. They're tied in all criteria, except for one, where J1 is better than J2. Which means that J1 will dominate J2, but weakly, because there's some ties. Right? And then there's a stronger definition, which is that J1 strongly dominates dominates J2 if and only if it is better strictly better in all attributes. So for J1 to dominate J2 strongly, it's got to be better in all attributes than J2. Does that make sense?

OK. So I know this is a bit abstract. So let's do a concept question that's sort of based on a hypothetical but real example. So assume-- let's say you're in charge of designing in new commercial aircraft, and there's four criteria. You want to maximize range. You want to minimize the costs-- dollars per kilometer flown. You want to maximize the capacity of the airplane-- number of passengers. And you want to maximize cruise speed in kilometers per hour. So it's a multi-objective aircraft design.

You do all the work. You do concept generation. You come up with eight concepts for airplanes, and you evaluate them. And they're shown here-- one, two, three, et cetera, through eight. So let's just look at number one. What does this mean?

Airplane number one, concept number one, has a range of 7,587 kilometers. It has a operating cost of $321 per kilometer. It can carry 112 passengers. And it has a cruise speed or max speed of 950 kilometers per hour.

So I'm going to leave this up. This is like a five minute-- this is-- on the next slide, I'm going to ask you-- and we'll come back to this-- which of these airplane designs, these eight, are non-dominated. Meaning that they're not dominated by any of the other. And we're going to apply weak dominance here.

That was going to be your question, right? Weak dominance. So I don't think it actually matters. I don't think there's too many ties in here, but apply weak dominance. So which of these are non-dominated? Meaning that you wouldn't discard them right off the bat, because there are some strong features that they have. So work through this. You might want to write down which ones are non-dominated. And then we'll do the concert question, and then we'll show you the answer.

AUDIENCE: I just want to make sure. So non-dominated as in there is not one that could be considered better than it?

PROFESSOR: Yeah. So, this is the definition. All right. So let me-- so please submit your answers. So here are the choices. Which airplane designs are non-dominated? Five, six, and seven. One, three, four, and eight. One, two, three, four, and eight. Two, three, five, and six. One, three, four, and seven. Or you need-- this is not answerable. You need more information to answer the question. OK?

So we have a pretty good distribution here. So the wisdom of crowds I think prevails. So the correct answer is the third one-- one, two, three, four, and eight. So 36% of you got it right. Now let me let me show you the solution to this. So it is, in fact, possible to answer this without additional information.

So the way to do this is you have to do pairwise comparisons. Now you said about-- I think your strategy-- What did you say, early on?

AUDIENCE: I was looking at maximums.

PROFESSOR: You were looking at maximums. So you were saying who's the best performer among the concepts for each criteria? And you found those pretty easily, I would assume, right? So what--

AUDIENCE: [INAUDIBLE]

PROFESSOR: OK. So you've got to get the sign right. Do you have the mic on, by the way. So what would-- if that concept is the best performer in a criterion, what does that tell you?

AUDIENCE: It means that none of the other performers have something equal to or greater than that performer for that criterion.

PROFESSOR: And this relates to the question-- is it Martin? Martin? In the red shirt, first row? I didn't get your name right. I'm sorry.

AUDIENCE: Bastian.

PROFESSOR: Bastian. Bastian. Bastian. You remember at the start of the lecture, you said-- what did you say at the start of the lecture?

AUDIENCE: I said that if you evaluate all the concepts based on just plus or minus for the criteria, you might have one where you evaluate all the-- a bunch of criteria that are negatives, but there's just one that is positive. But that one might be very, very positive.

PROFESSOR: Right. So that's exactly the point here. That's exactly this point. If you are-- if this concept is the best performer in just one of the criteria. Like it has the best speed, or the lowest cost, but it's terrible, terrible in everything else, it doesn't matter. In terms of dominance, it is non-dominated. It cannot be dominated by any other concept. If you are the best in class, even just for one criteria-- does that make sense? You cannot be dominated by any other design if you're the best performer on just one criterion.

So let's think about track and field. So the people-- who's doing track and field? Any athletes here? Like runners or javelin or-- Veronica. I didn't know that about you. Cross country and track. OK. So 5K? All right.

So let's say-- I'm sure you were very good at 5K. And how was your shot putt?

AUDIENCE: I never competed in the shot putt.

PROFESSOR: OK. So then we don't know, right? But the point I'm trying to make is you could be a super duper specialist in one criterion, and that's-- you can will win a gold medal with that. But you're very maybe not good in other things. So that's the point here is in order to be dominated, if you're the best performer-- so, right off the bat, in this exercise here, in our little exercise, you can remove all the concepts that are best in class from the set. Because they cannot be dominated. So they have to be non-dominated.

OK, so the way you do the scoring with pairwise comparisons is you say, let's compare one and two. Concept one and two. Well, one is better in range, right? And it's better in speed. But then the second concept is better in two and three. So the score here is two versus two, which means neither one nor two dominate each other, right? Even if it's-- what if it's three to one?

What if it was a three to one score? It would still be true, right? That one doesn't-- one is, in a sense, better because it's better than the other. But it doesn't dominate it, because dominance is a very crisp definition. Now this is different, right? Compare one versus number six. Concept one versus concept six.

One is better in all four criteria to six. So clearly, solution one dominates solution six, and as a rational decision maker, you can eliminate concept six from the set because it is completely dominated by concept one. OK? So in order to be dominated, a solution must have essentially a score of zero in a pairwise comparison.

Now, if we apply this to the full set of eight we can actually show this as a matrix. I call that the domination matrix. It shows which solutions dominate which other solution-- the horizontal rows and the vertical rows. So the way you read this is so where you see these dots, these are dominance relationships in the pairwise comparison. So this tells you that solution two dominates solution five. This tells us, if we look column-wise, solution seven is dominated by solution two and solution eight.

So, if we do the row sum-- if we sum this along the rows, the row indicates how many solutions this particular solution dominates. And-- but the question was about non-dominance, so we need to look at the column sum. The column sum indicates by how many other solutions the kth solution or concept is dominated. And when you do this column sum, you can see that concepts one, two, three, and four have a zero. They're not dominated by any other design. Column eight has a zero, and concepts five, six, and seven are dominated each by at least one other concept, which means that they're dominated, and the others are non dominated.

So when you do your concept selection, you can actually apply this to a much bigger set. Your filter out all the dominated solutions. And it turns out the bigger your number of alternatives or architectures, the smaller percentage-wise the set will be of non-dominated solutions. And those are the ones you really want to focus on. Is that clear? I know this takes a little thinking about, what does this really mean? But but that's a very rigorous way to do it.

And then this gets us to, eventually, this gets us to the notion of Pareto optimality. So what's the relationship between Pareto optimality and non-dominance. So let's say we have two objectives, J1-- we want to maximize J1. Maybe that's performance, endurance. We want to minimize J2-- maybe cost, for example. So the Utopian point, or the optimal corner to be in, is the lower right. We want to maximize J1, minimize J2. There's nothing here. It's empty. There's nothing feasible here.

So the best we can do is get close to it. So when you look at a set of discrete points, discrete concepts, those concepts can have three properties. They can be D, meaning dominated. So all the red points shown here are dominated. And the way you can see this if you draw a little-- let me try to let me try to draw a box here.

Where's the eraser?

So let's pick a point, a particular point-- Let's pick this point right here. Why do we know that this point is dominated, just looking at it graphically? How do you know that? Go ahead.

AUDIENCE: The point is not at the front.

PROFESSOR: It's not at the front, but the way the way to find first of all, whether it's dominated, which is relative to the other solutions, is we can just draw a horizontal line here, and a vertical line here. And so in a multi-dimensional space, you're essentially drawing a hyper-cube. And down here is our utopia. This is where we would love to be, but we can't be. It's not feasible.

And you see in this box, between the point and the utopia, there is this other guy here. This point here. And because it's there, it's closer to the utopia, and it dominates this point in both objectives. Let's take this point here. Let's do the same thing. Draw a vertical line and a horizontal line. And there's a whole bunch of points, all these points right here are all the box, closer to the Utopian point. So this point here is actually dominated by one, two, three, four, five, six seven other points.

So that's essentially dominated. Let me erase this again. Then we have these darker red, purple points. Let me just circle them real quick. So we have this point here, this point, this point, this point, this one, and this one. And those are what we call non-dominated, like we just did the filtering, meaning there there's no other point in the set that's better than it in all criteria. So you would consider them as concepts to choose from.

But they're not exactly on that dashed line, which means that there is a way to improve them. There's a way to tweak them, to optimize them further, if you could to get them right onto that dashed line. They're close but they're not exactly on that dashed line. So you could tweak these concepts to get them closer to the Pareto front. But they're still pretty good, because they're non-dominated.

And then we have those grey points, which are shown here. There's three of them. Let me use a different color for these. Let me use a different color, maybe green. Here. This point, this point, and this point, they are Pareto optimal. Meaning they're on the Pareto front. There's no way to improve them, such that you could make both objectives better.

Right? If you want to make J1 better, you have to sacrifice on J2. So the only way to modify these points is to slide along the Pareto front, meaning you're trading off the objectives. Do you see that? And so, in my spring class that I teach, called Multi-disciplinary System Optimization, we get into this pretty deeply. Like how do you actually do this?

But the point I want you to keep for now is that when you select the concept, do not select a dominated one. Get rid of the dominated ones. Filter them out. And then you still have to choose among the others, and that's where your preferences come in. Right, so given the blue and green choices here, which one are you going to recommend at the end? And that this is sort of the preferences between maximizing J1 and minimizing J2. That's a social process of coming up with your preferences.

But you now determine those preferences after you've scored all the concepts. Do you see the difference? That's pretty fundamental. OK. So, one more thing and then we'll wrap it up.

So now you've gone through all this work. You've done your concept generation, concept selections, scoring, and you found the one concept that you're going to go forward with. That's when you do PDR.

So what is the PDR? The Preliminary Design Review. You explain what concept and system architecture was chosen. You explain why it was chosen, and you want this to be approved as the design baseline for going forward and doing more detailed design work. You compare it against the rejected alternatives. Typically at the PDR, you will spend some time on saying, these are the other interesting concepts that almost were selected, but we didn't because of these reasons. You show some quantitative analysis that gives confidence, especially you want to show that the requirements that you had defined and agreed upon at the SRR can be satisfied by your chosen concept.

There's maybe still some risk or some uncertainty. So any risk reduction experiments or prototypes. So you can actually do prototyping at this stage. You don't want to spend too much on that, but you want to do enough of it so you can reduce risks. And then you do a preview of the detailed design phase between PDR leading up to CDR. So here-- I'm not going to read this-- but this is the description of what the PDR is all about. In the NASA system engineering handbook on page 177, there's a very detailed description of entrance criteria, exit criteria, et cetera, et cetera.

So here's a description from a PDR from a European program, and you can see it's a pretty big group, right? This is like 30, 40 people. And because the PDR is such an important decision, that's when you also invite external stakeholders. And you want to make sure that people come away from the PDR with confidence that, yeah, we have a good system architecture. We've picked the good concept, and it's worthwhile now putting all this work into getting it to a detailed design that can actually be built.

OK. Any questions about PDR, or any experiences from PDRs that people want to share?

AUDIENCE: I have a question about the PDR. So you said there's 30 people in that picture. How do you reconcile that with seven plus, minus two rule?

PROFESSOR: Well, it's seven, plus or minus two, squared, right? So you have a hierarchy of people. Not everybody is a decision maker here.

AUDIENCE: I see.

PROFESSOR: So you probably have people who did some of the thermal design, some people did the cost modeling. So, it's a great question actually. You want many, many voices and people who contributed to make sure the PDR presents all the information you have. You don't want to do a PDR with just seven people.

AUDIENCE: But I guess you could say that in the presentation, I guess, only seven plus or minus two, will people be driving at that point in time.

PROFESSOR: So there will be a leadership, right? There is an organizational issue. But the point here is that it's a big review. It's a big milestone. Please, go ahead.

AUDIENCE: The big issue here is that all the people who are not decision makers, that are in the picture, they will have to implement the detailed design, so it's very important to have them in the process. Otherwise, you have to go back to your organization and do the PDR again to explain the what's and the why's. This is why you try to have these very large meetings. When you do a pre-PDR, you do a dry-run, and you send stuff in advance and make sure that you're really all agreeing finally, if possible, you get the discrepancies off the table. But then you all agree that this is how to go forward and then the worker bees can go and do it.

PROFESSOR: Yes, that's another reason for being inclusive. I agree with that. Any other-- who has gone through a PDR in their prior work? Anybody want to share? Marissa. Tell us about how that went. I mean as much-- I know you can't probably we share everything but--

AUDIENCE: For us, I mean, it was a pretty big milestone. You work a lot leading up to a PDR and then you finally have all the really big stakeholders in the room. And a lot of them have been involved during the design phase, but when you actually have them all sitting there. And I think getting the buy-in was also really-- I think that's a really big part of PDR. Like normally you're feeling pretty confident in your designs, but then actually making sure whoever the agency is that you're working with, then your stakeholders are also kind of all on the same page. It's pretty critical.

PROFESSOR: Did you discover problems or discrepancies during the PDR?

AUDIENCE: Yeah. I definitely think-- just generally we've had delta PDRs, if things weren't completely thought through. Or you'd have a fair number of action items that come out of PDR to go and address before you really are allowed to start moving towards CDR.

PROFESSOR: Good. Any comments or experiences with PDRs at EPFL?

AUDIENCE: I did a PDR at the European Space Agency during the REXUS BEXUS program three years ago, and as far as I remember it was very appreciated to bring as many details as possible. And to be very close to that critical design review stage in the PDR.

PROFESSOR: That's interesting. So, that's good. I think there's a good to have detail and so forth. But what's the downside? Can you think of a negative what would be what would be not so good about almost being a CDR right there? Can you think about a downside to it?

AUDIENCE: Probably not taking too much time in order to clarify the concept and the requirement because you are in a rush to finish the product.

PROFESSOR: That's a good point. So it takes more time. But what I'm thinking about is it's very difficult to then undo the decision or backtrack. Let's say that at the PDR that there's some-- it could be that you really forgot something very important. And that your concept is actually flawed. But you didn't know it as you were working inside your team. And then if you have a design that's almost CDR quality, it's very difficult to undo that. So I would say that a PDR-- I actually believe that at a PDR you should have some detail, but not too much yet.

I think a true PDR should be at the system architectural level, with some detail but not all the details. But thank you. That's a great comment. Thank you very much.

So let me close it here. So in terms of concept selection during conceptual design, we can use Pugh matrix selection. And that's essentially useful when we don't yet have those detailed mathematical models. It's qualitative. And then as we move more into preliminary design, we use utility analysis, especially for noncommercial systems where a net present value-- a financial-- in commercial companies, you will often do a financial analysis as a measure of utility.

And then this concept of non-dominance is very, very important. Generating many designs and then applying Pareto filters to try to come down to the non-dominated set. So that's essentially the essence of it. So I hope that was clear. A-4 will be posted later today.