[ Home | Links ] Updated 2005-12-27

Pairwise Comparison

Filippo A. Salustri, Ph.D., P.Eng.

Introduction

Pairwise comparison is a kind of divide-and-conquer problem-solving method. It allows one to determine the relative order (ranking) of a group of items. This is often used as part of a process of assigning weights to criteria in design concept development.

Consider the example of the concept evaluation of a construction ladder. In that problem, the following criteria were determined to pertain to the evaluation of various syringe concepts.

• functionality
• durability
• quality
• affordability
• fabricability
• usability
• maintainability
• safety
• marketability

In order to weigh these criteria, we could simply tackle the problem all at once and, for example, discuss what weights should be assigned to each criterion. This can be very difficult; it can become an insurmountable task in more complex problems where there may be dozens of criteria. An alternative is to divide the problem of assigning weights into two parts:

1. determine qualitatively which criteria are more important - i.e. establish a ranking of the criteria, and
2. assign each criterion a quantitative weight so that the qualitative ranking is satisfied.

Pairwise comparison can be used for step 1. Here's how it would work.

Step 1: Identify the Criteria to be Ranked

This is already done. Typically, the criteria can be derived from the functional requirements and product characteristics determined during the development of a Product Design Specification. Assuming you keep the PDS very handy while you evaluate concepts, you can just use the main PCs as the criteria - as we've done here.

Step 2: Arrange the Criteria in a NxN Matrix

For the ladder design problem, the matrix would look like:

Obviously, we need only one triangle of the matrix. That is, since the rows and columns contain exactly the same things in the same order, one triangle of the matrix will contain exactly the same cells as the other triangle.

Furthermore, the diagonal itself does not matter - it simply doesn't make sense to consider how important one criterion is with respect to itself! So now we have:

Step 3: Compare Pairs of Items

For each row, consider the item in the row with respect to each item in the rest of the row. In the example, we begin with functionality versus durability. Which is more important?

In the corresponding cell of the matrix, we put the letter that we consider most important in each pairwise comparison. If we really, really think the two criteria are equally important, we put both letters in the cell. There are of course other ways to fill the cells. The point is that whatever way is chosen, it must represent which of the items is more important.

Note that the comparison is pairwise - we completely ignore all other criteria.

Say we determine that functionality is more important than durability. We would put a A in cell (2,4) of the matrix. We continue doing this till all the first row is complete. We then proceed to the second row and repeat until the upper triangle of the matrix is filled. We could end up with the following.

Note the double letters. We have used this convention to indicate that there is no difference in importance between the items being compared.

Step 4: Create the Ranking of Items

Now we simply create an ordered list of the items, ranked by the number of cells containing their flag letter. This leads to:

1. safety (7)
2. usability (6)
3. affordability (6)
4. quality (5)
5. marketability (5)
6. functionality (4)
7. durability (4)
8. fabricability (2)
9. maintainability (0)

Conclusion

We now have a ranked list of the relative importance of the various criteria. Given this, we can now begin considering the actual weights we wish to attach. This is the extent of the pairwise comparison technique, but we can take this particular problem one step further and consider how we could assign the actual weights.

Consider the constraints on the problem of assigning weights.

1. The total of all the weights must be 100%.
2. The weights must obey the qualitative ranking given above.

We can either begin to wrestle with the problem in a strictly ad-hoc manner, or we can try to structure our solution. It's inevitable that some iteration will be required, so there's no point in looking for a method that will give us actual weights in one pass.

However, we can try to set up an initial set of values that does satisfy the constraints, and then tweak the values until they are satisfactory to all stakeholders.

One very easy way to get that initial set of values is to assume a linear proportion between all the weights and solve the following equation:

100 = 7x + 6x + 6x + 5x + 5x + 4x + 4x + 2x + 0x
Therefore, x = 2.56 (approx).

where the coefficients are the number of occurences of each criterion in the matrix. This leads to:

• safety: 7x = 18
• usability & affordability: 6x = 15
• quality & marketability: 5x = 13
• functionality & durability: 4x = 10
• fabricability: 2x = 5
• maintainability: 0x = 1

NOTE: the "1" for maintainability arose by gathering all the round-off from the other calculations.

There are obviously problems with this. For example, the lowest ranked item in a pairwise comparison will, strictly speaking always end up with zero importance. This means that we cannot assume that zero importance implies we can omit it altogether. If, for example, we omit handling because it ranked zero, and redo the pairwise comparison, we will then find that readability and portability will now rank zero. This would quickly lead to a set of NO criteria at all.

Still, there is now a baseline from which meaningful discussion can ensue. It is noted that we can get the actual weights shown in the concept evaluation matrix for the syringe by taking a few points away from each criterion and giving them to the handling criterion. This may be enough.

Whatever is done, however, it is essential that all stakeholders in the project agree to the actual weights.