design:pairwise_comparison

THIS PAGE IS OBSOLETE AS OF 1 JAN 2020

Pairwise Comparison

Pairwise comparison is a tool to rank a set of decision-making criteria and rate the criteria on a relative scale of importance.

Making decisions requires comparing alternatives with respect to a set of criteria. If there are more than two criteria, determining which criteria are more important can itself be a serious problem. One would like to be able to rank the criteria in order of importance, and to assign to the criteria some relative ranking indicating the degree of importance of each criterion with respect to the other criteria.

Here are some examples:

  • Planning a Vacation: your criteria might be cost, number of locations visited, quality of the locations, travel time as a fraction of total vacation time, and closeness to foreign friends and family. Which one of these is most important? How can you plan your vacation to optimize your enjoyment? If you have many dear family members wherever you want to travel, you will likely choose to weigh that criterion more heavily than a friend of yours travelling to the same place but who has no family at all there.
  • Choosing a Job Offer: your criteria might be salary, benefits, opportunity for advancement, location, travel times, stock options and other incentives, viability of the company, and working atmosphere. Which is most important? How can you increase the likelihood of choosing the best job? If the job you're after is such that you can work from home, you will likely weigh “working atmosphere” lower than someone applying for a job that will require him to spend his working hours in a cubicle surrounded by dozens of other workers.
  • Designing a Toaster: your criteria are cost, quality, expected product life, reliability, manufacturability, aesthetics, and safety. Which is most important? How do you choose the best design with respect to all these criteria? A college student expecting to need a toaster for just a few years will likely not rate aesthetics as highly as an established professional who has just invested heavily in a major home renovation.

It can be very difficult to rank and weight criteria. It can become insurmountable in complex problems because every criterion must be weighted with respect to every other criterion simultaneously; this is a problem that grows arithmetically. (For example, given 5 criteria, there are 4+3+2+1=10 relationships to consider, and for 10 criteria there are 45 relationships to consider).

Pairwise comparison is one way to determine how to evaluate alternatives by providing an easy and reliable means to rate and rank decision-making criteria. It is often used to assign weights to design criteria in concept evaluation.

Pairwise comparison is implemented in two stages:

  1. determine qualitatively which criteria are more important – i.e. establish a rank order of the criteria, and
  2. assign to each criterion a quantitative weight such that the qualitative rank order is satisfied.

For our approach, this step has already been done, through the definition of a PRS. The criteria we will use are the product characteristics. Remember that each PC is really just a label for all the requirements flowing from it in the PRS.

Here is an example, using some fairly typical, general PCs. You would use the PCs from your specific PRS.

A B C D E F G H I
functionality A
durability B
quality C
affordability D
fabricability E
usability F
maintainability G
safety H
marketability I

Each cell in the matrix corresponds to a comparison of a pair of items (hence the name “pairwise comparison”). The cells will contain the item that is considered the most important of the pair.

Obviously, we only need one triangle of the matrix, because comparing A to B is the same as comparing B to A. Since the rows and columns contain exactly the same things in the same order, one triangle of the matrix will contain just a mirror image of the other.

Furthermore, the diagonal of the matrix is irrelevant – it simply doesn't make sense to consider how important one criterion is with respect to itself!

So now we have:

A B C D E F G H I
functionality A -
durability B - -
quality C - - -
affordability D - - - -
fabricability E - - - - -
usability F - - - - - -
maintainability G - - - - - - -
safety H - - - - - - - -
marketability I - - - - - - - - -

For each row, consider the criterion in the row with respect to each criterion in the rest of that row.

The first comparison in our matrix is functionality versus durability. Which is more important? Discuss this in your teams and reach a consensus to that question. In the corresponding cell of the matrix, put the letter of the criterion that is most important. Do not consider any other criteria beyond the two that you are comparing.

Then go on to the next pair (in our example, functionality versus quality). If we really, really think that two criteria are of equal importance, then we put both letters in the corresponding cell.

Note that the individual comparisons are pairwise – we completely ignore the other criteria.

Say we decide that functionality is more important than durability. We would then put a A into cell (2, 4) of the matrix. We continue doing this till all the empty cells have been filled.

We might end up with something like the following:

A B C D E F G H I
functionality A - A C D A F A AH I
durability B - - C D B B B H BI
quality C - - - D C F C H C
affordability D - - - - D F D D I
fabricability E - - - - - F E H E
usability F - - - - - - F FH I
maintainability G - - - - - - - H I
safety H - - - - - - - - H
marketability I - - - - - - - - -

Notice the double letters in some cells. We have used this convention to indicate that the two criteria being compared are of equal importance; that is, you just could not decide which of the two criteria was more important.

This step must be documented to provide justifications for your decisions.

  • Ideally, every decision you make in constructing the PWC chart needs to be justified.
  • In the example above, that's 36 decisions - one for each pair that you compared.
  • Justifications would be given with respect to your requirements and framed with respect to your product strategy, user groups, and personas.
  • You can document this by simply listing each pair of PCs you compared and giving a short (1-4 sentences) justification - the shorter, the better - referencing specific elements of your PSS and PRS as required.

Now we simply make a sorted list of the criteria, ranked by the number of cells containing their flag letter. For example, functionality is marked by the letter A, so functionality gets a rank value of 4, because there are 4 A in the matrix. This leads to:

  • safety (7)
  • usability and affordability(6)
  • quality and marketability (5)
  • functionality and durability (4)
  • fabricability (2)
  • maintainability (0)

Finally, we want to associate weights with our criteria so that the relative ranking from the pairwise comparison is satisfied.

There are two basic constraints on how we assign the weights.

  1. The total of all the weights must be 100%.
  2. The weights must obey the relative ranking given by the pairwise comparison.

We can either begin to wrestle with the problem in a strictly ad-hoc manner, or we can try to structure our solution. It's inevitable that some iteration will be required, so there's no point in looking for a method that will give us actual weights in one pass. However, we can try to set up an initial set of values that does satisfy the constraints, and then tweak the values until they are satisfactory for all stakeholders.

One very easy way to set that initial set of values is to assume a linear proportion between all the weights and solve the following equation (for our example):

100 = 7x + 6x + 6x + 5x + 5x + 4x + 4x + 2x + 0x

or, x = 2.56 (approx), where the coefficients in the equation are the number of occurrences of each criterion in the pairwise comparison matrix. This leads to:

  • safety: 7x = 18%
  • usability and affordability: 6x = 15% each
  • quality and marketability: 5x = 13% each
  • functionality and durability: 4x = 10% each
  • fabricability: 2x = 5%
  • maintainability: 0x = 1%

NOTE: the 1% for maintainability arose by gathering up all the round-off error from the other calculations.

There are obvious problems with this approach. For example, the lowest ranked item in a pairwise comparison can often turn out to have a weight of zero, as we found in the example above. However, we cannot assume that zero importance implies we can omit it altogether. If we do that, we have to redo the entire pairwise comparison, and then some other criterion may end up having zero importance.

So, when we have a criterion that appears to have zero importance, we borrow a fractional amount from other criteria and give it to that last-ranked criterion, just to make sure it counts for something.

Other strange effects can occur. For example, it is possible to get a situation where given three criteria, say A, B, and C, we find that A is more important than B, that B is more important than C, and that C is more important than A. This is a paradoxical situation (known as Arrow's Paradox) in that it makes no sense. Generally, this happens when people work individually rather than in teams.

No matter what else happens, however, it is essential that all stakeholders in a design agree to the actual weights. Pairwise comparison is therefore best done in teams and not individually.

Salustri has a simple Google spreadsheet that can be copied and used as a template for carrying out a pairwise comparison task. The sheet will track the sums and percentages for you; all you have to do is list the criteria and make the actual comparisons.

Remember to enter into the cells the letters corresponding to the attributes themselves not the column letters.

You may only change cells that have a white background; do not change anything in gray cells, or you will break the spreadsheet.

  • design/pairwise_comparison.txt
  • Last modified: 2020.03.12 13:30
  • (external edit)