A card sort exercise is used in Information Architecture so that a common set of categories and relationships emerges that can be used to organise such areas as website navigation.

To perform a card sort, a participant is given a set of index cards with terms written on them. This person sorts the cards either into a predefined category or chooses there own name for the category name.

Tullis & Wood’s research in 2004 used 168 participants, with 48 cards.  Kath Straub draws the following conclusion from the paper:-

Based on their simulations, Tullis & Woods concluded cardsorting studies with just 20-30 participants are robust and predictive.

Tullis and Wood actually point to some area of concern about their research, namely that :-

As always, we must exercise appropriate caution in generalizing results from one study. Results will obviously differ as a function of the homogeneity of the participants in a sample and such things as the instructions given to the participants for the card-sorting task.

The 168 participants, even though they where from all over the world, they all worked at one corporation. Fidelity Investments. Corporations have cultures. If you where to do the study at another company would the numbers be the same?

What always matters in these sorts of studies is the variance of the data. Living in Europe one quickly realises that people are not homogeneous. The Germans have many ways of categorising bread, while the British just have three, brown, white, and hovis. There is the famous global example of the Eskimos having 48 different names for snow. How an Eskimos will categorise the weather will be very different than a person living in the desert.

Even within one field there can be much disagreement in how to categorise data. If you take the field of Taxonomy, which is the science of classification, there is much disagreement on how to categorise things.  One of the reasons we do not know how many livings species there are is because of a disagreement on what belongs to what species. For example scientists can not agree on the number of genius of the Eucalyptus.

So how many people are enough. Well it depends on the homogeneity of your target market and what you want them to classify. Twenty participants may be enough and may not be. What effects how people will categorise items depends on their knowledge and opinion. The more diverse the knowledge and opinion the more participants you will need. The only way in finding out is doing some tests and look at the variance.

In a later post we will discuss this more about the number of particpants needed for different kinds of tests.

The Babylonians, who were famous for their astrological observations and calculations (aided by their invention of the abacus)

Last night Martyn Perks gave a talk at the monthly UPA event in London arguing that creativity is stifled by an over reliance on metrics. He argued that insight for good design can not come from number crunching.

I spent allot of Martyn’s talk shaking my head.

One of the points that he made was that Usability was driven by metrics. I could not agree, because there are few people using real metrics to measure usability. Jeff Sauro and his SUM method is one of the few exceptions. The practise of the most common methods are dominated by Expert Reviews and Lab Studies with few participants. Many of the reports that I have seen are dominated by opinion and are not objective.

I can understand Martyn’s anger at some of methods that are used to collect and analyse numbers. As we have found with Webnographer it is easy to collect data, what is hard is to find out what the data means. It is easy to jump to conclusions from the data, that don’t hold out to be true.

A good example of misleading data is people trying to make sense of Google’s trends. If you look at the chart above it may make you jump to the conclusion that Palin is leading the race to the Whitehouse. Obviously this is not the case. (Thanks otrops for the link.)

The argument that I made to Martyn is that metrics can help at the right stage, and make the difference between an implementation of a design that is good, and one that is great. Many great designs from the walkman to the iPod have been dominated by a great ideas person, Akio Morita or Steve Jobs for example, who pushes through the great leap in innovation. After the initial design there are many versions of the same product, each one an improvement on the last.

The research used to inform the changes to the product should be tested using Sientific Method to see if the numbers that one is using hold true.

While data gathering and metrics can not come up with an idea, they can help to refine a product and improve it.