Kath Straub aka “DriveBy” points out Tom Tullis and Larry Wood’s research, How Many Users Are Enough for a Card-Sorting Study?

A card sort exercise is used in Information Architecture so that a common set of categories and relationships emerges that can be used to organise such areas as website navigation.

To perform a card sort, a participant is given a set of index cards with terms written on them. This person sorts the cards either into a predefined category or chooses there own name for the category name.

Tullis & Wood’s research in 2004 used 168 participants, with 48 cards.  Kath Straub draws the following conclusion from the paper:-

Based on their simulations, Tullis & Woods concluded cardsorting studies with just 20-30 participants are robust and predictive.

Tullis and Wood actually point to some area of concern about their research, namely that :-

As always, we must exercise appropriate caution in generalizing results from one study. Results will obviously differ as a function of the homogeneity of the participants in a sample and such things as the instructions given to the participants for the card-sorting task.

The 168 participants, even though they where from all over the world, they all worked at one corporation. Fidelity Investments. Corporations have cultures. If you where to do the study at another company would the numbers be the same?

What always matters in these sorts of studies is the variance of the data. Living in Europe one quickly realises that people are not homogeneous. The Germans have many ways of categorising bread, while the British just have three, brown, white, and hovis. There is the famous global example of the Eskimos having 48 different names for snow. How an Eskimos will categorise the weather will be very different than a person living in the desert.

Even within one field there can be much disagreement in how to categorise data. If you take the field of Taxonomy, which is the science of classification, there is much disagreement on how to categorise things.  One of the reasons we do not know how many livings species there are is because of a disagreement on what belongs to what species. For example scientists can not agree on the number of genius of the Eucalyptus.

So how many people are enough. Well it depends on the homogeneity of your target market and what you want them to classify. Twenty participants may be enough and may not be. What effects how people will categorise items depends on their knowledge and opinion. The more diverse the knowledge and opinion the more participants you will need. The only way in finding out is doing some tests and look at the variance.

In a later post we will discuss this more about the number of particpants needed for different kinds of tests.