Every time we talk about remote usability testing, many people have many different ideas of what remote testing is.  And there are many different remote testing methods.

Rex Hartson in 1996, defined remote usability testing as the method “wherein the evaluator, performing observation and analysis, is separated in space and/or time from the user”.

Two simple distinguishing factors, which are now commonly accepted, are that remote testing falls into two main categories:

1. Synchronous / moderated remote testing, where the evaluator has direct communication with the participant during the evaluation. This is normally carried out using screen sharing software, or talking over the phone.
2. Asynchronous / automated remote testing/ automated remote testing, where the evaluator has no contact with test participants. Asynchronous remote testing is the method used by Webnographer.

Asynchronous remote testing falls into three sub categories.

The simplest is analog remote testing, such as critical incident reporting, where the participant is sent a paper questionnaire which is filled in after carrying out the test. While this may seem like an easy method for evaluators, this is a method which can be very time consuming for the participant.

A bit more advanced are task-based online questionnaires, which are online tools that query the user for feedback after carrying out the task. Yet, those do not track any interaction on the page which makes it difficult to infer why problems are occurred. They therefore can only offer summative results such as success rates, time on task and satisfaction.

The most advanced tools use the CARUT methodology (Combined Asynchronous Remote Testing methodology), which combine logs of the user interaction during the task (hovering, clicks, key press, etc.) with questionnaires, which record rich qualitative feedback. This allows inferences about where and why problems occurred and offers both summative and formative results. Webnographer uses the CARUT methodology.

Yet, a challenge with most of the remote tracking tools which use the CARUT methodology is that they require the test participant to download software for it to work. We have found most people, worried about viruses and trojans, do not want to download software, especially one that tracks them. The other disadvantage of downloaded software is that it normally only works with certain types of browsers and operating systems. We have found that being able to work with different combinations of computers and browsers is important as we found the test participant’s configuration of their computers has a major impact on how they use a site.

Webnographer does not hold the above challenges. It works with nearly all computers, and with the most browsers. The user does not have to download anything and the website does not need to be modified. Webnographer gives therefore full freedom to carry out any test, including tests on your competitor’s website for competitive analysis.

Shopping in Europe by supermuch / James

I was asked last week, if culture effects the user experience and the users’ use of the system.

What we have found is that consumers knowledge has a major impact on online behaviour. For example most Europeans hardly use the internet to shop. Therefore they have both little knowledge and experience of online buying. Using numbers from Jupiter and Forester research shows the UK online user spends twice as much online as the German, and French user.

Rory Sutherland of Ogilvy points out that most marketers base their strategies on US research, Ogilvy has done a study that shows key differences between shopper habits offline in the US and the UK. We believe this is the case online as well.

Other significant differences in behaviour that we have found include that the British find reading a timetable from top to bottom easier. Most Europeans are the opposite, and find reading a timetable left to right faster. We do not know if a similar behaviour would exist with product listings, and maybe a study is needed to find out.

Online Shoping by Gareth Saunders

Many other differences in culture and knowledge also affect behaviour. Including time — some people do not understand the 24 hour clock others do not understand the 12 hour clock. The reading of prices — Europeans use a dot to denote thousands, and coma to denote the decimal point. Anglo Saxons do the opposite. This can lead somebody at a fast glance to think a product is cheaper or more expensive than it is.

To make matters even more complex Europe is becoming more culturally mixed. Over fifty percent of Londoners where not born in London. There are so many French people living in London, that President Sarkozy of France visited London during his election bid. He claimed that “London, has become one of the great French cities”. A Polish, or German, or Scandinavian Politician would make the same claim for their country. This cultural mix doesn’t just apply to London, but holds true for many other countries, and cites in Europe, including Spain, Italy, and France.

We have found from previous studies that a Dutch person living in London, is quite willing to go through the whole process of purchasing online in English until they reached the terms and conditions which they want to read in Dutch. They could have switched languages earlier in the process. And the Dutch user wants far more reassurances over being protected from online fraud than the Italians.

What this means for an e-commerce site is that it needs to work with people with different cultural backgrounds in different cities in different countries. It needs to work for an English people living in Spain, just as much that it needs to work for a Dutch person living in Berlin.

Not only have we found many issues on culture, but additionally the configuration of the users computer effects usability. The size of the screen, and if you are displaying rich media, the type and version of the browser, and the power of the processor. Most lab studies are done with one standard set-up of the computer, using the same size monitor.

As Dr. Harry Brignull points out it is critical to use the right participants for your usability testing. If you restricted yourself to a lab, but your users are distributed all around the world, then the question is: “Are you testing the right users?”

Webnographer can help identifying cultural issues, and many more. Using some of our own analytic techniques we can gain insight into qualitative feedback and satisfaction ratings from the questions asked of the participants, but also gain knowledge from the participants behaviour while carrying out the task, by tracking their interactions on the page.

Kath Straub aka “DriveBy” points out Tom Tullis and Larry Wood’s research, How Many Users Are Enough for a Card-Sorting Study?

A card sort exercise is used in Information Architecture so that a common set of categories and relationships emerges that can be used to organise such areas as website navigation.

To perform a card sort, a participant is given a set of index cards with terms written on them. This person sorts the cards either into a predefined category or chooses there own name for the category name.

Tullis & Wood’s research in 2004 used 168 participants, with 48 cards.  Kath Straub draws the following conclusion from the paper:-

Based on their simulations, Tullis & Woods concluded cardsorting studies with just 20-30 participants are robust and predictive.

Tullis and Wood actually point to some area of concern about their research, namely that :-

As always, we must exercise appropriate caution in generalizing results from one study. Results will obviously differ as a function of the homogeneity of the participants in a sample and such things as the instructions given to the participants for the card-sorting task.

The 168 participants, even though they where from all over the world, they all worked at one corporation. Fidelity Investments. Corporations have cultures. If you where to do the study at another company would the numbers be the same?

What always matters in these sorts of studies is the variance of the data. Living in Europe one quickly realises that people are not homogeneous. The Germans have many ways of categorising bread, while the British just have three, brown, white, and hovis. There is the famous global example of the Eskimos having 48 different names for snow. How an Eskimos will categorise the weather will be very different than a person living in the desert.

Even within one field there can be much disagreement in how to categorise data. If you take the field of Taxonomy, which is the science of classification, there is much disagreement on how to categorise things.  One of the reasons we do not know how many livings species there are is because of a disagreement on what belongs to what species. For example scientists can not agree on the number of genius of the Eucalyptus.

So how many people are enough. Well it depends on the homogeneity of your target market and what you want them to classify. Twenty participants may be enough and may not be. What effects how people will categorise items depends on their knowledge and opinion. The more diverse the knowledge and opinion the more participants you will need. The only way in finding out is doing some tests and look at the variance.

In a later post we will discuss this more about the number of particpants needed for different kinds of tests.