Every time we talk about remote usability testing, many people have many different ideas of what remote testing is.  And there are many different remote testing methods.

Rex Hartson in 1996, defined remote usability testing as the method “wherein the evaluator, performing observation and analysis, is separated in space and/or time from the user”.

Two simple distinguishing factors, which are now commonly accepted, are that remote testing falls into two main categories:

1. Synchronous / moderated remote testing, where the evaluator has direct communication with the participant during the evaluation. This is normally carried out using screen sharing software, or talking over the phone.
2. Asynchronous / automated remote testing/ automated remote testing, where the evaluator has no contact with test participants. Asynchronous remote testing is the method used by Webnographer.

Asynchronous remote testing falls into three sub categories.

The simplest is analog remote testing, such as critical incident reporting, where the participant is sent a paper questionnaire which is filled in after carrying out the test. While this may seem like an easy method for evaluators, this is a method which can be very time consuming for the participant.

A bit more advanced are task-based online questionnaires, which are online tools that query the user for feedback after carrying out the task. Yet, those do not track any interaction on the page which makes it difficult to infer why problems are occurred. They therefore can only offer summative results such as success rates, time on task and satisfaction.

The most advanced tools use the CARUT methodology (Combined Asynchronous Remote Testing methodology), which combine logs of the user interaction during the task (hovering, clicks, key press, etc.) with questionnaires, which record rich qualitative feedback. This allows inferences about where and why problems occurred and offers both summative and formative results. Webnographer uses the CARUT methodology.

Yet, a challenge with most of the remote tracking tools which use the CARUT methodology is that they require the test participant to download software for it to work. We have found most people, worried about viruses and trojans, do not want to download software, especially one that tracks them. The other disadvantage of downloaded software is that it normally only works with certain types of browsers and operating systems. We have found that being able to work with different combinations of computers and browsers is important as we found the test participant’s configuration of their computers has a major impact on how they use a site.

Webnographer does not hold the above challenges. It works with nearly all computers, and with the most browsers. The user does not have to download anything and the website does not need to be modified. Webnographer gives therefore full freedom to carry out any test, including tests on your competitor’s website for competitive analysis.

Without innovation the internet would still look like this. (Gutenberg's printing press. Photograph by Matthias Kabel)

The dilemma of visual convention vs. ground breaking new design seems to be a fearsome concern for usability specialists.  In a recent blog post on the Concept 7 blog, Stefan Wobben quotes a paper by Luis Santa-Maria and Mary C. Dyson form the University of Reading that investigated the impact of violating visual conventions on user’s performance and orientation. Santa-Maria and Dyson explain:

“Although initially violating visual conventions might hinder user performance and leave users disoriented this experiment indicates their experiment indicates that these problems can be short-lived and users can adapt reasonably fast to a new set of visual conventions.”

This is good to hear, yet its no news as such. If design had always only followed convention we would not have progressed from the written word to the printing press to computers and the internet.

“So the decision to whether conform or violate visual conventions when designing a website should ponder that although users might adapt quickly to novelty there is an initial performance hindrance and disorientation.”

They have a good point there in encouraging those violations. Too many studies focus on first time use, but not repeat users, how behaviour changes over time, and the experience and use of the system by expert users. A single lab study as a Q+A exercise just before the launch of your website is not going to do the trick in gaining this understanding. Usability is an ongoing process, not a one of label of approval. As Harry Brignal pointed out on his blog: “A UX designer’s job is never done.”

The one great thing about remote usability testing is that it is cost efficient and can therefore be carried out more often, than a lab study. Webnographer, as a remote usability testing tool, makes ongoing testing simple and affordable. It makes it easy to test the learning curve and behaviour and satisfaction of experienced users.

As Jared Spool explained back in 2003 small ongoing changes carry far less risk, then a major relaunch and re-design, which is very likely going to fail. As with the prinitng press, changes in improvements were small streched out over a lenght of time and we ented up with the internet, which makes the spreading of ideas and information easier than ever. For website design, the small changes allow you to measure the effect of that change on your users, and you will find out whether the change has made your site better or just different, and how it affects your users over time.