Every time we talk about remote usability testing, many people have many different ideas of what remote testing is.  And there are many different remote testing methods.

Rex Hartson in 1996, defined remote usability testing as the method “wherein the evaluator, performing observation and analysis, is separated in space and/or time from the user”.

Two simple distinguishing factors, which are now commonly accepted, are that remote testing falls into two main categories:

1. Synchronous / moderated remote testing, where the evaluator has direct communication with the participant during the evaluation. This is normally carried out using screen sharing software, or talking over the phone.
2. Asynchronous / automated remote testing/ automated remote testing, where the evaluator has no contact with test participants. Asynchronous remote testing is the method used by Webnographer.

Asynchronous remote testing falls into three sub categories.

The simplest is analog remote testing, such as critical incident reporting, where the participant is sent a paper questionnaire which is filled in after carrying out the test. While this may seem like an easy method for evaluators, this is a method which can be very time consuming for the participant.

A bit more advanced are task-based online questionnaires, which are online tools that query the user for feedback after carrying out the task. Yet, those do not track any interaction on the page which makes it difficult to infer why problems are occurred. They therefore can only offer summative results such as success rates, time on task and satisfaction.

The most advanced tools use the CARUT methodology (Combined Asynchronous Remote Testing methodology), which combine logs of the user interaction during the task (hovering, clicks, key press, etc.) with questionnaires, which record rich qualitative feedback. This allows inferences about where and why problems occurred and offers both summative and formative results. Webnographer uses the CARUT methodology.

Yet, a challenge with most of the remote tracking tools which use the CARUT methodology is that they require the test participant to download software for it to work. We have found most people, worried about viruses and trojans, do not want to download software, especially one that tracks them. The other disadvantage of downloaded software is that it normally only works with certain types of browsers and operating systems. We have found that being able to work with different combinations of computers and browsers is important as we found the test participant’s configuration of their computers has a major impact on how they use a site.

Webnographer does not hold the above challenges. It works with nearly all computers, and with the most browsers. The user does not have to download anything and the website does not need to be modified. Webnographer gives therefore full freedom to carry out any test, including tests on your competitor’s website for competitive analysis.

This is an update of where we are with Webnographer.

We are developing Webnographer using Agile Project Management Techniques. Agile builds the project in very small iterations of between 1 and 4 weeks. At the end of each iteration the software should be working. To give an example with Webnographer in our first iteration the whole system was usable from start to end. I.e. we could build a test, get participants to test a website, and analyse the results.

Not only did we want to just build a tool that could give summative results, for example how long a user takes on a task. But we wanted to build a remote testing tool that could be used for formative testing as well. Webnographer enables this by capturing users interactions on the page, as well as qualitative and quantitative questionnaire data. This means that Webnographer does not just report the state of a system (as in summative evaluations), but also provides insight into where, how and why user errors occur (as in formative evaluations).

So how could we build complex usability testing software in a week? We didn’t. When we started, the building of the test and the analysis of the results had to be done by hand. Over a number of iterations what had to be done by hand got automated.

The Agile technique is a revolt against the traditional waterfall approach. Under the old fashioned waterfall approach the business specifies exactly the software that will be built, the developers build it, and then it gets tested, and hopefully released. The challenge with the waterfall approach is that it often led to software projects running over time and budget.

With Agile each iteration can either add features or rework what has been built before. This has the advantage for developing Webnographer, that as we get feedback from each of the tests that we run with clients, the results feed back into each iteration and add to an improvement of the tool.

Up until now, we have mainly focused on the design of the test set up. The test set up was the most time consuming part and easiest to automate for Webnographer. Before it had its simple interface, to create a test each one had to be hand crafted, taking half a day. Modification of a test was hard, time consuming, and error prone. So we have focused on developing an easy to use interface for the test design. This is now complete.

The next iterations will focus on the analysis part of the product. The driving force is that we want to show people actionable results in an easy to understand way. Sample reports are to follow soon.

When I was 11 years old my mother gave me for my birthday an Acorn Atom (One of the first personal computers). That was 28 years ago. Since that day I have been the one to rescue most of the technology problems in my family. Luckily when my brothers got older they took over some of the rescue calls.

This computer led me to start being interested in how people use computers.

Most of the technology challenges my family faced have been them struggling with usability issues.  Later when I was paid to write programs I wanted to make software easy to use, because I wanted people to use the software. When I started Deckchair.com with Bob Geldof our whole ethos was to make booking flights less frustrating, and simpler. I think we succeeded on the whole. Before the Internet flight booking engines, it took months for somebody to learn how to book a flight using a computer. When online bookings became available consumers where able to do it themselves without training. How many people still use a Travel Agent now?

Many Usability experts blame the programmer for the Usability Challenge. If you read Alan Coopers book The Inmates Are Running the Asylum you will find a rant in how programmers are to blame. He uses a very narrow stereotype of the programmer to illustrate his argument. The second issue I have with Coopers blame game, is that he does not acknowledge that most of the innovation in usability has actually come from programmers. Jef Raskin and the apple Mac, Douglas Engelbart and the GUI, the mouse, and hypertext. Or even more recent movements like Web 2.0 the innovations have mainly been led from the programming field.

Most programmers I have worked with are as frustrated by usability as much as anybody else, maybe even more so because they have to use computers all the time, and as Paul Graham, the creator of the first web based application said :-

Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.

I believe that we can start solving the challenge of Usability once everybody starts understanding what the issues are. Very few people design systems not to be usable. If there is an usability issue, it is there because the programmer or designer overlooked something, or did not expect the user to behave in that way, or did not realise the importance of the issue, and lastly did not understand what the user did not understand.

We need more ways to discover how people behave using computers, and then communicate that back to the designers and programmers. That is why I co-started Webnographer to help the designers and programmers understand how people use the software that they have created.

It boils down to what Stuart Card said “All designs are predictions, all predictions fail.”

 

 

James giving the first Webnographer demo.

Today, James gave the first pubic demo of Webnographer at UX Brighton.

His short talk explained how Webnographer can offer fast insight into Usablity issues, and how it enables the collection of user behavioural data. James also highlighted the academic research that Webnographer is based on, such as the link between the eye and the mouse movements.

In a nut shell: Webnographer is a new service for Asynchronous Unmoderated Remote User Testing method. It enables faster and cheaper user testing outside of the labs, with user in their real environment. The services has been designed so that a team can get results back in just a few hours, while using more than the normal number of participants.

The talk was well received and stimulated interesting discussions. We were overwhelmed by the positive feedback and interest.