Usability testing with Crowdsourcing

How many users typically try out your web site before it goes live? How have users contributed to refining the features on your site over time?

I don’t think that usability testing has been a core development practice on any of the systems I’ve implemented. Sometimes we have brought in future users for a one-off test, or the development team has tried to act as users for final testing of stories before being handed over, or QA sometimes does some exploratory testing, but I’ve never seen usability testing as part of the weekly process on a project.

There are a number of ways that feedback can be collected from popular websites without too much effort, for example But what if you don’t have many users yet, or you want some feedback on specific actions? You can ask people around the office not on your project, or hire some expensive consultants, however what about an online solution?

Feedback Army

Michael Buckley sent me a link to Feedback Army. For $10 you get 10 people visiting your website to provide answers/feedback on 6 questions you have set.

I tried this out on developing it we were concerned about usability but there was no process in place to test and refine this beyond gut reaction, so there is lots of changes that could be recommended. On the down side, it is not really a site for the general public but rather for development teams or customers working with a development team on a project, so how would a common web user provide much useful feedback?

Setting up your test on feedbackarmy is straightforward.

Fill in the website address, fill in up to 6 questions – I just went with the example set, enter an email for where the results should be sent, went with 10 reviewers for $10, and then paid. Within an hour the first result came in, and after 4 hours all the results had been logged.

So what do you get a few hours later?

Some answers were really not interesting, for example:

1.this site is used to improve software quality .
2.there is nothing on the site that confused is very user friendly.
3.this site can be improved by adding some animation effects and by adding bright colors. works quickly and very simple to use.Firefox 3.5 (Windows)
Delhi, India

You can Reject responses. I didn’t bother.

Answers went up from there in terms of value. Representative of the best 2 or 3:

  1. This site helps software development teams analyze, monitor, and learn from metrics collected by their ongoing project activities.

2. When I first viewed the site and read what DevCreek is about from the text on the home page, I still did not know where to go from there. I then looked at the top for an “about us” link but it was not there.

3. The main content area should be in the center of the page. There is way too much empty space on the right. Another thing that may help is adding a right sidebar also. Putting the “About Us” in the top navigation menu area where login and register is located since most new users may look their first. May I suggest that you guys add a video tutorial on the home page. Another thing I would add is a search box at the top right of the site.

4. This site works by providing tools that make gathering raw project metrics easier.

Firefox 3.5 (Linux)
Cleveland, Ohio, United States

They haven’t gotten very far browsing the site, and their comments seem quite generic. No magic here, still we have talked about the video tutorial before, someone else recommended centering the content, and so on. So it has some merit.

The rest of the feedback

Now I’ve seen it in action, could you improve the questions to increase the value of average and best responses, perhaps by being more specific rather than the existing general, front-page like questions? For specialist sites like DevCreek without filtering down the users for a better match on the target audience perhaps there are some low limits on what you could expect to learn.

Disappointed by FeedbackArmy I tried an alternative site that trades closer to my understanding of what a typical usability test would involve while still being online, relatively cheap, and prompt:

This is, again, a crowdsourcing approach to usability testing, however they charge $29 per test but provide a screencast of the test with commentary in addition to a text summary. The structure differs quite significantly from FeedbackArmy so instead of questions and reviewers who answer the questions, they have users who test a site framed by a scenario and a series of tasks to complete during the test. You can also target an audience of testers through some simple filters, and by providing a custom requirement as text at the beginning of the tasks section.

They give an indication that tasks should take approximately 15 minutes to complete. I’m assuming feedbackarmy is in the minute range unless a rare individual takes a fancy to the task and abandons the financial benefits.

This time I tried to limit the testers to only people that had been involved in a software development project before, and went with a series of tasks that walked the testers through a path on the website rather than the approach on FeedbackArmy where the questions were related to understanding the purpose of the website and general questions about how to improve the site.

After an hour or so the first result came in and all three arrived over the course of the night. Slower than FeedbackArmy, though I don’t know how much the limit of software development experience impacted this.

Here is the results summary page. For each tester you can see a summary of their findings:

These aren’t the “answers” to the task rather they seem to be a reflection based on a common form set by UserTesting.

You can also watch a screencast of the testers trying to complete the set tasks together with an audio track from the testers describing their thought process. They have an on-line tool to watch the videos that allows you to add comments at particular points or you can download an off-line copy in WMV or MOV format.

Screencast Video

Though the summary page doesn’t seem that much better than FeedbackArmy the screen cast is significantly more interesting. All three testers were good at talking through what they were doing during the test, as well as including side comments on recommendations or general preferences of theirs. The recordings were from 7mins – 13mins so there is an investment by the development team to work through the material. However the video can be captivating. The commentary might very well be coherent but by watching the video you see the user struggling with navigation, or misunderstanding what they are seeing, and so on. For example one of the testers believed that they had completed one of the tasks by finding a report, in fact they had misunderstood entirely a pair of charts; this would not have come out in a summary and required an expert of the site to be able to understand what had happened. For someone who has only read about the “gold standard” of usability testing labs it was fascinating to watch the videos as I recognized similar reactions to watching the testers actions coming out, so I was flipping from frustration at the difficulties the testers were experiencing to the relief from realising that much of what I was seeing could be improved quite straightforwardly. Perhaps with a group reviewing the recordings some deeper interpretations of the users assumptions and issues with the site could come out.

UserTesting provides far more contextual details describing the tester than FeedbackArmy. You can see basic information about the user, you can alse rate the tester, and be able to choose specific testers for follow-up tests.

What Next?

Could I use this on my project?

  • Your system has to be available on the internet for these services to come into the picture.
  • If you are developing a solution for a customer, would you be allowed to include enough information to identify the customer or would you need to mask that out somehow?
  • But our site needs password control! UserTesting suggests including a username/password in the tasks description that could be used by the tester to login for the session.
  • The application is not for the general user and requires specific contextual knowledge for it to be used. I’m sure that there are lots of fundamental navigation issues that could gained from any kind of user. UserTesting allows you to describe some constraints that the user should have before accepting the test – might help. UserTesting also allows you to request an earlier user, so perhaps they can start building up the skills to progress further into the system.
  • Will the development team know how to act upon the results? The difficulties that I was seeing suggested some quite obvious first steps that would help the situation. I suspect the bigger problem is trying to prioritise all the ideas for improvements that come out of reviewing the sessions.
  • How could you include this in the ongoing development process? I suppose the temptation is to only do this at the end but delivery pressures would probably dominate. Would you try and do this for each story, or in case one of the development team raises a concern about a specific issue on the site.

I really liked the $29 usability test and screencast of I would like to start using this routinely on a project, though the hire cost and analysis time would be a dampener on its wilder uses. How could we use this on a project? Could we sell this as part of our development process to the customer? Would facing the limitations of our systems through this mechanism lead to the commitment to expand the team to include a permanent User Experience role?

$10 for 10 answers of (and the Mechanical Turk¬†under the covers) is so out there that even though I was disappointed by these first usability results I would happily spend $50 to see if it was possible to improve the quality of the answers by learning how to set more effective questions. Perhaps this approach is not really suitable for usability testing of websites but it may still be interesting to investigate what other uses this mechanism could be adapted for to gain feedback on our systems or design choices. For $10 a throw you don’t have to have many hits to be up by a healthy amount.

It's only fair to share...
Share on Facebook
Tweet about this on Twitter
Share on LinkedIn

Leave a Reply