One of my last missions at my former workplace was to lead a usability testing. We chose to do those tests for new services, that where written by Nati in WCF technology.
Please prepare yourself because I'm going to share with you some of my work.
Let's start…
Usability testing purpose:
The main purpose for usability testing is to measure the comfortable when using a public API. These tests should be run on specific component or small number of components with high relationship between. The measure will make by watching other people that using that API. You should notice to their comments, behaviors and fillings when they had written that code (And not just asking them "Is this API is enough understandable?").
What we need to test?
In usability testing we should check four things:
1. Time to accomplish the task – How much time it took to the user to accomplish this task.
2. Accuracy – How many mistake the user made until he accomplish the task?
3. Recall – Check the users memory about how to using this API.
4. Emotional response – What are the user fill after he finish the task (confidence\ Nervous, will he give a good recommends on this API to his friends?)
The Usability testing phases:
First phase – prepare the tests
1. The first thing is to make sure that the API is ready.
2. Scenarios - Write some real scenarios that uses this API. These scenarios should be simple and short. Each scenario should be written on a separated page. For example: We develop an API to add an attachment to mail message. The scenario will be: Create a new mail message with attachment and send it.
3. Running tests planning
a. Each test should be no longer than half hour.
b. It's really recommended to start with some simple tests (Its will give the user good fillings and improve his productivity).
c. You can return on the same test twice (It's will be better if you change the test explanation).
d. Publish a details schedule to the running tests day.
4. Write a basic documentation for this API but without a details example how to use it. You should publish the documentation to the testers a few days before the tests will occur.
5. It's highly recommended to create a series of questions. Ask them during the running the tests process.
Second phase – choose appropriate resources (testers & environment)
1. Choose 5-6 programmers to run the usability testing. These programmers should be unfamiliar with this API (from the "Hallway usability testing" methodology).
2. The tests execute in the natural environment of the testers.
Third phase – running the usability tests
1. Before your tester starting the tests, you should give them a short briefing. The briefing will contains some instructions like:
a. Explain them that the purpose for this session is how mach this API is easy to use and not how much they good programmers.
b. Ask them to speak loudly and to share all theirs responses, fillings and everything that come up (Part of the "Think aloud protocol").
2. Don't help them! Remember that the API isn't including you as an on-line helpdesk resource. If the tester stuck, you can show him the global way in order to continue in the next tests.
3. Ask the relevant questions (That written in the preparation phase) and remember to document it.
4. You can record the running tests process (Example to implementation of the "eye tracing" method); it does can help you to collect the feedback.
5. Try not to tell when the tester need to move to the next task, let them handle their time.
6. Prepare yourself to get a lot comments about that the API isn't so perfect (As you thought before…).
7. Enjoy from been viewer! It is really fun and will cause you to be better on what you are so loved to do.
Fourth phase – Feedback Analysis
1. Summarize all feedback that you got from the testers in one big table (Excel will do this job wonderful). The table should contain the parameters that I mentioned before (Time to accomplish the task, Accuracy, recall and emotional response).
2. Now it's the time to make the decision about the things that we are going to fix.
3. In this phase you should write what you learnt in this process.
4. You should notify the testers about all the things what fixed from their feedback.
If you want to learn more about Usability testing try these references: