Human Interface Notes
_____________________________________________________________________________

Note #1:   User Observation:  Guidelines for Apple Developers       

           Written by:  Kathleen Gomoll & Anne Nicol             January 1990        
           (Supersedes Human Interface Update #11)
_____________________________________________________________________________

Discussion of guidelines for user observation.
_____________________________________________________________________________

Introduction

User testing covers a wide range of activities designed to obtain information
on the interactions between users and computers.  Most user testing requires 
considerable expertise in research methods, as well as skill in using complex 
data collection tools.  For example, user testing techniques include: 
interviews, focus groups, surveys, timed performance tests, keystroke
protocols, and controlled laboratory experiments.  Of the many user testing
techniques available, user observation is one technique that can be used by
anyone with a concern for including the user in the product development
process.

User observation involves watching and listening carefully to users as they
work with a product.  Although it is possible to collect far more elaborate 
data, observing users is a quick way to obtain an objective view of a product. 


When to observe users

User observation should be an integral part of the design process--from the
initial concept to the product's release.  Software design that includes user
observation is an iterative process; user feedback provides the data for making 
design modifications.

As Figure 1 demonstrates, this iterative process assumes that preliminary 
human interface designs should exist prior to the development of underlying
code.  Interface designs should be tested frequently to determine which design
should be implemented.  Then, as the code develops, the entire product should
be tested and revised several times.

                            ___/ Design /___
                           /   \        \   \
                          /                  \
                         /                    \
                        /                      \
                        |                     /|\
                       \|/                     |
                    Prototype ----->-----Test With Users

               Figure 1 - User observation in software design


Preparing for a user observation

Set an objective

Before you do any testing, you should take time to figure out what you're 
testing and what you're not.   In other words, determine an objective for 
your test that focuses on a specific aspect of the product.  By limiting the 
scope of the test, you're more likely to get information that helps you 
solve a specific problem.

Design the tasks

Your test participant will work through one or more specific tasks.  
These tasks should be real tasks that you expect most users will do when 
they use your product.  The entire user observation should not run over 
anhour, so you should design tasks that focus on the part of the product 
you're studying.  For example, if you want to know whether your menus are 
useful, you could design a task that requires the participant to access 
the menus frequently.  After you determine which tasks to use, write them 
out as short, simple instructions.

    Important:  Your instructions must be clear and complete,
    but they should not explain how to do things you're trying
    to test.  For example, if you want to find out whether users
    can navigatethrough your program easily, don't give them 
    instructions for navigation.  Or, if you want to know 
    whether your interface is self-explanatory, don't describe 
    how it works.  This concept is extremely important to remember.
    If you teach your participants about something you're trying
    to test, your data will not be useful.


Decide upon the use of videotape

Although you can observe users effectively without using special recording 
equipment, you may want to use videotape to capture the entire session.  
By videotaping the session, you collect an enormous amount of valuable 
information that you can review and analyze after the test is over. If video 
equipment is not available, a tape recorder can be helpful for recording 
what is said during the test.

Determine the setting

The ideal setting for user observation is a quiet, enclosed room with a 
desk, the appropriate hardware and software, a video camera, and two 
microphones (one for you and one for the participant).  Of course, you may
not have all these things available when you need to observe; therefore, you 
should try to approximate the ideal setting as closely as you can. If you have
to conduct the observation in a regular office, ask the people around you to
keep the noise level down during the observation.  The key is to make the 
environment as interruption-free as possible.  Get the participants out of
their offices, away from phone calls and people who might drop by.

Find representative users

When looking for participants, try to find people who have the same 
experience level as the typical user for your product.  Don't ask people 
you work with regularly to be participants because they are probably familiar
with your product or your opinions about the product.  Generally, you 
should look for people who are familiar with the hardware you 
use but are not familiar with your product.

You may want to ask pairs of people to work together on your tasks.  
You'll find that people working in pairs usually talk more than people 
working alone, and they also tend to discuss features of the product 
and explain things to each other.


10 steps for conducting a user observation

The following instructions guide you through a simple user observation.
Remember, this test is not designed as an experiment, so you will not
get statistical results.  You can, however, see where people have
difficulty using your product, and you can use that information to 
improve it.

These instructions are organized into steps.  Under most of the steps, 
there is some explanatory text and a bulleted list.  The bulleted list
contains sample statements that you can read to the participant. 
(Feel free to modify the statements to suit your product and the situation.)

  1.  Introduce yourself.

  2.  Describe the purpose of the observation (in general terms).

      Set the participant at ease by stressing that you're 
      trying to find problems in the product.  For example, you 
      could say:

        o  You're helping us by trying out this product 
           in its early stages.
        o  We're looking for places where the product may 
           be difficult to use.
        o  If you have trouble with some of the tasks, 
           it's the product's fault, not yours.  Don't feel 
           bad; that's exactly what we're looking for.
        o  If we can locate the trouble spots, then we 
           can go back and improve the product.
        o  Remember, we're testing the product, not you.

  3.  Tell the participant that it's okay to quit at 
      any time.

      Never leave this step out.  Make sure you inform 
      participants that they can quit at any time if they find 
      themselves becoming uncomfortable.  Participants shouldn't 
      feel like they're locked into completing tasks.  Say 
      something like this:

        "Although I don't know of any reason for this to 
        happen, if you should become uncomfortable or find 
        this test objectionable in any way, you are free to 
        quit at any time."

  4.  Talk about the equipment in the room.

      Explain the purpose of each piece of equipment (hardware, 
      software, video camera, microphones, etc.) and how it is 
      used in the test.

  5.  Explain how to "think aloud."

      Ask participants to think aloud during the observation, 
      saying what comes to mind as they work.  By listening to 
      participants think and plan, you can examine their 
      expectations for your product, as well as their intentions 
      and their problem solving strategies.  You'll find that 
      listening to users as they work provides you with an 
      enormous amount of useful information that you can get no 
      other way.

      Unfortunately, most people feel awkward or self-conscious 
      about thinking aloud.  Explain why you want participants to 
      think aloud, and demonstrate how to do it.  For example, 
      you could say:

        o  We have found that we get a great deal of 
           information from these informal tests if we ask 
           people to think aloud as they work through the 
           exercises.
        o  It may be a bit awkward at first, but it's 
           really very easy once you get used to it.
        o  All you have to do is speak your thoughts as 
           you work.
        o  If you forget to think aloud, I'll remind you 
           to keep talking.
        o  Would you like me to demonstrate?

  6.  Explain that you cannot provide help.

      It is very important that you allow participants to work 
      with your product without any interference or extra help.  
      This is the best way to see how people really interact with 
      the product.  For example, if you see a participant begin 
      to have difficulty and you immediately provide an answer, 
      you lose the most valuable information you can gain from 
      user observation--where users have trouble, and how they 
      figure out what to do.

      Of course, there may be situations where you have to step 
      in and provide assistance, but you should decide what those 
      situations might be before you begin testing.  For example, 
      you may decide that you can allow someone to flounder for 
      at least 3 minutes before you provide assistance.  Or, you 
      may decide that there is a distinct set of problems with 
      which you can provide help.

      As a rule of thumb, try not to give your test 
      participants any more information than the true users of 
      your product will have.  Following are some things you can 
      say to the participant:

        o  As you're working through the exercises, I 
           won't be able to provide help or answer 
           questions.  This is because we want to create 
           the most realistic situation possible.
        o  Even though I won't be able to answer your 
           questions, please ask them anyway.  It's very 
           important that I capture all your questions and 
           comments on tape.
        o  When you've finished all the exercises, I'll 
           answer any questions you still have.

  7.  Describe the tasks and introduce the product.

      Explain what the participant should do and in what order.  
      Give the participant written instructions for the tasks.

        Important:  If you need to demonstrate your 
        product before the user observation begins, be sure 
        you don't demonstrate something you're trying to 
        test.  (For example, if you want to know whether 
        users can figure out how to use certain tools, 
        don't show them how to use the tools before the 
        test.)

  8.  Ask if there are any questions before you start; 
      then begin the observation.

  9.  Conclude the observation.

      When the test is over:

        o  explain what you were trying to find out 
           during the test.
        o  answer any remaining questions the participant 
           may have.
        o  discuss any interesting behaviors you would 
           like the participant to explain.

 10.  Use the results.

      As you observe, you see users doing things you never 
      expect them to do.  When you see participants making 
      mistakes, your first instinct may be to blame the mistakes 
      on the participant's inexperience or lack of intelligence.  
      This is the wrong focus.  The purpose of observing users is 
      to see what parts of your product might be difficult or 
      ineffective.  Therefore, if you see a participant 
      struggling or making mistakes, you should attribute the 
      difficulties to faulty product design, not to the 
      participant.

      To get the most out of your test results, review all your 
      data carefully and thoroughly (notes, the video tape or 
      cassette tape, the tasks, etc.).  Look for places where 
      participants had trouble, and see if you can determine how 
      your product could be changed to alleviate the problems.  
      Look for patterns in the participants' behavior that might 
      tell you whether the product was understood correctly.

      It's a good idea to keep a record of what you found 
      during the test.  That way, you have documentation to 
      support your design decisions and you can see trends in 
      users' behavior.  After you've examined the results and 
      summarized the important findings, fix the problems you 
      found and test the product again.  By testing your product 
      more than once, you can see how your changes affect users' 
      performance.
