See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220383245

Liberating usability testing

Article  in  interactions · March 2007

DOI: 10.1145/1229863.1229864 · Source: DBLP

CITATIONS READS

25 1,166

1 author:

Phil Carter
Auckland University of Technology
26 PUBLICATIONS   101 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Phil Carter on 16 December 2013.

The user has requested enhancement of the downloaded file.

 liBeRAtinG
Usability Testing
Phil Carter | Auckland University of Technology | [email protected]

To find out if something is useful, have abstractions or distances away cognitive walkthroughs—are also
you use it and find out. There is from this object of study. distanced from the core focus of
movement between your use of the In a survey or questionnaire, the inquiry but are less likely to spill
thing and your thinking about it. the respondent is involved in a into reification because they do not
Remarkably, this everyday inquiry reflective process; that is, they are pretend otherwise.
within experience has not been high- thinking about an event (their use We might expect that the anthro-
lighted as the essence of usability of the artifact) that occurred in pological approach, which has the
testing. When it is, usability testing the past in a different situation. So studying of behavior within context
can be liberated from specialists there are abstractions in the dimen- as a distinguishing characteristic,
and restrictive procedures [3]. It is sions of time and place. There is an would highlight the mechanism of
humanized so it can be owned by abstraction in the primary focus of inquiry within experience. However,
anyone and used in a wide range of the study in that it is the question- people using this approach have
ways in response to the emergent naire, not the artifact. And there their research sensibilities attuned
needs of a software-development is an abstraction in the cognitive to the complex and fascinating
endeavor. In addition, when usabil- framework in that the respondent is socio-environmental contexts of
ity expands to include usefulness not relating to their own framework, use [5], not to the precision of a
and what works well, it becomes but to the questionnaire designer’s. person’s actual experience of using
much more generative. This article Table 1 lists these abstractions and an artifact.
seeks to build awareness and pre- also outlines the abstractions in Usability testing, however, aims
cision in our practice of usability other methods which are typically a direct inquiry into the experience
testing. used to evaluate usability. of using an artifact while the artifact
In a similar way to question- is being used. From the user’s point
Abstractions naires, the typical application of of view, they simply relate to their
Many methods can be used to interviewing does not deal directly own thinking and experience at the
investigate a person’s use of an with a person’s actual thinking, time of using the thing. As consum-
artifact. If we say that the person’s feeling, and actions when using ers with generations of shopping
actual experience of using an arti- an artifact. Two other common practice, we might also see this
fact at the time of using it is the methods used in usability evalu- inquiry as an everyday type of thing.
object of study, then most methods ations—heuristic evaluations and It is perhaps this marvellous sim-

Cognitive
Question- Inter- heuristic walk- usability
naire viewing evaluation through testing
Time No No Yes Yes Yes
Place No No No No No
Artifact (as
No No No Yes Yes
primary focus)
user Yes Yes No No Yes
user’s cognitive
No Yes No No Yes
framework
table 1: Inquiring into the actual use of an artifact: are these methods typically done to
the conditions of use?

:/ 18 i n t e r a c t i o n s     /     m a r c h + a p r i l 2 0 0 7

IA XIV-2.V20.indd 18 2/8/07 1:37:32 PM

 FEATURE>

plicity that has obscured the power process that most users will seek to gives a few for the moderator in
of the mechanism of directness, the satisfy. If they are primarily engaged usability testing: Don’t force opin-
heart of usability testing. in feeling or acting, they will switch ions; restate answers; follow up
attention. Even if they are thinking, with examples; use artifacts to
Expression Within Experience it will most likely require a shift to keep people focused on the pres-
There are differences between a finding reasons, a process that can ent and to trigger ideas; be aware
person’s own testing of a recently dominate attention. Table 2 outlines of your own expectations; never
purchased product and a usability the possible effects of some other say the participant is wrong; listen
test. In a usability test there is an types of interventions that can be carefully to the questions that are
expression of the experience of given in response to a test user’s asked of you; keep questions sim-
using an artifact, a description of frowning but saying nothing. ple in language and in intent; probe
that, and then a communication
to the developers of the artifact. Intervention user response
However, these activities can also Why are you frowning? Requests a cognitive process that may interfere
be built on everyday competencies, with the user’s experience if it is affective or kin-
aesthetic. Searching for reasons and rationale is a
and so effective practice can be constructive cognitive activity.
achieved by nonspecialists.
What are you thinking? The user’s experience may still be affective/kinaes-
Talk aloud or think aloud has thetic. Requesting a cognitive process may interfere
been usability’s main technique with this.
for a user to articulate their expe- What you are experiencing? Use of the word “experiencing” removes the threat
of not being in line with affective or cognitive focus
rience—particularly their think- of the test user; however, the question still requires
ing—to the others involved in the a cognitive process that may be premature.
testing. Nielsen et al [7] provide an What is it? What’s happening? More informal and more neutral than the previous
informative genealogy of this tech-
two statements, and so likely to give the user more
room to continue to use the system and give feed-
nique from psychology. However, back when ready.
this thing we are talking about, You’re frowning... Offers information from an external source that may
self-reflective consciousness, is at assist the user to increase their self-awareness.
However, this may also be presumptuous, and the
the core of what it is to be a human user may argue against it.
being. Every functional one of us
You’re wrinkling your brow. More observational and less processed than the
is competent. The ease with which previous statement. However, the user may feel
most people can immediately talk exposed.
aloud on their first usability test something’s up, eh? If said with emotive tone and force that’s similar
to what the user is displaying, this statement may
is evidence of this shared compe- assist the user in staying with their experience and
tence. feeling invited to express it in their own time.
Our experience over many years nothing... Something may eventuate. If not, and the moment
in a usability lab is that talk aloud
is lost and the user becomes involved with another
aspect of the system, then a pause may be used for
is very effective in exposing many reflection, or the user may be invited to back up.
of the factors at play within a user. table 2. Possible effects on user by different interventions
However, at times further clarifica-
tion is needed. For example, the test While the preceding analysis
user may be frowning, but there can wake us up to the precision
is no verbal expression relating to of language, it is not sufficient
that. Is it because of the color con- as a guideline for practice. There
trast on the screen, incomprehen- are many other factors that have
sion of a button icon, or indecision impact, such as the moderator’s
about navigation? Kuniavsky sug- tone, attitude, comfort level, and
gests the moderator could say, “You friendliness. Likewise, there is
frowned when that dialog box came the personality, attitude and capa-
up. Is there anything about it that bilities of the user. And so there
caused you to do that?” [6, p. 119]. is also the relationship between
What is the effect of a question such these two people. Guidelines can
as this? It introduces a cognitive quickly proliferate. Kuniavsky [6]

i n t e r a c t i o n s     /     m a r c h + a p r i l 2 0 0 7 :/ 1

IA XIV-2.V20.indd 19 2/8/07 1:37:34 PM

 expectations; ask why a lot; inves- the user, also having a good time, pursuing them, but unfold from the
tigate mistakes; probe nonverbal says, “This is fun, eh?” It’s an easy generation of a friendly, cooperative
cues; respect the evaluator’s ideas; opening for the user to keep con- working relationship.
and focus on personal experience. nected with their experience and Many guidelines on interview-
While these are all good guide- to elaborate on it in their own time. ing strongly emphasize that the
lines that may be integrated into a There is also a lot of space for the moderator be neutral. Seeking
moderator’s functioning with pro- moderator to keep connecting with neutrality will interfere with the
longed and repeated practice, they the emergent functioning of the moderator’s ability to be alongside
are too numerous for nonspecial- user. The subsequent interaction and be informed about the user.
ists to own and adopt them. can have the quality of a conversa- Worse, neutrality is impossible to
tion between friends. sustain for any length of time, and
Being Alongside Such friendly conversations are its pretence will destroy genuine-
Table 2 indicates that the inter- not dominated by safety concerns; ness and threaten the user. If, in
vention “Something’s up, eh? ” rather, they evoke an inquiry that contrast, a moderator can inculcate
results in a desirable outcome. is informed by all the dimensions in themselves a flexible movement
Such an intervention is not gener- human beings generate. The intel- between being open and naive and
ated from a focus on what to say lect can be informed by the affect having responses, then the user is
or adherence to a complex list of and the imagination, as well as more likely to feel secure, knowing
guidelines but emerges from a type what the eyes see and the brain they are dealing with a real human
of relationship between the user thinks. Friendliness is not some- being. The user will also be more
and the moderator where the mod- thing that requires an emotional likely to bring forward tentatively
erator is alongside the user. Being precursor to get going or emotional ask questions because they antici-
alongside is one way to describe the fusion to be sustained. Friendliness pate they will be treated in a genu-
attitude a moderator can adopt that is uplifting for the moderator and ine way.
will evoke effective interventions creates a sense of ease and enjoy- The relationship establishes a
such as this one. By being alongside, ment that stimulates intrigue and strong motivation for a truly ethical
the moderator’s own experience curiosity. Users benefit enormously approach that can have an ongo-
will inform her understanding of from naive inquiry because naivete ing attention to the well-being of
the user’s experience. They need destroys distrust and getting it right. all. There is no need to give the
not, therefore, ask so many ques- There is less anxiety to seek under- advice that a user often hears at
tions, but in combination with standing before experiencing and the beginning: It is not about you.
simple observations, they will be less pressure to fix anything up; all It is about them. It is about their
able to offer a comment that is gift- habits are grounded in culture and encounter with the artifact. When
like—it doesn’t demand anything in are accentuated by our technical they experience feeling dumb or
response. understanding. It appears that the vulnerable, do we then say it is not
For example, the user may be beneficial qualities of inquisitive- about them? This invalidation is not
smiling, clearly enjoying them- ness and openness tend not to useful. Instead, a cooperative inter-
selves, and the moderator alongside emerge when someone is actively est in the work can be established.
If right from the beginning, all par-
ticipants know that the common
purpose is to uncover and explore
the usability of this particular arti-
fact; then the attraction of discovery
becomes stronger, and the sense
of companionship in the journey
develops. Under such conditions,
the emotional need for safety ceases
to dominate and true safety is
established.
Much can be done in the pretest-
ing warm-up phase to establish a
cooperative working relationship.

:/ 0 i n t e r a c t i o n s     /     m a r c h + a p r i l 2 0 0 7

IA XIV-2.V20.indd 20 2/8/07 1:37:36 PM

 FEATURE>

Typically, the warm-up phase is thinking about usability; ambiguity the actual human value of an
seen as being for the benefit of the is uncovered and so can be dealt artifact becomes the core criteria
test user, but it is also a great oppor- with during the testing, removing on which usability inquiries are
tunity for the moderator to generate post-testing guesswork. grounded. For example, if learning
respect and gratitude for the test In terms of physical arrange- outcomes become integrated into
user. The test user has often come ment, participants can be flexibly the usability assessment criteria for
along out of a sense of generosity arranged according to each situ- an educational system, then clear
and wanting to contribute to the ation to facilitate this cooperative links can be made between the
work of making a better artifact. work. Often a triangle setup is use- elements of the system that sup-
When one person experiences grati- ful because everyone can see every- port or hinder learning. Likewise, if
tude, it is enormously generative one else. In this setup, the modera- the different interface features and
within a relationship. In addition, tor sits beside but slightly behind the interactions of an e-mail applica-
emphasize and clarify the value user, and the scribe sits across on tion are related to people’s ability to
of the artifact during the warm-up an angle. The scribe can have a sec- communicate and work, then deep
phase so that the whole enterprise ond monitor to make following the understanding results about the
can take on greater respect and user’s onscreen actions easy. interaction between the human and
dignity. A clear and explicit group com- the tool. “Usability is meaningless
mitment to the importance of unless it is usability for something
Descriptions for the Developers identifying the usability pluses and worthwhile.” [11, p. 58]. Without the
With the common everyday habit minuses encourages most test users focus on worthwhileness, what will
of inquiry within experience firmly to be authentic and to value and keep usability from being compro-
established as the functioning heart express both their joys and difficul- mised by compliance mentalities
of usability testing, there is also lots ties. Under such conditions, system and converted into checklists? The
of space for fresh ideas in response developers can be productively building of scenarios and personas
to the different challenges of soft- introduced into the room. This can and much of the good work grouped
ware-development projects. be very curative if developers have under user-centered design also
One area is how to create precise experienced—or even perceive— help us to put the artifact into its
and pertinent descriptions of usabil- usability as something being done proper living context. Once again,
ity that are immediately sensible to them. How much more produc- the heart of these approaches is
to developers and do not consume tive to have the intelligence and simple and humane: Get to know
precious time in post-testing analy- the design that options developers someone and appreciate the differ-
sis. With a cooperative working are weighing up, and the deep and ent aspects of their life.
relationship established, the logger extended thinking they are throw- Usability findings gain a much
or scribe can become more involved ing into the work to be part of the greater generative capacity when
and feed tentative descriptions back mix [4]. the focus is what works as well
to the user for clarification and fur- as what doesn’t. Extensions and
ther refinement. The describing task Usability as Usefulness enhancements to what works well
need not be a logger grinding out Great advantage occurs when can be precisely identified. Software
logs on their own. The descriptions
get their own airtime and “usability
test.” Users can be engaged in a co-
inquiry that can, at any time, return
to and be grounded in the actual
use of the artifact. The moderator
and scribe can pay particular atten-
tion to creating descriptions that are
precise and detailed enough for the
readers (system analysts, designers,
developers, programmers, project
managers) to understand. This
cooperative in-vivo process of cre-
ating clear descriptions can assist

i n t e r a c t i o n s     /     m a r c h + a p r i l 2 0 0 7 :/ 1

IA XIV-2.V20.indd 21 2/8/07 1:37:38 PM

 designers and developers get the put aside even the treasured maxim the usability inquiry and see it as
praise and appreciation they deserve of designing tools that are transpar- based on usefulness, then usability
and get the encouragement and ent to the task so that we can attend may offer even more than has been
motivation to continue. One should to the actuality. imagined [11]. We will surely need
be alert to the limitations that can everyone’s involvement and cre-
occur when a mindset or methodol- Wrap-Up ativity if we are to move forward in
ogy begins with problematic find- Sit alongside someone as they these times.
ings. However, escape can occur use something, and find out how it’s referenCes 1. Eno, B. (1999). Revenge
through expansion; we ask, what is going: That’s usability testing. It’s of the Intuitive. Wired, 7(1). 2. Gladwell,
M. (2005). Blink: The Power of Thinking
the human value of the artifact we that simple. When we appreciate the
without Thinking. Victoria, Australia: Allen
are involved in designing and devel- everyday, common-sense essence Lane. 3. Holzinger, A. (2005). Usability
oping? When we see meaning and of usability testing, we free our- engineering methods for software devel-
opers. Communications of the ACM, 48
value in our work, we invest more selves from the specific mandates of
(1), 71-74. 4. Hornbaek, K. & Frokjaer, E.
imagination and careful thinking. different methods [9]. There is room (2005). Comparing usability problems and
We will not be so easily governed by for us to attend to what is actually redesign proposals as input to practical
systems development. In Proceedings of
compliance or committee, but we happening. We avoid the narrow-
CHI 2005. Portland, Oregan, USA (April 2-
will be freed up no matter what the ness of thinking we are dealing with 7), 391-400. 5. Kantner, L., Sova, D. H., &
external structures. a process of extraction—we are Rosenbaum, S. (2003). Alternative methods
for field usability research. In Proceedings
Freed up, we will not preempt in a human process. We bring the
of SIGDOC’03. San Francisco, California,
our judgements or get narrowed human qualities of inquisitiveness USA (12-15 October), 68-72. 6. Kuniasky,
in our thinking. History gives us and friendliness because they are M. (2003). Observing the User Experience:
A Practitioner’s Guide to User Research. San
many examples that might help most effective. Human beings with
Francisco: Morgan Kaufmann. 7. Nielsen,
us become humble in the knowl- life experience become the experts J., Clemmensen, T., & Yssing, C. (2002).
edge that we don’t know what we [8]. No group need get precious or Getting access to what goes on in people’s
heads? - Reflections on the think-aloud
don’t know. Test showings of pilot struggle to own usability [10]. It can
technique. In Proceedings of NordiCHI
episodes of All in the Family and be taken up and integrated into any 10/02. Arthu, Denmark (October), 101-
the Mary Tyler Moore Show scored framework. 110. 8. Norman, D. A. (2005). Whose
profession is this? Everybody’s, nobody’s.
low in terms of likability, and both Do not underestimate the engine
Interactions, May/June, 51. 9. Seffah, A.
shows were nearly dumped before of usability testing just because it is & Metzker, E. (2004). The obstacles and
they got going [2]. New was shock- simple. Own it, and overly control- myths of usability and software engineer-
ing. Communications of the ACM, 47 (12),
ing, and not altogether welcome. In ling and fear-motivated forces will
71-76. 10. Siegal, D. & Dray, S. (2005).
another fascinating example, Brian not so easily get purchase and colo- Avoiding the next schism: ethnography
Eno illustrates that the relationship nize usability for their restrictive and usability. Interactions, March/April,
58-61. 11. Spool, J. & Schaffer, E. M.
with an artifact and its restrictions purposes. See the simple nobility of
(2005). In Proceedings of CHI 2005.
can be highly desired and even this inquiry. Do one thing at a time Portland, Oregon, USA (April 2-7), 1174-
mandatory to creativity [1]; we can and notice that even one little piece 1175.

of cooperative work ripples influ-

ence throughout an organization. about tHe autHor

Do not defer to academic structures Phil Carter began working

just because they appear more in computing by operating

sophisticated. Perhaps it will be the a HP 3000 in the early

direct and grounded inquiries at the 1980s. He then lived in Taiwan for seven

heart of usability testing that will years before returning to New Zealand to

provide the missing link between study computer and cognitive science.

the design and natural sciences that His PhD focused on Expert Systems and

continues to be elusive. When we Psychodrama. He is also a certified

own the common human core of Psychodramatist. He has setup and been
director of a usability research centre and
is currently teaching usability to comput-
ing and business students at the
Auckland University of Technology.

:/  i n t e r a c t i o n s     /     m a r c h + a p r i l 2 0 0 7

IA XIV-2.V20.indd 22 2/8/07 1:37:40 PM

View publication stats