Lesson09

Introducing Evaluation
• Evaluation Studies: From Controlled to Natural Settings
o Introduction
o Usability Testing
o Conducting Experiments
o Field Studies
• Evaluation: Inspections, Analytics, and Models
o introduction
o Inspections: Heuristic Evaluation and Walk-Throughs
o Analytics and A/B Testing

http://fpt.edu.vn 12/01/24 2
Objectives
The main aims of this chapter are to: Explain how to do
usability testing. Outline the basics of experimental
design. Describe how to do field studies.

http://fpt.edu.vn 12/01/24 3
1. Introduction
•Imagine that you have designed a new app to allow school
children ages 9 or 10 and their parents to share caring for
the class hamster over the school holidays. The app will
schedule which children are responsible for the hamster
and when, and it will also record when it is fed. The app
will also provide detailed instructions about when the
hamster is scheduled to go to another family and the
arrangements about when and where it will be handed
over. In addition, both teachers and parents will be able to
access the schedule and send and leave messages for
each other.
http://fpt.edu.vn 12/01/24 4
1. Introduction
How would you find out whether the children, their
teacher, and their parents can use the app effectively
and whether it is satisfying to use? What evaluation
methods would you employ? In this chapter, we describe
evaluation studies that take place in a spectrum of
settings, from controlled laboratories to natural settings.
Within this range we focus on the following:

http://fpt.edu.vn 12/01/24 5
1. Introduction
•Usability testing, which takes place in usability labs and
other controlled lab-like settings
• Experiments, which take place in research labs
• Field studies, which take place in natural settings, such
as people’s homes, schools, work, and leisure
environments

http://fpt.edu.vn 12/01/24 6
2. Usability Testing
The usability of products has traditionally been tested in
controlled laboratory settings. This approach emphasizes
how usable a product is. Initially, it was most commonly
used to evaluate desktop applications, such as websites,
word processors, and search tools. It is also important now,
however, to test the usability of apps and other digital
products. Performing usability testing in a laboratory, or in
a temporarily assigned controlled environment, enables
designers to control what users do and allows them to
control the environmental and social influences that might
impact the user’s performance.
http://fpt.edu.vn 12/01/24 7
2. Usability Testing
The goal is to test whether the product being developed is
usable by the intended user in order to achieve the tasks
for which it was designed and whether users are satisfied
with their experience. For some products, such as games,
designers will also want to know whether their product is
enjoyable and fun to use. (Chapter 1, “What is Interaction
Design,” discusses usability and user experience goals.)

Readmore (Wiley.Interaction.Design.Beyond.Human- Computer.Interaction.5th.Edition- page 525)

http://fpt.edu.vn 12/01/24 8
3. Conducting Experiments
In research contexts, specific hypotheses are tested that
make a prediction about the way users will perform with
an interface. The benefits are more rigor and confidence
that one interface feature is easier to understand or
faster to use than another. An example of a hypothesis is
that context menus (that is, menus that provide options
related to the context determined by the users’ previous
choices) are easier to select as compared to cascading
menus.

http://fpt.edu.vn 12/01/24 9
3. Conducting Experiments
Hypotheses are often based on a theory, such as Fitts’
Law, or previous research findings. Specific measurements
provide a way of testing the hypothesis. In the previous
example, the accuracy of selecting menu options could be
compared by counting the number of errors made by
participants when selecting from each menu type.

Readmore (Wiley.Interaction.Design.Beyond.Human- Computer.Interaction.5th.Edition- page 534)

http://fpt.edu.vn 12/01/24 10
4. Field Studies
Increasingly, more evaluation studies are being done in
natural settings with either little or no control imposed on
participants’ activities. This change is largely a response
to technologies being developed for use outside office
settings. For example, mobile, ambient, IoT, and other
technologies are now available for use in the home,
outdoors, and in public places. Typically, field studies are
conducted to evaluate these user experiences

http://fpt.edu.vn 12/01/24 11
4. Field Studies
Evaluations conducted in natural settings are very different from those
conducted in controlled environments, where tasks are set and completed
in an orderly way. In contrast, studies in natural settings tend to be messy
in the sense that activities often overlap and are constantly interrupted by
events that are not predicted or controlled such as phone calls, texts, rain if
the study is outside, and people coming and going. This follows the way
that people interact with products in their everyday messy worlds, which is
generally different from how they perform on fixed tasks in a laboratory
setting. Evaluating how people think about, interact with, and integrate
products within the settings in which they will ultimately be used, gives a
better sense of how successful the products will be in the real world. The
trade-off is that it is harder to test specific hypotheses about an interface
because many environmental factors that influence the interaction cannot
be controlled
http://fpt.edu.vn 12/01/24 12
4. Field Studies
Therefore, t is not possible to account, with the same
degree of certainty, for how people react to or use a
product as can be done in controlled settings like
laboratories. This makes it more difficult to determine
what causes a particular type of behavior or what is
problematic about the usability of a product. Instead,
qualitative accounts and descriptions of people’s
behavior and activities are obtained that reveal how they
used the product and reacted to its design

http://fpt.edu.vn 12/01/24 13
4. Field Studies
Field studies can range in time from just a few minutes to a
period of several months or even years. Data is collected
primarily by observing and interviewing people, such as by
collecting video, audio, field notes, and photos to record what
occurs in the chosen setting. In addition, participants may be
asked to fill out paper-based or electronic diaries, which run on
smartphones, tablets, or other handheld devices, at particular
points during the day. The kinds of reports that can be of interest
include being interrupted during an ongoing activity or when
they encounter a problem when interacting with a product or
when they are in a particular location, as well as how, when, and
if they return to the task that was interrupted.
http://fpt.edu.vn 12/01/24 14
4. Field Studies
This technique is based on the experience sampling method
(ESM), which is often used in healthcare (Price et al., 2018). Data
on the frequency and patterns of certain daily activities, such as
the monitoring of eating and drinking habits, or social
interactions like phone and face-to-face conversations, are often
recorded. Software running on the smartphones triggers
messages to study participants at certain intervals, requesting
them to answer questions or fill out dynamic forms and
checklists. These might include recording what they are doing,
what they are feeling like at a particular time, where they are, or
how many conversations they have had in the last hour

http://fpt.edu.vn 12/01/24 15
4. Field Studies
As in any kind of evaluation, when conducting a field study,
deciding whether to tell the people being observed, or asked to
record information, that they are being studied and how long
the study or session will last is more difficult than in a laboratory
situation. For example, when studying people’s interactions with
an ambient display, or the displays in a shopping mall described
earlier (Dalton et al. 2016), telling them that they are part of a
study will likely change the way they behave. Similarly, if people
are using an online street map while walking in a city, their
interactions may take only a few seconds, so informing them
that they are being studied would disrupt their behavior

http://fpt.edu.vn 12/01/24 16
4. Field Studies
. It is also important to ensure the privacy of participants in field
studies. For example, participants in field studies that run over a
period of weeks or months should be informed about the study
and asked to sign an informed consent form in the usual way, as
mentioned in Chapter 14. In studies that last for a long time,
such as those in people’s homes, the designers will need to
work out and agree with the participants what part of the
activity is to be recorded and how. For example, if the designers
want to set up cameras, they need to be situated unobtrusively,
and participants need to be informed in advance about where
the cameras will be and when they will be recording their
activities.
http://fpt.edu.vn 12/01/24 17
4. Field Studies
The designers will also need to work out in advance what to
do if the prototype or product breaks down. Can the
participants be instructed to fix the problem themselves, or
will the designers need to be called in? Security
arrangements will also need to be made if expensive or
precious equipment is being evaluated in a public place.
Other practical issues may also need to be considered
depending on the location, product being evaluated, and the
participants in the study

http://fpt.edu.vn 12/01/24 18
4. Field Studies
The study in which the Ethnobot (Tallyn et al., 2018) was used to
collect information about what users did and how they felt while
walking around at the Royal Highland Show in Scotland was an
example of a field study. A wide range of other studies have
explored how new technologies have been used and adopted by
people in their own cultures and settings. By adopted, we mean
how the participants use, integrate, and adapt the technology to
suit their needs, desires, and ways of living. The findings from
studies in natural settings are typically reported in the form of
vignettes, excerpts, critical incidents, patterns of behavior, and
narratives to show how the products are being used, adopted,
and integrated into their surroundings
http://fpt.edu.vn 12/01/24 19
Evaluation: Inspections, Analytics, and Models
Objectives
The main aims of this chapter are to: Describe the key
concepts associated with inspection methods. Explain
how to do heuristic evaluation and walkthroughs. Explain
the role of analytics in evaluation. Describe how to use
Fitts’ Law – a predictive model.

http://fpt.edu.vn 12/01/24 20
introduction
The evaluation methods described in this book so far
have involved interaction with, or direct observation of,
users. In this chapter, we introduce methods that are
based on understanding users through one of the
following:
oKnowledge codified in heuristics
oData collected remotely
oModels that predict users’ performance

http://fpt.edu.vn 12/01/24 21
None of these methods requires users to be present
during the evaluation. Inspection methods often involve a
researcher, sometimes known as an expert, role-playing
the users for whom the product is designed, analyzing
aspects of an interface, and identifying potential usability
problems. The most well-known methods are heuristic
evaluation and walkthroughs. Analytics involves user
interaction logging, and A/B testing is an experimental
method. Both analytics and A/B testing are usually
carried out remotely.

http://fpt.edu.vn 12/01/24 22
Predictive modeling involves analyzing the various physical
and mental operations that are needed to perform particular
tasks at the interface and operationalizing them as
quantitative measures. One of the most commonly used
predictive models is Fitts’ law.

Readmore (Wiley.Interaction.Design.Beyond.Human- Computer.Interaction.5th.Edition- page 550)

http://fpt.edu.vn 12/01/24 23
Inspections: Heuristic Evaluation and Walk-Throughs
Sometimes, it is not practical to involve users in an
evaluation because they are not available, there is
insufficient time, or it is difficult to find people. In such
circumstances, other people, often referred to as experts or
researchers, can provide feedback. These are people who are
knowledgeable about both interaction design and the needs
and typical behavior of users. Various inspection methods
were developed as alternatives to usability testing in the
early 1990s, drawing on software engineering practice where
code and other types of inspections are commonly used.

http://fpt.edu.vn 12/01/24 24
Inspections: Heuristic Evaluation and Walk-Throughs
Inspection methods for interaction design include heuristic
evaluations and walk-throughs, in which researchers
examine the interface of an interactive product, often role-
playing typical users, and suggest problems that users
would likely have when interacting with the product. One of
the attractions of these methods is that they can be used at
any stage of a design project. They can also be used to
complement user testing.

Readmore (Wiley.Interaction.Design.Beyond.Human- Computer.Interaction.5th.Edition- page 550)

http://fpt.edu.vn 12/01/24 25
Analytics and A/B Testing
A variety of users’ actions can be recorded by software
automatically, including key presses, mouse or other pointing
device movements, time spent searching a web page, looking at
help systems, and task flow through software modules. A key
advantage of logging activity automatically is that it is unobtrusive
provided the system’s performance is not affected, but it also
raises ethical concerns about observing participants if this is done
without their knowledge, as discussed in Chapter 10, “Data at
Scale.” Another advantage is that large volumes of data can be
logged automatically and then explored and analyzed using
visualization and other tools.

Readmore (Wiley.Interaction.Design.Beyond.Human- Computer.Interaction.5th.Edition- page 567)

http://fpt.edu.vn 12/01/24 26
Predictive Models
Like inspection methods and analytics, predictive models
can be used to evaluate a product without users being
present. Rather than user researchers being involved in role-
playing during inspections, or tracking their behavior using
analytics, predictive models use formulas to derive various
measures of user performance. Predictive modeling provides
estimates of the efficiency of different systems for various
kinds of tasks. For example, a smartphone designer may
choose to use a predictive model because it enables them to
determine accurately, which is the optimal sequence of keys
for performing a particular operation
Readmore (Wiley.Interaction.Design.Beyond.Human- Computer.Interaction.5th.Edition- page 576)

http://fpt.edu.vn 12/01/24 27