Field methods

• Involves data gathering from observations and

Observation

Interview

Survey

Ethnography

Contextual enquiry

Experiment ……

28-Apr-23 6. Evaluation 27
Author: C. B-A
 Experiment
• Provides evidence from data to support claims
(hypothesis)
• Can be used to evaluate different UI design issues
• Two main types:
– Absolute: specific UI on its own
– Comparative: compares two or more
• A formal method
– Usually in controlled environments (e.g., specialized
laboratories)
– Often referred to as controlled experiment
• 28-Apr-23
Basic assumption: cause and effect
6. Evaluation 28
 Experiment
• Procedure
– Identify problem or purpose
– Formulate hypothesis (i.e., claim or statement), e.g.,
• Colour coding improves accuracy of selection
– Design and conduct experiment to collect data
– Analyze data for evidence to accept or reject
hypothesis

28-Apr-23 6. Evaluation 29
 Experiment
• Experiment is designed around the hypothesis
– Independent variables (IV), i.e., controlled or
manipulated variables
– Dependent variable (DV), i.e., variable which changes
due to changes in IV
– Select sample of users and divide into groups
• Experiment group (experience the IV)
• Control group (does not experience IV)
• Note: groups can be
– Independent (e.g., two different groups of users) or
– The same group which (1) Does not experience IV and (2) Experience
IV
– Compare results
• Between groups
28-Apr-23 • Within groups 6. Evaluation 30
 Experiment
• Example
– Hypothesis: “Colour coding improves accurate selection”
– IV: Colour
– DV: accurate selection
– Experiment Design: Independent (Between) 2 groups
• Experiment group: provided with colour coding for selection
• Control group no colour coding
• Collect data as users make selections
• Experiment is designed around the hypothesis (e.g., number of
errors users make, etc.)
– Any other measuments
– Compare results (data analysis) for differences between the
groups
– Accept or reject the hypothesis
28-Apr-23 6. Evaluation 31
 Experiment
• Example (cont’d)
– Explain how the same experiment may be designed as
within group

28-Apr-23 6. Evaluation 32
 Experiment
• Advantages
– Collects quantitative, objective data
– Provides evidence to support claims
– Allows comparison of alternate designs
• Common problems
– Evidence may not be conclusive, e.g.,
• Results may be due to confounding variables, e.g., personal
circumstances (distraction, motivation, etc.)
– Variations, e.g., in human abilities; sample may not
represent entire users
– Ethical issues

28-Apr-23 6. Evaluation 33
 Analytics (Softwares)
• Software tools may be used for evaluation
• May be conducted locally or remotely
• Logs user activities to understand, e.g.,
– How UI is being used
– Users, e.g., behaviour, preferences, etc.
• Example 1: To understand how learners
interact with e-learning application, e.g.,
– To compare characteristics of learners who
complete and those who drop out

28-Apr-23 6. Evaluation 34
 Analytics (Softwares)
• Example 2: output from TechSmith’s Morae
– Measures mouse clicks for task 2 (selected in the tabbed
bar for Participant 4 (drop down list )
– Across the horizontal axis time is shown

28-Apr-23 6. Evaluation 35
 Physiological techniques: Eye tracking
• Eye movement tracking shows participants’
changing focus on different areas of screen
• Indicates which features of UI attracted attention
and in which order…
• Large scale gaze patterns show how people move
around the screen
• Can highlight which parts of a page are most
looked (hot spots)
• Technology
– You need a basic understanding of the hardware and
software
28-Apr-23 6. Evaluation 36
 Physiological techniques: Eye tracking
• Participant wears a helmet that monitors and records
where on the screen the participant is looking at.

28-Apr-23 6. Evaluation 37
 Physiological techniques: Eye tracking

More portable eye-tracking devices

28-Apr-23 6. Evaluation 38
 Physiological techniques: Eye tracking
• The heatmaps from the eyetracking study
- Red: area where the user looked most; yellow fewer; and blue the fewest;
gray not viewed;
- Image on the left “About us” section of a company web site; center: a
product page of e-commerce web site, and right search engine results page
(Jakob Nielson)

28-Apr-23 6. Evaluation 39
 Physiological techniques: Other
• Common changes which are measured
– Heart rate
– Perspiration
– Skin temperature
• All are effects of changes in arousal as a result of
emotional response
• Problems
– The emotion evoked cannot be identified

28-Apr-23 6. Evaluation 40
 Observation(Thinkaloud)
• A type of observations
• Self-study

28-Apr-23 6. Evaluation 41
 Summary
• We have covered the following:
– Factors to consider when selecting a method of evaluation
– Inspection techniques: Heuristic Evaluation and Cognitive
Walkthrough
– Experiments
– Physiological (eye tracking) and other techniques
– In each case, we have looked at the procedure, context of
use, advantage and common problems

28-Apr-23 6. Evaluation 44
 Questions and Feedback

28-Apr-23 6. Evaluation 45
 End of Session

28-Apr-23 6. Evaluation 46