Marion Koelle, Patrick Lindemann, Tobias Stockinger, and Matthias Kranz (Editors)

Human-Computer
Interaction with
Augmented Reality
Advances in Embedded Interactive Systems
Technical Report – Summer 2014

Volume 2, Issue 4. ISSN: 2198-9494

 Human-Computer Interaction with Augmented
Reality

Marion Koelle, Patrick Lindemann, Tobias Stockinger, and Matthias Kranz

October 2014

2
Contents
Preface 4

A brief Introduction to Augmented Reality 5

Prisca Bonnet, Pierre Ducher, and Alexander Kubiak

Evaluation of Augmented Reality in the Field 7

Alexander Kubiak

Evaluation of Augmented Reality Applications in a Laboratory

Context 15
Prisca Bonnet

Interaction with Augmented Reality 23

Pierre Ducher

Copyright Notes 30

3
Preface
Augmented Reality (AR) enhances the user’s view of the world with computer-
generated information. While the research field of Augmented Reality recently
made great progress in terms of tracking accuracy and rendering quality, human
factors often play only a minor role in the development process of Augmented
Reality applications. Nevertheless, there is an increasing number of developers
that create AR applications. We try to satisfy their needs for design guide-
lines, recommendations and best practices by providing spotlights on selected
applications and highlighting the relevant recommendations.

This technical report gives a brief introduction to Augmented Reality, and

provides an overview of user-centered research methods that are available to
Augmented Reality development as well as interaction methods and technologies.

During the summer term in 2014, the Embedded Interactive Systems Laboratory
at the University of Passau encouraged students to conduct research on the
general topic of “Human-Computer Interaction with Augmented Reality”. Each
student analyzed a number of scientific publications and summarized the findings
in a paper.

Thus, each chapter within this technical report depicts a survey of specific aspects
of a topic in the area of human factors in Augmented Reality.

Passau, October 2014

The Editors

Marion Koelle, Patrick Lindemann, Tobias Stockinger, and Matthias Kranz

4
 A Brief Introduction to Augmented Reality

Prisca Bonnet Pierre Ducher Alexander Kubiak

Universität Passau Universität Passau Universität Passau
[email protected] [email protected] [email protected]
passau.de passau.de passau.de

ABSTRACT in 3D. In contrast to other definitions this allows AR systems

In this report, we briefly introduce Augmented Reality (AR). to use other technologies besides Head-Mounted Displays [1].
Besides presenting the definitions of AR by Azuma and While Furht and Carmigniani define AR as enhancing the
Milgram, this paper lists challenges introduced by human real environment by adding virtual information [4], Azuma’s
factors like discoverability or interpretability. Furthermore, definition also includes applications that remove real objects
technologies for AR are examined, presenting different AR from the environment.
devices and technical challenges. Already three years earlier, Milgram and Kishino defined
a Reality-Virtuality Continuum spanning between the real
and virtual environment creating a Mixed Reality in [7] as
Keywords shown in Fig. 1.
Augmented Reality, AR, Mixed Reality
Mixed Reality (MR)
1. INTRODUCTION
Today more than ever people have a lot of information Real Augmented Augmented Virtual
Environment Reality (AR) Virtuality (AV) Environment
at their disposal. Presenting these data to the user is not
an easy task, one must be careful not to overwhelm him
with them. Augmented Reality proposes a way to mix in- Figure 1: Milgram’s Virtuality Continuum based on
formation with the real environment world by using differ- the definition in [7]
ent techniques. By blending data with a live view of the
real world, we can better integrate them and make it feel As shown in Fig. 1, Mixed Reality also contains the Aug-
more natural for the user. To embed information into the mented Virtuality (AV). While AR enhances the real envi-
real environment, the most accessible and used solution is ronment by adding virtual objects, AV adds some real ob-
using a smartphone and its camera. However this is only one jects to a virtual environment [7]. Figure 2 shows an exam-
possibility. Other scenarios might require different means to ple of an AR application meeting Azuma’s definition. The
display information. There are two categories: mobile and application combines the real world, registered in 3D with
stationary AR systems. For mobile systems, the device can virtual addons and reacts to interactions in real time.
be worn or hand held. Stationary systems usually have a
surface, like a table, to interact with them.
We will have a more formal definition of Augmented Re-
ality in the next part, followed by the state of the art in
part three. Finally, we will discuss the human factors in the
conclusion.

2. DEFINITION
In 1997, Azuma defined Augmented Reality (AR) in his
survey [1]: A system delivers AR if following characteristics
are satisfied: The real environment is combined with a vir-
tual one, the system is interactive in real time and registers

• Prisca Bonnet, Pierre Ducher, and Alexander Kubiak

Figure 2: Example of an AR application [8]
are master’s students at the University of Passau, Ger-
many
When it comes to developing mobile AR applications,
• This research report was written for Advances in Em- there are not only the definitions of Azuma and Milgram
bedded Interactive Systems (2014), Volume 2, Issue 4 to be considered. Also, human factors present the follow-
(October 2014). ISSN: 2198-9494 ing challenges [6]: The first challenge is discoverability as
many users may not be aware of what services are avail-
able. The application needs to assist the user by showing normally don’t speak to computers. For this reason we need
e.g. hints. The next challenge is the interpretability of the better interfaces increasing the need for more sensors, so
virtual objects by having clear meanings. Another impor- that we can interact more naturally with computers. To ful-
tant challenge is the usability by providing easy learnability fil this latter goal, devices may have an integrated camera
and usage of the application. Also, the usefulness of an ap- potentially filming what is in front of the user. This is also a
plication is important, meaning the application offers a long concern of the citizens. They are not keen to have this kind
term value [6]. An example of usefulness would be the use of technology being used around them, especially because
of AR for navigation instead of advertisement. of privacy issues. All of these questions will need answers
before Augmented Reality becomes more popular.
3. TECHNOLOGIES FOR AR
Augmented Reality can be applied to a variety of domains 5. REFERENCES
is a vast domain, using different types of devices. In this [1] R. Azuma. A survey of augmented reality. Presence,
section, we will describe the different devices and methods 6(4):355–385, 1997.
mostly used in AR, and the technical challenges they raise. [2] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier,
and B. MacIntyre. Recent advances in augmented
3.1 Devices in AR reality. Computer Graphics and Applications, IEEE,
We can distinguish two main groups of devices used in 21(6):34–47, 2001.
Augmented Reality applications. First, there are fixed de- [3] O. Bimber and R. Raskar. Spatial augmented reality.
vices, that can be used in museum or public places. They Peters, 2005.
can consist of screens, pedestal-mounted binoculars or pro- [4] B. Furht, editor. Handbook of Augmented Reality.
jectors displaying images directly on physical objects [3]. Springer, 2011.
Second, mobile devices can be carried by the user. The most [5] Y. Genc, S. Riedel, F. Souvannavong, C. Akinlar, and
widespread devices are hand-held devices such as smart- N. Navab. Marker-less tracking for ar: A learning-based
phones and tablets, but head-mounted displays are getting approach. In Mixed and Augmented Reality, 2002.
more and more popular, with the commercialization of data- ISMAR 2002. Proceedings. International Symposium
glasses such as Google’s Glass. on, pages 295–304. IEEE, 2002.
In AR, devices can be either optical see-through or video [6] W. Huang, L. Alem, and M. A. Livingston, editors.
see-through [2]. Optical see-through devices consist of a Human Factors in Augmented Reality Environments.
semi-transparent screen, on which information is overlaid Springer, 2013.
to appear as if it was part of the real world. Dataglasses [7] P. Milgram and F. Kishino. A taxonomy of mixed
or fixed transparent screens usually are optical see-through. reality visual displays. IEICE TRANSACTIONS on
Video see-through devices capture the scene using one or Information and Systems, 77(12):1321–1329, 1994.
several cameras, add virtual information on the recorded
[8] Wikimedia Commons. Augmented GeoTravel.
image, and display it to the user. Most hand-held devices
http://commons.wikimedia.org/wiki/File:
are video see-through.
Augmented_GeoTravel.jpg, 2010. (accessed August 28,
3.2 Challenges in AR 2014).
Depending on the type of device, AR applications are fac-
ing several technical challenges [4]. Most applications using
mobile devices are based on computer vision techniques, in
order to analyze a scene before adding corresponding virtual
information. Computer vision is a vast and complex field,
and the algorithms still have to be improved. Besides, the
algorithms used in AR have to be very effective, in order
to minimize the latency. AR applications often use one or
more tracking technologies to determine the user’s absolute
position, or his relative position to an object. To this end,
the application can use GPS data, marker-based or marker-
less tracking [5]. AR applications often rely on other data,
such as compass, gyroscope or voice commands. The quality
of a given application will depend on the precision of those
sensors, which can cause accuracy and latency issues.

4. CONCLUSION
We have seen that with the rise of better and more ef-
ficient technology, Augmented Reality is now possible. It
could change and improve the everyday life of everyone [4].
Nevertheless, wearing a computer equipped with sensors all
day can rise some concerns. To achieve good information
quality, we need to share some personal information with
the device, thus causing privacy issues. Also, communica-
tion with the device may seem unnatural to most as we
 Evaluation of Augmented Reality in the Field

Alexander Kubiak
Universität Passau
[email protected]

ABSTRACT VW and EADS invest in big projects like ARVIKA [5, 10]
While there already exist several surveys analyzing the eval- and AVILUS [28] to use AR in the manufacturing process.
uation of usability of Augmented Reality (AR) applications, This shows that AR is past being of interest for research ex-
most reviewed papers did not conduct any field studies. clusively. Not only the industry invests in AR. The military
Therefor, this work reviews papers in the context of AR also starts using AR for their purposes [11, 12, 13]. Those
that conducted field evaluations. With usability becoming projects do not only help advancing the technology used in
the central goal in the design process, field studies become AR, by using user-driven development and new methods like
more important to evaluate the user experience under real Wizard of Oz and field evaluation, they also set examples
conditions. To integrate evaluation in the design process, how to create good AR systems.
new techniques will be required. This paper also lists meth-
ods describing how to make usability the central goal in The paper is structured as follows: Chapter 2 lists re-
developing AR applications. lated work evaluating usability of AR systems and compar-
ing laboratory vs. field studies. Chapter 3 lists techniques
how evaluation can be integrated into the design process of
Keywords AR applications by applying traditional HCI principles or
Augmented Reality, AR, Field Study, User Experience, Us- by using new approaches like the Wizard of Oz. In Chapter
ability, Evaluation 4, papers conducting field studies will be reviewed. Finally,
Chapter 5 presents a summary of the topic.
1. INTRODUCTION
With modern technology, AR is becoming more popular. 2. RELATED WORK
As it is still a considerably new domain, especially for the In this section, related work will be listed. It is divided
users, research about user experience is still in its infancy into two parts. In Section 2.1, surveys analyzing the usabil-
[24]. Nevertheless, to make AR applications more attractive, ity evaluation in different papers will be described. Section
user experience should be a central goal and design strategy 2.2 reviews papers comparing usability evaluation in the lab-
[24]. To identify usability problems early in the develop- oratory vs. in the field and arguing about the importance
ment, user testing is required. Studies show that usability of field studies.
evaluation is rising [3, 8]. However, most usability evalua-
tions are conducted in the laboratory [7, 24]. 2.1 Surveys
Dünser et al. [8] published a survey of user evaluation
While testing the performance and system functionality techniques used in AR. They reviewed all papers published
in the laboratory is good practice, evaluating the usability between 1993 and 2007 evaluating AR applications. Over-
and user experience should be conducted in the field. Even all, they identified 165 papers which were grouped in the
though analyzing the usability in the laboratory is less time following categories: perception, user task performance and
consuming and more cost-effective [15, 22], more usability collaboration. Furthermore, the papers were categorized ac-
problems are found in the field [7, 22, 27]. An important cording to the user study approaches. However, most papers
factor is the increased cognitive load in the field compared mentioned did not conduct field studies.
to the laboratory [7, 22, 23] which cannot be simulated.
Bai and Blackwell [3] review usability evaluations of pa-
Fortunately, field studies become more popular as big pers published in ISMAR between 2001 and 2010. They
companies start investing into AR technology. Especially identified 71 papers which were grouped in the categories
task performance, perception and cognition, collaboration
• Alexander Kubiak is a master’s student at the Univer- and finally user experience. The categories were adopted
sity of Passau, Germany from [8] and extended by user experience. For each category,
the authors analyzed common metrics used to measure the
• This research report was written for Advances in Em- category’s goal. Also, the authors analyzed the evaluation
bedded Interactive Systems (2014), Volume 2, Issue 4 approaches and challenges. Similar to [8], most papers men-
(October 2014). ISSN: 2198-9494 tioned did not conduct any field studies. Still, this paper
is the basis for Chapter 4 as the evaluation approaches are
also grouped in the suggested categories. recordings, the field study reveals a lot more usability prob-
lems than the laboratory evaluation.
Olson and Salo present experiences of mobile AR applica-
tions in [24]. Conducting an online survey, they explored the The biggest difference found between laboratory and field
user experience of applications available in 2011. Users gave testing was presented by Duh et al. [7]. They conducted an
mostly narrative feedback about their experiences, for exam- evaluation of the common use of a mobile phone, like calling
ple when using Google Goggles which was renamed Google or messaging other people. With 171 problems found, the
Glass. While the paper does not present other work about field test revealed nearly twice as many usability problems
evaluating user experience, it summarizes common experi- as the laboratory test with 92 problems. When it comes to
ences and discusses design considerations for future work. critical problems which prevent participants from complet-
Also, the paper lists different measurements for user expe- ing tasks, the difference was even more severe: Field testing
rience and states that user experience is becoming a central revealed 64 critical problems while testing in the lab only
goal and design strategy. found 12. Aside from higher cognitive load as argued above,
users also stated the lack of privacy and the noise level in the
2.2 Lab vs. Field field as possible reasons for the different amount of revealed
usability problems.
To analyze the use of usability testing in the field, several
papers discussing evaluation in the laboratory vs. in the
Concluding can be said that field studies usually reveal
field were published. Kjeldskov et al. [15] present a mobile
more usability problems, especially problems caused by in-
system to support planning and conducting work tasks in
creased cognitive demands that are very hard to be simu-
a hospital. The system was evaluated in both the labora-
lated in a laboratory. However, the evaluation in the field is
tory and the field. The authors argue that a study in the
more expensive and time consuming as recording the partici-
laboratory is less time consuming and more cost-effective as
pants for analysis after the testing is difficult. Also, common
transportation cost do not exist and observation of the par-
practices used in the laboratory as thinking aloud might be
ticipants is a lot easier. Cameras can be mounted to the
difficult to apply in the field as there are many other people
ceilings allowing different angles of view. Also, the partici-
nearby and there is more noise. Furthermore the equipment
pants can focus more on the system instead of other tasks
has to be transported to the field which also increases the
that arise in the hospital. The authors conclude that lab-
cost and time usage.
oratory testing is cheaper, easier and finds more usability
problems than field testing.
3. EVALUATION IN THE DESIGN PROCESS
In contrast, Nielsen et al. [22] and Rogers et al. [27] both As user experience and usability become the central goal
conclude that field testing is ”worth the hassle”, meaning in AR applications [24], it is important to model them as
that even though it is more expensive and time consuming, early as in the design process. While user experience in
field studies reveal more usability problems. While [22] is a AR is still in its infancy [24], design goals for HCI exist.
direct reply to [15] by evaluating and discussing the results Dünser et al. [9] identify fundamental differences between
of both laboratory and field studies, [27] argues in a more traditional GUI systems and AR systems: GUI guidelines
informal way. assume that the user uses mouse, keyboard and a screen to
interact with the system. However, these typically do not
Nielsen et al. [22] compare the two approaches by evalu- exist in AR. There are numerous other possible interactions
ating a mobile system which is usually used by skilled work- between the user and an AR application. Also, while tradi-
ers to register the use of equipment, materials, mileage and tional HCI guidelines for interfaces work in 2D, AR registers
time. The system consists of a mobile phone and a scanner and often displays information in 3D [1].
attached to the phone. As mentioned before, the authors
argue that testing in the field found more usability prob- Lacking guidelines for user interfaces in AR, applications
lems than testing in the laboratory. While the laboratory are often developed using rapid prototyping to allow eval-
evaluation revealed 48 problems, field testing detected 60. uation to be a recursive activity during the entire design
An important remark is that only field evaluation was able cycle [2]. One approach is using the Wizard of Oz. After
to identify usability problems related to cognitive load and discussing the use of HCI techniques and principles in AR
interaction style. This goes well with Nunes and Recarte in Section 3.1 and showing a model for usability engineer-
[23]: In their paper, they state that there exists an inverse ing applied to AR in Section 3.2, the Wizard of Oz will be
relationship between cognitively demanding activities and introduced in Section 3.3.
visual processing capacities.
3.1 HCI Techniques & Principles in AR
Similar to [22], the authors of [27] also argue that field As mentioned above, there exist many HCI guidelines for
testing reveales more usability problems. Rogers et al. [27] GUI user interfaces. However, these cannot easily be ap-
present a system for scientists and students to observe the plied to AR as information is registered and displayed in
growth of a freshly planted forest. While the laboratory 3D [1] and AR incorporates other means of interaction [9].
testing, conducted by environmental scientists, could not Several papers propose using HCI techniques in AR. Dünser
reveal usability problems, the field study performed by stu- et al. [9] introduce the following common design principles
dents found several problems using the system. As in [22], for AR allowing the development of applications with better
a large factor was the increased cognitive load in the field. usability and user experience:
The authors conclude that despite the difficulty of collect-
ing observations for analysis, for example in form of video Affordance to make the application have an inherent con-
 nection between a user interface and its functions. This Rapid Prototyping to implement user interaction designs.
can be achieved by providing a model describing subject-
object relationships. Usability Evaluation to evaluate the usability.
Reducing cognitive overhead to allow the focus on the Based on an application for the military, the authors fo-
actual task instead of overwhelming the user with in- cused on the domain analysis to show the difficulties of de-
formation resulting in poor user experience. veloping good AR applications. They state that user profiles
Low physical effort to make a task accomplishable with sometimes are generated using surveys. However, with new
a minimum of interaction steps. Applications with in- AR applications the actual end-use population may not be
teraction that is too complex will not be successful. easily definable or reachable [11]. They state that a user
profile should include information about a user, such as the
Learnability to provide easy learnability for a user. As amount of computing or AR experience.
AR provides novel interaction techniques, the usage of By studying military books and manuals the authors iden-
those techniques has to be easy to learn. tified around 100 tasks that could be enhanced by using AR.
For each task they analyzed how the AR system could sup-
Responsiveness to guarantee good performance of the ap- port perceptual and cognitive demands. As they identified
plication as users only tolerate a certain amount of a list of features that cannot be delivered by existing AR
system lag [35]. systems, the authors want to conduct more research before
Error tolerance to deliver stable applications. As AR sys- continuing the development. This shows that the process
tems are mostly in early development stages there ex- of usability engineering identified problems that have to be
ist many bugs. Applications should be able to continue solved first instead of creating prototypes that will not de-
working even when experiencing an error. liver satisfactory usability.

While above principles are just some desirable design prin- 3.3 Wizard of Oz
ciples to be integrated into the design process of an AR sys- To evaluate the user experience and usability of a system,
tem, Swan and Gabbard [30] discuss the usage of traditional rapid prototyping is helpful as design flaws are revealed early
HCI methods in AR. In their opinion, methods like domain in the development. One promising approach is the usage of
analysis, user needs, task analysis and use case development Wizard of Oz. In this approach, a wizard usually simulates
can easily be applied to AR. However, those methods only either some part of the model or the complete envisioned in-
determine what information need to be presented to users, terface [6]. Therefor, a first prototype can be evaluated very
not how these information should be displayed. early in the design process, even when it is not clear what the
underlying technology should be [6]. The wizard then works
3.2 Usability Engineering in AR as controller by simulating unbuilt system components.
Gabbard et al. [11] describe how to use usability engineer- Later in the development the wizard can also be used as
ing for AR systems. This allows the reduction of usability supervisor controlling the system, still able to override deci-
problems already in the design process. They propose an sions made by either the system or the user. The wizard can
iterative process consisting of the following phases as shown be a moderator as well which lies between the controller and
in Figure 1. supervisor. This role is used when a component is already
working but not trusted [6]. The advantage of the modera-
tor is that it can override the component’s output before it
1. Domain reaches the rest of the system.
Analysis 2. User- The Wizard of Oz is often applied by using persons as
Centered “wizards”, triggering events with a remote or a controller
Requirements 3. Conceptual
application when the participant arrives at some point or
User Interaction
Design activates an event. This approach is used in [6] and [20].
4. Rapid
Prototyping
5. Usability Even though the wizard helps conducting more realistic
Engineering evaluations, the awareness of it or incorrect behavior can
corrupt an evaluation and compromise the results [19]. How-
Analysis & Design Development Evaluation ever, as user experience is becoming the central goal of the
design process [24], early evaluations are very important.
Figure 1: Usability Enginnering as proposed in [11] Without the usage of the wizard, the system, early in its
development, might behave unexpectedly and contain many
As it can be seen in Figure 1, the model consists of five bugs. Therefor, the Wizard of Oz is a great method for
phases that can easily be re-iterated. The following describes developing AR applications as its usage allows very early
the purpose of each phase: usability evaluations.

Domain Analysis to specify the system domain by ana- Examples

lyzing which users attempt to perform which tasks. Dow et al. [6] use this approach to develop a system for cre-
User-Centered Requirements to specify the user inter- ating a mixed reality on a cemetery by using location-based
action with the system. audio recordings. They showed how well the Wizard of Oz
approach works with the iterative design as they developed
User Interaction Design to design user interactions. and implemented the system in three stages. While the first
stage used the wizard as controller, the second stage only re- As most of the papers cited in [3] and [8] did not conduct
quired a moderator whereas the last stage used the wizard any field studies, the following sections describe the results
just as supervisor. found by Bai and Blackwell in [3] as well as results of an
The authors used the Wizard of Oz approach to perform own research in the context of AR and field evaluation.
early evaluation of the content the users could listen to as
well as of the interface. During the iterations, they did not 4.1 Task Performance
only improve the Mixed Reality system but also the inter-
This section describes papers evaluating task performance.
face for the Wizard of Oz controller.
As seen in [3], the task performance is always measured in
time. Usually the time to complete a task is measured. An-
Friedrich [10] introduces ARVIKA, a large project to use
other metric for the performance is accuracy, which is used
AR for development, production and service in the industry.
in 63% of the papers according to [3]. Furthermore, the er-
The pilot project, sponsored by the German ministry of Ed-
ror rate is used to measure the performance, as a system
ucation and Research and big companies like VW, EADS,
works better if the user makes less errors. This metric is
Siemens and others, also used the Wizard of Oz to include
interchangeable with accuracy which increases if the error
concepts in the tests that have not been implemented yet.
rate decreases.
Therefor, test persons have been able to evaluate the system
Most of the papers listed in [3] did not conduct field
very early in the development.
evaluations. Listed below are papers cited in [3] that per-
Möller et al. [20] present different user interfaces for in- formed field studies as well as the results of an own research
door navigation using a mobile phone. They implemented which evaluated the user task performance by conducting
AR as well as Virtual Reality to evaluate the user expe- field studies.
rience. The navigation mechanism was also implemented
using the Wizard of Oz approach. In contrast to above sys- Schwerdtfeger et al. [29] introduce an AR system to as-
tems, the wizard was used to modify the accuracy of a user’s sist workers in a warehouse. Using head-mounted displays,
position as well as guaranteeing comparable conditions for the worker will be guided to the correct box and shown the
all participants. To do so, a Wizard of Oz app was built to amount of its content to be picked. The AR system was
control the instructions shown on the participants’ devices. compared to the common paper approach. When evaluat-
Also, the control application sent location information to ing the field study, the authors used error rate and picking
the participant’s device. Each participant was followed by a time as measurements for performance. To observe the par-
researcher operating the control application. ticipants, they used a special observer who was placed on
an observation deck above the experimentation area. The
observer analyzed each step the participant did.
4. EVALUATION IN THE FIELD Besides measuring performance, the field test also evalu-
Evaluating AR in the field is still not very common. Kjeld- ated the subjective strain using the NASA-TLX test and the
skov et al. [15] state that field evaluation is very expensive user experience by questionnaire which will be described in
and time consuming. Even though the actual evaluation Section 4.2 and Section 4.4.
sessions took less time in the field, the overhead in plan-
ning and transporting requires many man-hours, resulting Similar to [29], Henderson and Feiner [12, 13] also used
in higher costs. Also, recording is difficult in the field as completion time and error rate to evaluate their AR system
cameras cannot be mounted to walls or ceilings. for task localization in maintenance of an armored personnel
Most of the papers conducting field studies mainly eval- carrier turret. The system was evaluated in field conditions
uate usability and user experience. The success of an AR with six participants. The evaluation purpose was not only
application depends a lot on the user satisfaction. Therefor, to evaluate the AR system but also to compare it to a system
user experience is not only a central goal, it is also a design only using a LCD display showing the next steps. Besides
strategy [24]. This was already discussed in Chapter 3. Ols- completion time and error rate, the authors also used local-
son and Salo [24] state that user experience research is still ization time analysis to measure the performance. However,
in its infancy. they did not describe how the observations were made.
In addition to measuring the task performance, the field
Bai and Blackwell [3] and Dünser et al. [8] both catego- study also evaluated the user experience using a formal ques-
rize published papers in terms of their evaluation. In both, tionnaire. This will be described in Section 4.4.
papers that conduct usability evaluation have been grouped
according to categories introduced by Swan and Gabbard in Kjeldskov and Stage [16] also used completion time to
[30]. In [3], these categories are extended by user experience. evaluate task performance. Instead of error rate, they used
The following list describes the categories adopted from [3] accuracy which can be interpreted the same way. The au-
that are also used in this paper. thors introduce new techniques to evaluate the usability of
mobile systems. They conducted two experiments consid-
Task Performance to evaluate the user accomplishments ering mobile messaging on phones. The goal was to com-
when using the system. pare different scenarios like walking on a treadmill or walk-
ing outside next to other pedestrians. To identify usability
Perception and Cognition to understand how perception
problems, evaluations were initially recorded on video to be
and cognition work in AR.
analyzed by experts afterwards. As the authors found col-
Collaboration to evaluate the interaction of multiple users. lecting high-quality video data in the field to be very diffi-
cult, they only recorded audio for the second experiment and
User experience to evaluate the user’s subjective feelings. took written notes. This resulted in a lack of detailed data.
The authors state that because of recording the participant’s work load indicates that the perception of the application is
actions, it was difficult to experience realistic pedestrian mo- reduced when the user is moving.
tion as other pedestrians tended to move away [16].
Besides measuring task performance the authors also an- In contrast to [16] and [29], Henderson and Feiner [12, 13]
alyzed the work load which will be described in Section 4.2. did not use the work load to evaluate cognition. Instead,
they measured the head movement. In their paper which
Completion time was also used by Morrison et al. [21] to was presented in Section 4.1, they described that partici-
evaluate the task performance. They present a mobile AR pants had to wear a head band to collect the tracking data.
game using paper maps and overlay information on screen. As they compared their AR system to an LCD approach,
They conducted several evaluations with the third one being the amount and degree of head movements is an indicator
a field study. Participants were asked to take photos for for the user’s perception. They stated that the AR system
completing tasks. Besides measuring the completion time, had less movement in nearly all tasks.
the authors also counted the photos taken and grouped them
by task-related and non-task-related. To collect the data, Perception and cognition can also be evaluated informally,
each team was accompanied by a researcher taking notes, as conducted by Hile et al. [14]. They present an application
photographs and videos. for a mobile phone using AR to navigate pedestrians. To de-
Additionally, the authors also evaluated collaboration and termine a user’s position, they use geotagged photographs.
user experience. The results will be analyzed in Section 4.3 The navigation was performed either using a 2D map or by
and Section 4.4. showing floating arrows on images to direct the user. To
evaluate the system in a field study, an informal study was
Concluding can be said that the metrics used to evaluate performed by evaluating the participants’ user experience in
task performance in the field are very similar to the findings semi-structured interviews. Participants complained about
presented in [3]. In every paper the task completion time is missing depth of arrows making the instructions not clear.
measured and analyzed. The usage of error rate or accuracy
is also very common. Combined with task completion time Summarizing can be said that there is no common mea-
this shows the effectiveness of a system. sure for evaluating perception and cognition. While the work
While field evaluations conducted indoors allow good ob- load can be evaluated formally by using the NASA-TLX
servations similar to laboratory evaluations, researchers test- and is often applied, it is only a good indicator for cogni-
ing their applications in the outdoors found collecting high- tion when comparing different systems. Evaluating the work
quality video recordings very difficult [16]. Therefor, they load in the field does not seem to be more difficult compared
often used written notes or photographs which however lack to evaluation in the laboratory, as it is either measured us-
of detailed data. ing pulse recorders or evaluated using the NASA-TLX test
immediately after each test.
4.2 Perception and Cognition Perception can also be evaluated using informal inter-
views where users express their experience which then can
To measure perception and cognition, Bai and Blackwell
be traced back to perception or cognition.
[3] found that common measures are depth and occlusion
perception, display technology, virtual information layout
and registration error. Also, time and accuracy are used in 4.3 Collaboration
half of the presented papers. However, in this category, all In [3], the papers grouped in this category usually mea-
papers listed in [3] did not conduct any field studies. There- sured collaboration with awareness and ease of use. As al-
for this section only contains papers of an own research. ready in Section 4.2, the papers listed in [3] did not conduct
any field studies. The following two papers evaluated col-
Schwerdtfeger et al. [29] conducted a cognition evaluation laboration very informally.
as well. The paper that was introduced in Section 4.1. To
evaluate the cognition, the authors used the NASA-TLX to Wagner [32, 33] presents an invisible train game using
analyze the task load and compare it to the approach not AR. The game consists of a real miniature wooden train
using AR. For this purpose, the heart rate variability was an- track and PDAs simulating the trains. Users are able to
alyzed by making participants wear a pulse recorder. Their operate junctions and control the simulation. Collaboration
study showed that the task load is higher when using AR, was evaluated informally by conducting unsupervised field
but the difference is not significant. Also, they report about tests. Researchers could observe users passing around the
users having problems using the head-mounted display, es- PDAs and explaining the game to each other. This showed
pecially with focussing. This shows that the perception of the ease of use of the application as users could easily ex-
the system did not work as well as expected. plain it to each other. Also, unstructured interviews were
As the heart rate was measured with a pulse recorder, the conducted with some users.
analysis was not more difficult than it would have been in
the laboratory. A similar approach for evaluating collaboration was used
by Morrison et al [21]. In their paper which was presented
Kjeldskov and Stage [16] also used the work load to evalu- in Section 4.1, they state that the application itself did not
ate cognition by performing a NASA-TLX test immediately require collaboration. However, as the field study was con-
after each evaluation. Therefor, the evaluation was not more ducted in teams, the authors observed team members inter-
difficult than it would have been in the laboratory. The pa- acting to use the application efficiently.
per was already presented in Section 4.1. As they evaluated It can be seen that collaboration is not evaluated fre-
the impact of movement on the ability to perform a task, the quently. While Bai and Blackwell [3] just found 9 papers
evaluating collaboration, an own research only resulted in 4.4.2 Informal User Experience
two papers. Both presented an informal evaluation of col- Hile et al. [14] evaluated user experience using semi-
laboration by just observing the participants or interviewing structured interviews. The paper was already presented in
them afterwards. Section 4.2. By analyzing the interviews, the authors evalu-
ated the usage patterns, learned about confusion while navi-
4.4 User Experience gating and received suggestions for improvement. Also, they
The evaluation of user experience is split into formal and received feedback about the satisfaction of the users. They
informal evaluation, similar to [3]. Formal evaluation mostly conclude that most participants would use the AR system
uses a quantitative approach by using structured question- instead of a common map if given the choice.
naires or interviews, the informal evaluation uses a qualita-
tive approach allowing users to communicate their feelings Completely unstructured and informal interviews were used
about the application. by Wagner [32, 33] to evaluate the user experience. The
While the performance of field studies is usually more dif- papers which were presented in Section 4.3 state that no
ficult than laboratory testing, the evaluation of user expe- specific or predetermined sets of questions were asked [33].
rience in the field can similar to the evaluation in the lab- Still, those interviews helped to learn about the usability
oratory. As the evaluation is mostly performed by using and improve the system. Also, user experience could be ob-
questionnaires and interviews, there is no difference between served unsupervised as visitors passed around the devices
laboratory and field. However, observing users in the field and explained the usage to each other.
which is often used when conducting informal user experi-
ence evaluation, is more difficult. Aside from interviews, Miyashita et al. [18] also used ques-
tionnaires, observations by the museum staff and analysis of
4.4.1 Formal User Experience action logs to evaluate the user experience. They present a
In addition to evaluating task performance and cognition, system to enhance museum visits with AR. Augmentation
Schwerdtfeger et al. [29] also evaluated user experience. The had two functions: Providing background knowledge to art-
paper was presented in Section 4.1. By conducting question- work as well as guiding a visitor through the exhibition. The
naires using the Likert scale and semi-structured interviews, observations showed that the text displayed on the device is
they formally evaluated the user experience, particularly the often too small and the brightness to low. Also, the device
discomfort of using the system. For example, using the AR is too heavy for many users. Interviews confirmed the ob-
system caused more headache or other discomfort concern- servations. Still, users admitted that the system motivated
ing eyes compared to the paper system. As the observer was them to examine the artwork more closely.
standing on a platform above the participant, recording and
analyzing the user’s actions was simple. Rakkolainen et al. [26] used interviews to evaluate the user
experience as well. Furthermore, the participants were asked
Henderson and Feinder [12, 13] evaluated the user experi- to think aloud during the evaluation which was recorded for
ence as well. Similar to Schwerdtfeger et al. [29], they used further analysis. The authors introduce a navigation sys-
questionnaires with Lickert scale. This paper also was pre- tem for pedestrians using AR with a handheld device. As
sented in Section 4.1. To determine the ease of use, the sat- the paper compares different models, the usability study de-
isfaction level and intuitiveness were evaluated using ques- termined that users prefer the 3D model combined with a
tionnaires. The authors found that the LCD system is easier map rather than 3D only or map only.
to use, while the AR system is more satisfiable. For intu-
itiveness both systems were tied. Using simple feedback by users was the choice of Pasman
et al. [25]. They built a system to display large 3D models
Instead of a questionnaire using the Likert scale, Walter- on a handheld device using AR. The purpose of their sys-
Franks and Malaka [34] used the System Usability Scale to tem is to simulate the presence of a large 3D model, e.g. a
evaluate the user experience. Furthermore, they conducted a building outside or a sofa inside. In a field study the sys-
structured interview. They present a pedestrian navigation tem was tested. Aside from determining problems with the
application using AR and photos. To navigate a pedestrian, tracking system and the brightness of the display, the field
the system uses GPS to locate the user. Then, the user test showed that the building being rendered at the actual
is navigated using pre-recorded images augmented with ar- site was appreciated by the users. However, the authors do
rows showing the directions. A field study was conducted not describe if the evaluations were conducted using inter-
to compare the AR navigation mode to a 2D map. While views, questionnaires or recordings.
the questionnaire showed a better user experience using the
AR system, the interviews revealed usability problems for A similar approach was used by Thomas et al. [31]. They
all modes that were evaluated. introduce an AR implementation of the game Quake. Using
the system, the campus of the University of South Australia
It can be seen that formal evaluation of user experience is was turned into a level of the game. The system was eval-
usually performed by using structured questionnaires and in- uated with an informal user study as the participants were
terviews. Formal questionnaires mainly use the Likert scale asked how they felt about the system. Users commended
or the System Usability Scale to evaluate the user experi- the ease of use of the system but complained about the lag
ence. Interviews can also be formal by structuring them which made it difficult to aim at the monsters. Many users
with questions expecting clear answers. Usually an open also found bright lightning to make it difficult to see through
question is added at the end to allow the user to comment the display. As earlier, the authors do not describe how the
on usability issues not covered by the previous questions. participants’ feelings were evaluated.
 Bartie and Mackaness [4] evaluated the user experience nition and collaboration in the field. Task performance is
using interviews and logs of dialogues between the user and usually measured in time to complete an assignment. Other
the system. They present a speech-based system to help the metrics are accuracy or error rate. Measuring perception &
user explore a city. The user wears a laptop in a backpack cognition is often performed using the NASA-TLX to eval-
and communicates with the system by using speech com- uate the work load in comparison to non-AR systems. An-
mands. Using GPS, the system locates the user and gives other metric is informal feedback, for example users com-
information about what buildings the user can see. If de- plaining about headaches when using the AR system [29].
sired, the system also gives additional information about a Collaboration is rarely evaluated. If analyzed, the study is
building. As the system was running on a laptop carried in a usually very informal by just observing how people inter-
backpack, data logging was easy. The evaluation of the logs act with the system. To evaluate the user experience most
revealed some problems addressing the performance. The papers state questionnaires, interviews that can either be
informal user feedback collected in interviews also revealed structured or unstructured and feedback. Some also use
possible improvements. video or audio recordings or action logs to gather informa-
tion about the usability.
Solely interviews were used by Kray et al. [17] to evaluate
the user experience. They introduce a mobile navigation sys- Despite higher costs and more time consumption [15, 22]
tem for pedestrians using mobile phones to display routing compared to laboratory evaluation, it can be seen that many
information. The interviews were used to collect the opin- papers conduct field studies to evaluate the usability under
ions of the users. Even though the attitude towards the sys- realistic conditions. As the cognitive load is higher in the
tem was positive, interviews showed that users complained field [7, 23], users are more stressed than in the laboratory.
about the slow usage of 3D maps. Also, the interaction be- Reasons can be privacy issues, noise, low visibility because of
tween 2D and 3D maps did not work well. sunlight, low GPS signal and others. Those reasons cannot
completely be simulated in the laboratory which makes field
Concluding can be said that informal evaluation of user testing indispensable for good usability evaluation.
experience is mostly conducted using interviews or feed-
back. The interviews are often unstructured using open 6. REFERENCES
questions. Also, observations help evaluating the user expe-
rience. In [18], the interviews mostly confirmed the observa- [1] R. T. Azuma et al. A survey of augmented reality.
tions. When conducting informal evaluation, interviews are Presence, 6(4):355–385, 1997.
usually recorded and analyzed afterwards to extract the us- [2] C. Baber. Evaluating mobile human-computer
ability flaws. As seen in [4], logging can also help identifying interaction. Handbook of Research on User Interface
usability problems. Especially if the field test takes longer Design and Evaluation for Mobile Technology,
than just a few minutes, users probably will not remember 1:731–744, 2008.
all usability problems they detected. [3] Z. Bai and A. F. Blackwell. Analytic review of
usability evaluation in ISMAR. Interacting with
5. CONCLUSION Computers, 24(6):450–460, 2012.
[4] P. J. Bartie and W. A. Mackaness. Development of a
Even though usability evaluation of AR applications is ris-
speech-based augmented reality system to support
ing [3, 8], field studies are still not common. Most papers
exploration of cityscape. Transactions in GIS,
reviewed in [3, 8] only evaluate usability in the laboratory.
10(1):63–86, 2006.
However, several papers comparing laboratory vs. field eval-
[5] F. Doil, W. Schreiber, T. Alt, and C. Patron.
uation show that field studies reveal more usability problems
Augmented reality for manufacturing planning. In
[7, 22, 27]. They conclude that even though field testing is
Proceedings of the workshop on Virtual environments
more expensive and time consuming, it is worth the effort
2003, pages 71–76. ACM, 2003.
and cost.
[6] S. Dow, B. MacIntyre, J. Lee, C. Oezbek, J. D. Bolter,
As usability becomes more important, user experience and M. Gandy. Wizard of Oz support throughout an
should be a central design goal [24]. To allow user-driven iterative design process. Pervasive Computing, IEEE,
development, different strategies exist. One is to apply tra- 4(4):18–26, 2005.
ditional HCI principles to AR. However, this is difficult, as [7] H. B.-L. Duh, G. C. Tan, and V. H.-h. Chen. Usability
AR registers information in 3D [1] and offers different inter- evaluation for mobile device: a comparison of
action possibilities [9]. While traditional HCI methods can laboratory and field tests. In Proceedings of the 8th
be used to determine what information should be presented, conference on Human-computer interaction with
they do not show how to present those information [30]. To mobile devices and services, pages 181–186. ACM,
allow user-driven development the model for usability engi- 2006.
neering can be used [11]. Another technique allowing fast [8] A. Dünser, R. Grasset, and M. Billinghurst. A survey
usability evaluation is the use of the Wizard of Oz to simu- of evaluation techniques used in augmented reality
late components that do no exist yet. studies. Technical report, 2008.
[9] A. Dünser, R. Grasset, H. Seichter, and
When evaluating the usability of an AR application, there M. Billinghurst. Applying HCI principles to AR
are four categories, as presented in [3]. Even though most systems design. 2007.
field studies mainly address the user experience by using in- [10] W. Friedrich, D. Jahn, and L. Schmidt. ARVIKA -
formal evaluation techniques such as feedback or interviews, augmented reality for development, production and
some also evaluate the task performance, perception & cog- service. In ISMAR, volume 2002, pages 3–4, 2002.
[11] J. Gabbard, J. E. Swan II, D. Hix, M. O. Lanzagorta, conference on Human-computer interaction: changing
M. Livingston, D. B. Brown, and S. J. Julier. roles, pages 272–280. ACM, 2006.
Usability engineering: domain analysis activities for [23] L. Nunes and M. A. Recarte. Cognitive demands of
augmented-reality systems. In Electronic Imaging hands-free-phone conversation while driving.
2002, pages 445–457. International Society for Optics Transportation Research Part F: Traffic Psychology
and Photonics, 2002. and Behaviour, 5(2):133–144, 2002.
[12] S. Henderson and S. Feiner. Exploring the benefits of [24] T. Olsson and M. Salo. Narratives of satisfying and
augmented reality documentation for maintenance and unsatisfying experiences of current mobile augmented
repair. Visualization and Computer Graphics, IEEE reality applications. In Proceedings of the 2012 ACM
Transactions on, 17(10):1355–1368, 2011. annual conference on Human Factors in Computing
[13] S. J. Henderson and S. Feiner. Evaluating the benefits Systems, pages 2779–2788. ACM, 2012.
of augmented reality for task localization in [25] W. Pasman, C. Woodward, M. Hakkarainen,
maintenance of an armored personnel carrier turret. In P. Honkamaa, and J. Hyväkkä. Augmented reality
Mixed and Augmented Reality, 2009. ISMAR 2009. with large 3D models on a PDA: implementation,
8th IEEE International Symposium on, pages 135–144. performance and use experiences. In Proceedings of the
IEEE, 2009. 2004 ACM SIGGRAPH international conference on
[14] H. Hile, R. Vedantham, G. Cuellar, A. Liu, Virtual Reality continuum and its applications in
N. Gelfand, R. Grzeszczuk, and G. Borriello. industry, pages 344–351. ACM, 2004.
Landmark-based pedestrian navigation from [26] I. Rakkolainen and T. Vainio. A 3D city info for
collections of geotagged photos. In Proceedings of the mobile users. Computers & Graphics, 25(4):619–625,
7th international conference on mobile and ubiquitous 2001.
multimedia, pages 145–152. ACM, 2008. [27] Y. Rogers, K. Connelly, L. Tedesco, W. Hazlewood,
[15] J. Kjeldskov, M. B. Skov, B. S. Als, and R. T. Høegh. A. Kurtz, R. E. Hall, J. Hursey, and T. Toscos. Why
Is it worth the hassle? Exploring the added value of it’s worth the hassle: The value of in-situ studies when
evaluating the usability of context-aware mobile designing UbiComp. In UbiComp 2007: Ubiquitous
systems in the field. In Mobile Human-Computer Computing, pages 336–353. Springer, 2007.
Interaction-MobileHCI 2004, pages 61–73. Springer, [28] B. Schwerdtfeger. AVILUS tracking contest 2010 -
2004. Abschlussbericht. Technical report, Technical report,
[16] J. Kjeldskov and J. Stage. New techniques for usability TUM, 2010.
evaluation of mobile systems. International journal of [29] B. Schwerdtfeger, R. Reif, W. A. Gunthner,
human-computer studies, 60(5):599–620, 2004. G. Klinker, D. Hamacher, L. Schega, I. Bockelmann,
[17] C. Kray, C. Elting, K. Laakso, and V. Coors. F. Doil, and J. Tumler. Pick-by-Vision: A first stress
Presenting route instructions on mobile devices. In test. In Mixed and Augmented Reality, 2009. ISMAR
Proceedings of the 8th international conference on 2009. 8th IEEE International Symposium on, pages
Intelligent user interfaces, pages 117–124. ACM, 2003. 115–124. IEEE, 2009.
[18] T. Miyashita, P. Meier, T. Tachikawa, S. Orlic, [30] J. E. Swan and J. L. Gabbard. Survey of user-based
T. Eble, V. Scholz, A. Gapel, O. Gerl, S. Arnaudov, experimentation in augmented reality. In Proceedings
and S. Lieberknecht. An augmented reality museum of 1st International Conference on Virtual Reality,
guide. In Proceedings of the 7th IEEE/ACM pages 1–9, 2005.
International Symposium on Mixed and Augmented [31] B. Thomas, B. Close, J. Donoghue, J. Squires,
Reality, pages 103–106. IEEE Computer Society, 2008. P. De Bondi, M. Morris, and W. Piekarski. ARQuake:
[19] L. Molin. Wizard-of-Oz prototyping for co-operative An outdoor / indoor augmented reality first person
interaction design of graphical user interfaces. In application. In Wearable Computers, The Fourth
Proceedings of the third Nordic conference on International Symposium on, pages 139–146. IEEE,
Human-computer interaction, pages 425–428. ACM, 2000.
2004. [32] D. Wagner. Handheld augmented reality. PhD thesis,
[20] A. Möller, M. Kranz, S. Diewald, L. Roalter, R. Huitl, Citeseer, 2007.
T. Stockinger, M. Koelle, and P. Lindemann. [33] D. Wagner, T. Pintaric, F. Ledermann, and
Experimental evaluation of user interfaces for visual D. Schmalstieg. Towards massively multi-user
indoor navigation. In Proceedings of the SIGCHI augmented reality on handheld devices. In Pervasive
Conference on Human Factors in Computing Systems, Computing, pages 208–219. Springer, 2005.
CHI ’14, pages 3607–3616. ACM, 2014. [34] B. Walther-Franks and R. Malaka. Evaluation of an
[21] A. Morrison, A. Oulasvirta, P. Peltonen, S. Lemmela, augmented photograph-based pedestrian navigation
G. Jacucci, G. Reitmayr, J. Näsänen, and A. Juustila. system. In Smart Graphics, pages 94–105. Springer,
Like bees around the hive: a comparative study of a 2008.
mobile augmented reality map. In Proceedings of the [35] C. Ware and R. Balakrishnan. Reaching for objects in
SIGCHI Conference on Human Factors in Computing VR displays: lag and frame rate. ACM Transactions
Systems, pages 1889–1898. ACM, 2009. on Computer-Human Interaction (TOCHI),
[22] C. M. Nielsen, M. Overgaard, M. B. Pedersen, 1(4):331–356, 1994.
J. Stage, and S. Stenild. It’s worth the hassle!: the
added value of evaluating the usability of mobile
systems in the field. In Proceedings of the 4th Nordic
 Evaluation of Augmented Reality Applications in a
Laboratory Context

Prisca Bonnet
Universität Passau
[email protected]

ABSTRACT Therefore, a user-based evaluation is often suitable. As

The development of Augmented Reality (AR) applications AR creates a new user experience, new evaluation techniques
has become increasingly popular recently, with the spread have to be developed. The mobile aspect of AR applications
of portable and cheap high-fidelity mobile devices such as require to develop new methods during conception, develop-
smartphones, tablets and dataglasses. Those devices are ment and evaluation. “HCI methods, models and techniques
particularly suited for AR applications because of their mo- will need to be reconsidered if they are to address the con-
bility and their various input mechanisms e.g. voice com- cerns of interactions on the move.” [12].
mand, touchscreen, high-quality camera. Evaluation can be conducted either in the field or in a lab-
In the field of Human-computer interaction (HCI), usabil- oratory environment. Testing in the field is the natural way
ity and user experience (UX) studies are often conducted to conduct user-based evaluation, because it puts the user
via user-based evaluation. Those evaluation can be divided in realistic conditions, but it requires specific consideration
in two categories: laboratory and field-test studies. While compared to controlled laboratory experiment [18, 22].
evaluation in a laboratory allows to control precisely the ex- It can be difficult to collect data in the field, and it may
periment’s condition, evaluation in the field is more realistic be impossible to concentrate research on specific aspects of
and take more parameters into account. the interactions [22, 24]. Furthermore, several evaluation
Recently, new techniques has been developed to conduct techniques, such as Think aloud or video recording,can be
more realistic evaluations within a controlled laboratory en- difficult to use in the field [12].
vironment. In this report, we will consider several labo- Evaluating a product while sitting down in a laboratory
ratory evaluation setups and discuss their advantages and leads to different results than an evaluation in the field, or
drawbacks. We will describe and classify those setups using while moving. On the other hand, focusing only on the prod-
different aspects : display, interaction, controller and mea- uct can lead to the finding of more usability problems [15].
sures. Finally, we will discuss the main issues related to On the contrary, other studies [8, 22] point out that evalua-
those setups. tion in the field allows to discover more usability problems,
due to various parameters: ambient noise, movement, pri-
Keywords vacy concerns.
Augmented Reality, Evaluation, Usability, UX A survey by Kjeldskov and Graham in 2003 on evalua-
tion of mobile systems [14] pointed out that 71 percent of
1. INTRODUCTION the evaluations were conducted in a laboratory environment,
The evaluation on any application can be done either with but most of them preferred to use a trial and error mecha-
or without users. If conducted without users, the evaluation nism rather than focusing grounding engineering to develop
is based on heuristics [2]. If several studies have been con- new techniques to meet the challenge of mobile systems. In
ducted to create guidelines for the evaluation of virtual en- 2004, Kjeldskov and Stage [15] reviewed 114 papers (1996
vironments [28, 29], heuristics-based evaluation is difficult - 2002) dealing with HCI in mobile systems. Only half of
to apply to AR applications, because of the range of the the papers considered usability, and 6 proposed new evalu-
domain itself [2]. When an AR application can be needing ation techniques. More recently, several solutions have been
mobile or fixed devices, one or several users, be controlled developed, in order to create a realistic environment in a
by voice command or a touchscreen, and be intended for laboratory context. Different aspects of the user experience
novice or experimented users, it is indeed difficult to define have been studied, and have led to new ways of conducting
relevant heuristics [6, 9]. user evaluation in a laboratory.
In the following sections, we will describe and compare
several setups for evaluation in controlled laboratory envi-
ronment. It can rely on a display to immerse the user in a
Permission to make digital or hard copies of all or part of this work for given situation, or on specific interactions techniques. Those
personal or classroom use is granted without fee provided that copies are will be described in the next two sections. The different con-
not made or distributed for profit or commercial advantage and that copies trollers that can be used to simulate a real environment are
bear this notice and the full citation on the first page. To copy otherwise, to described in section 4. Section 5 lists the different measures
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
used during evaluation process. In section 6, we discuss the
Advances in Embedded Interactive Systems ’14 Passau, Germany issues related to evaluation in a laboratory environment.
Copyright 2014 ACM X-XXXXX-XX-X/XX/XX ...$15.00.
2. DISPLAYS supermarket model was created, and adjusted to the differ-
This section covers the different displays used to immerse ent experiments. With this approach, the developers were
the user in a given situation for the evaluation. We discuss able to modify the virtual supermarket to fit the needs of
advantages and drawbacks of the described methods. We the experiment, and the user could move freely in the virtual
distinguish two types of setups: those where a prerecorded space.
scene is displayed, and those which use computer-rendered The main drawback of this environment was the lack of
environments. products in the shelves, and of other visitors in the virtual
supermarket. Figure 2 shows the adaptability but lack of
2.1 Prerecorded video realism of their approach.
Singh et al. [27] propose to use immersive video as a tool To study navigational place findings, Schellenbach et al. [26]
for prototyping and evaluating AR applications. It displays used a virtual environment (VE) in which the user moves.
a recorded video in a wide field-of-view display. In their They created a virtual museum, and the display was a big
prototype, they used three screens to create a viewing angle screen in front of the treadmill. Two projectors allowed 3D
of about 140 degrees. Their setup is shown in Figure 1. projection. They tracked the head position of the user to
Ostkamp and Kray [23] created an environment where any correct the perspective accordingly.
android application could be evaluated. They used an im- Computer-rendered environments are also used in van Veen
mersive video environment with a viewing angle of 114 de- et al. experiment, using the laboratory at the Max Planck
grees. The video footage was recorded with three cameras Institute in Tuebingen [31].
mounted on a tripod and an audio recorder.
This kind of approach allows to use real video footage 3. INTERACTIONS
instead of a computer-rendered environment. Therefore, the When using an AR application, the user often interacts
simulation is more realistic. Furthermore, it is relatively with the system. The interaction can be limited to a mobile
cheap and fast. Thus, this approach does not allow the user device used to run the AR application, or consist of several
to move in the environment, and limits the control over the ways to move in a simulated environment.
displayed video.
3.1 Mobile devices
2.2 Computer-rendered environment The AR application runs often on a mobile device, e.g.
The main advantage of this approach is the possibility for a smartphone or a head-mounted display (HMD). This de-
the user to move freely in the computer-generated space. vice can be virtual, and the interaction will rely on a key-
This means that the user can walk, turn and even duck, board and/or a mouse. It can also be physical and interact
and see the corresponding image on the screen. With this if needed with the virtual environment.
approach, it is possible to control the content of the virtual
environment. Therefore, it is suitable to study the influence 3.1.1 Virtual mobile devices
of some parameters on the user’s reactions. When the environment is simulated with a VE, the mobile
To evaluate an advertising application in a supermarket, device can also be part of the simulation. Several prototyp-
Hühn et al. [10] used a computer assisted virtual environ- ing or evaluation tools follow this approach [3, 7, 11].
ment (CAVE). The virtual environment was projected on 4 For example, Ubiwise [3] is a simulation tool designed to
screens disposed as a close space around the user. A virtual investigate applications which use mobile phones as interac-

Figure 1: The immersive video environment of Singh et al [27].

 Figure 2: The computer-rendered supermarket of Hühn et al. [10]

tion devices. Those devices are represented in the 3D sim- 3.2.1 Treadmills
ulation, an therefore have to be interacted with via mouse In their study, Schellenbach et al. [26] used a treadmill to
and keyboard. The interaction process is neither intuitive simulate the user’s movement in their VE. They argue that
nor immersive, and cannot represent a realistic use of the self motion is an important component of navigational place
devices [10]. finding, as pointed out in [19]. In fact, it has been proved
Those approaches provide a cheap an easy way to conduct that cognitive and motor functions compete for shared men-
evaluation, even in the prototyping phase [3]. But the main tal resources [17, 32].
drawback is the disruption with the real usage of the mobile Shellenbach et al. [26] argue that laboratory evaluation
device [11]. The user has to use a mouse and keyboard allows to have a clean data collection with replicable condi-
instead of directly interacting with the device. tions. The different parameters on the user experience can
Therefore, hybrid simulations, using a virtual environ- be distinguished and studied. The treadmill they used can
ment and a physical mobile device, can be a good way to adapt to the user preferred speed. Since the treadmill only
have a more realistic and intuitive simulation. moves in one direction, only one screen is sufficient for the
user to immerse in the simulation.
3.1.2 Physical mobile devices
Using a physical mobile device for evaluation, even in early 3.2.2 Buttons
stages of development, can be a rapid way to discover many In the treadmill experiment by Schellenbach et al. [26],
usability issues [8, 10]. the user could only move in the direction of the treadmill.
Leichtenstern et al. [16] used a virtual world and physi- To enable turning moves in the VE, they used wireless han-
cal mobile phone as an evaluation platform for user studies. deld buttons to perform smooth virtual turns in the virtual
They implemented several interactions the user can make on museum. It was not a very intuitive approach, but it was
the virtual world, e.g. control a TV or a heater, through sev- an easy way to overcome the issue of turning moves.
eral interaction techniques: scanning, pointing and touching. Vadas et al. [30] wanted to study the action of reading on
They conducted two user studies, one in real condition and a mobile device while on the move. They wanted to compare
the other one within the hybrid environment, and obtained three mobile devices, but without taking their input mech-
similar results for both experiments. Since they used a stan- anisms into account. To that end, they asked their test
dard screen to display the virtual environment to the user, subjects to use a basic input device with buttons, regardless
some test subjects had difficulties immersing in the simula- of which mobile device they were testing. The subjects had
tion. Using a CAVE could improve the immersion. to read a text and answer multiple choice questions. The
Hühn et al. [10], Ostkamp and Kray [23] and several oth- text was displayed either on a mobile device, on an e-book
ers [26, 8, 15, 1] used a physical mobile device to conduct reader or on a head-mounted display. The keypad used for
their evaluation. Since they wanted to study the user ex- input allowed to choose an answer and validate the answer.
perience with a given application where the mobile device Figure 3 shows their setup. With this method, only the dis-
was in the center of the interaction, using a physical mobile play mechanism of the three devices had an influence on the
device was the best way to have a realistic simulation. the user’s experience.

3.2 Movement 3.2.3 Sensors

Most of the time, AR applications are used while “on the In their CAVE, Hühn et al. [10] used a head-tracking sys-
move“. Taking the user’s movement into consideration in a tem to know the position of the user and adjust the displayed
laboratory evaluation can be helpful both for the user’s im- image accordingly. The user could duck or jump during the
mersion and to consider that part of the user’s experience. experiment, and see the perspective adjusted. To control
his movement in the virtual supermarket, the user acted as
Figure 3: The experiment of Vadas et al [30]. Left: Input device. Right: Three reading devices compared in
their study.

a ”human joystick“. When standing in the center of the 4.2 Prototypes

CAVE, the virtual camera does not move. When the user In his research on interactive public displays, Nakanishi [21]
move away from the center, the virtual camera moves in the used two different methods to evaluation an application in
same direction as the user. Since the CAVE provides a 360 early stages.
degrees display, the user can turn freely in the virtual super- The first method used full virtual prototyping, where the
market. This was a good solution to allow the user to move environment, display and users are represented in a virtual
freely in the virtual environment. However, the users could world. He used Landolt rings to clarify whether an element
experience dizziness due to disorientation in the CAVE. was clearly visible to the users, which allows to adapt the
text or images to an appropriate size. The visual field of
4. CONTROLLER each virtual user was represented by a green cone-shaped
light. This method allows to discover many usability prob-
When evaluating an AR application in a laboratory con-
lem without needing to deploy the system, which can be
text, there is often a need to control the virtual environment,
very expensive, especially for public displays.
or to simulate real world conditions. In this section, we de-
The second method used by Nakanishi involve miniature
scribe several ways to achieve this goal in a limited amount
model of the system, using miniature characters, cameras
of time.
and displays. This is a good solution to deploy a miniature
4.1 Wizard of Oz version of a public display, but it can be difficult to take
many parameters into account, e.g. brightness level. The
The Wizard of Oz paradigm is described in [5]. Using a methods used by Nakanishi are illustrated in Figure 4.
Wizard of Oz system allows to conduct user studies without
the need of a high-fidelity prototype. The Wizard of Oz 4.3 Sensor overwriting
paradigm was e.g. used in [20] to simulate high-precision In their immersive video environment, Ostkamp and Kray [23]
indoor tracking. used android push notifications to manipulate the GPS in-
Hühn et al. [10] simulated the act of selecting a product formation without needing the source code of the applica-
in their virtual environment by asking the user to make a tion. However, this method only works with GPS informa-
grabbing gesture, while the developer played an auditory tion, and cannot be used for other sensors e.g. compass.
feedback. This is a easy and rapid way to simulate this
interaction, without spending to much time on development.
In this case, the act of grabbing a product was not in the 5. MEASURES
center of this experiment. Therefore, the lack of realism of Usability can be measured via objective and subjective
this interaction did not represent an important issue. measures [4]. Most of the time, usability problems are iden-
Schellenbach et al. [26] used a Wizard of Oz mechanism tified via subjective measures such as questionnaire or Think
as well. The user, who only could move in one direction aloud.
on the treadmill, had the possibility to use voice commands Objective measures are used to consider performance or
to control turns in the virtual environment. Voice recog- learning curve. In this section, we describe several types of
nition has not been implemented in this experiment, but measure commonly used during evaluation in a laboratory
the researcher acted as a wizard by manually running the context.
turning movement when the test subject emitted the voice
command. This was more intuitive for the user than wire- 5.1 Usability questionnaire
less buttons, and using a Wizard of Oz mechanism was a Usability questionnaire is a simple way to discover us-
way to reduce implementation time without modifying the ability issues [2]. Therefore, it is commonly used to collect
user’s experience. subjective data on an application.
 Figure 4: The experiment of Nakanishi [21]. Left: Miniaturing. Right: Full virtual prototyping.

In their paper, Duh et al [8] compared laboratory and field focus on the application during the experiment, and take
tests to evaluate a mobile application. They used a usability the time to analyze his reaction afterward.
questionnaire to gather information of the user’s subjective Kjeldskov et al. [15] compared six evaluation setup in the
response to the experiment. They used both Likert scale experiment. They recorded all evaluations and used three
and open-ended questions. experienced usability experts to identify usability problems
Hühn et al. [10] studied the perceived intrusiveness of an by viewing the video.
application in different conditions. After the experiment, Duh et al [8] also used video recording to analyse the user’s
they asked the participants to fill out a questionnaire to response to the given task.
measure the user’s reaction to the add. Using the same While video recording can be complicated in a field eval-
questionnaire in different conditions, they were able to verify uation [8, 15], it is easy to use in a controlled laboratory
their hypotheses. environment.
In their experiment, Vadas et al. [30] used the NASA Task
Load Index (TLX) to measure subjective workload ratings. 5.4 Sensors
They used this questionnaire to compare the perceived work- To study the user’s behavior toward an application, dif-
load while reading on three different display technologies: ferent sensors can be used. It can consist of head or eye
palmtop computer, e-book reader and HMD. With the re- tracking, or sensors to capture the user’s posture or move-
sults, they were able to determine which device was per- ment.
ceived the most frustrating or demanding the most effort. Schellenbach et al. [26] proposed a setup where the user’s
Kjeldskov et al. [15] also used the TLX to compare six movement on a treadmill can be studied. The VICON mo-
evaluation setups: walking in a pedestrian street, sitting at tion capture system allows for body motion capturing. Such
a table, walking on a treadmill at constant or varying speed, a setup can only be realized in a laboratory environment.
and walking on a constantly changing course at constant or
varying speed. 5.5 Performance
5.2 Think aloud To compare different applications or setups, objective mea-
sures can be taken, to allow a quantitative comparison.
The Think aloud technique is a simple way to identify us-
Vadas et al. [30] asked their test subjects to walk a path
ability problems. The participant is asked to speak as much
while reading short texts and answering multiple-choice ques-
as possible during the experiment. It allows to understand
tions. As they intended to compare three display technolo-
the user’s subjective response to an application, and to iden-
gies, they measured the walking and reading performances.
tify usability problems in a more spontaneous way than with
They considered the reading time, the percentage of cor-
a questionnaire.
rect answers and walking time to quantify the differences
To identify usability problems, Duh et al [8] used the
between the devices.
Think aloud technique and recorded the user’s reaction and
Kjeldskov et al. [15] also measured the test subject’s per-
comments. They noticed that this technique is easier to
formance on five different evaluation setups in order to com-
use in a laboratory environment than in the field, mostly
pare with the performance in real conditions (here, walking
because of social behavior issues in real conditions.
on a pedestrian street).
Because of the ambient noise and the presence of strangers,
Think aloud may be difficult to use in a field setting [8, 25].
But in a laboratory evaluation, it is a trivial way to observe 6. ISSUES
the user’s reaction to an application.
We described several setups allowing to conduct realistic
evaluation in a controlled laboratory environment. However,
a laboratory evaluation will not give the same results as a
5.3 Video recording field evaluation [8, 15, 22].
Video recording allows to study the user’s reaction to an In this section, we will describe the main issues related to
application (e.g. body language, frustration), but is also a evaluation in a laboratory environment.
way to conduct retrospective Think aloud. The main advan-
tage of retrospective Think aloud is that the participant can
6.1 Virtual Environment 6.3 Movement
When evaluating AR applications using a virtual environ- The user’s movement is an important part of the UX [17].
ment, different issues may occur. Allowing the participant to move in the virtual environment
First, the physical environment can be limiting during the is a way to improve the evaluation’s realism [10, 26], but the
experiment [4]. For example in the setup of Schellenbach evaluation’s results will still differ from the results in real
et al. [26], the user can only move in one direction. The conditions [8, 15].
physical limits when interacting with a virtual environment New VE technologies such as the Virtualizer by Cyberith,
may prevent the user from accomplishing some actions, e.g. shown in Figure 5, may help overcome some issues. The
reaching a given point in the VE. Virtualizer is an omnidirectional treadmill allowing the user
Second, the interactions between the participant and the to move in any direction within a very limited space. The
VE are limited. Interactions with an object or another per- user can walk, run or even duck and jump in a natural way,
son will be limited or artificially simulated, as in [3, 10, 11]. and his movements are directly transmitted to the computer.
This prevents the user from completely immerse in the sim- Combined with a CAVE or a HMD, this technology can
ulation and may cover up usability problems. allow to overcome movement-related issues, without needing
Third, computer-rendered may lack realism, and therefore extra space.
introduce bias in the user’s reaction to an application. Ad-
vances in computer graphics may allow to render more real- 6.4 Social behavior
istic virtual environments in the coming years, and therefore The main drawback of laboratory evaluation is the lack of
help overcome this issue. realism or the non-existence of social behavior. It has been
proven that social behavior has an impact on the UX [8,
22], but it is difficult to conceive a laboratory experiment
6.2 Device reliability including other users in a realistic way. Users can be added
When using devices in a laboratory environment or simu- in a VE [21] or played by real persons [8], but it will not be
lating interaction with a Wizard of oz technique, the device’s completely realistic.
reliability may not be taken into account. For example, the However, social behavior may be affected in a field evalu-
accuracy of GPS location is not always ideal in real condi- ation as well, due to the devices used to record the experi-
tions, especially for indoors applications. This lack of accu- ment [8, 15].
racy won’t occur when using a Wizard of Oz technique. Another issue linked to social behavior is the fact that the
The ambient noise of real conditions may not be taken into user knows he is being observed. Therefore, his behavior
account when using an application with voice recognition will be affected by this knowledge. It is possible to realize
features in a laboratory experiment. Therefore, a perfect observations without having the user aware of it [13], but
application in laboratory conditions may be difficult to use the observation will be limited. Furthermore, this kind of
in the field. experiment can only be conducted in the field and may rise
The brightness level or speaker volume of a smartphone privacy concern.
can appear ideal in laboratory conditions, but not in certain Another approach is to let the user ignore the main goal of
field conditions. the experiment. Hühn et al. [10] conducted their experiment
In those cases, several usability issues may be neglected [8]. in that way: they gave the user a task to accomplish in
the virtual environment, while it was their reaction to the

Figure 5: The Virtualizer by Cyberith. Source : http://www.cyberith.com/en/virtualizer and

http://www.cyberith.com/en/press
display of an advertisement on their mobile device which applications in virtual environments: A survey.
was observed. However, this approach is limited and cannot International Journal of Mobile Human Computer
be applied on every problems. Interaction (IJMHCI), 5(4):1–19, 2013.
[7] J. Dongsik, Y. Ungyeon, and S. Wookho. Design
7. CONCLUSION evaluation using virtual reality based prototypes :
In this paper, we described several setups to conduct re- towards realistic visualization and operations. In
alistic user-based evaluations in a controlled laboratory en- Mobile HCI’07 : Proceedings of the 9th international
vironment. We described the various displays, interactions, conference on Human computer interaction with
controllers and measures available to conduct those evalua- mobile devices and services, pages 246–258. ACM,
tions. 2007.
We discussed the advantages and drawbacks of one solu- [8] H. B.-L. Duh, G. C. Tan, and V. H.-h. Chen. Usability
tion against another: prerecorded video against computer- evaluation for mobile device: a comparison of
rendered environment, physical mobile device against virtual laboratory and field tests. In Proceedings of the 8th
mobile device, Wizard of Oz against Prototype. conference on Human-computer interaction with
We identified several evaluation techniques and measures, mobile devices and services, pages 181–186. ACM,
such as video recording, Think aloud and movement cap- 2006.
ture, which are easier to use in a controlled laboratory en- [9] A. Dünser and M. Billinghurst. Evaluating augmented
vironment. We also discussed to what extent laboratory reality systems. In Handbook of Augmented Reality,
evaluation allows to control an experiment’s conditions. pages 289–307. Springer, 2011.
We described several setups which offer interesting up- [10] A. E. Hühn, V.-J. Khan, A. Lucero, and P. Ketelaar.
sides: a CAVE with real control of the environment and On the use of virtual environments for the evaluation
possibility for the user to move freely in the VE [10], an of location-based applications. In Proceedings of the
immersive video environment using real video footage and 2012 ACM annual conference on Human Factors in
compatible with context-aware applications [23], and a setup Computing Systems, pages 2569–2578. ACM, 2012.
with a treadmill, a 3D display and the possibility to model [11] S. Ichiro. Flying emulator: Rapid building and testing
the participant’s movement [26]. of networked applications for mobile computers. In
However, even with a very realistic environment, they are Proceedings of Conference on Mobile Agents (MA
several issues remaining. First, real conditions, e.g. light- 2001), pages 103–118, 2001.
ning, ambient noise or sensor accuracy, are difficult to sim- [12] P. Johnson. Usability and mobility. Workshop on
ulate perfectly in a laboratory environment. Second, the Human-Computer Interaction with Mobile Devices,
user’s movement won’t be perfectly natural, due to the phys- Glasgow, 1998.
ical setup. Third, the user’s behavior will be affected by the [13] A. Karlson, B. Bederson, and J. Contreras-Vidal.
evaluation. The social interactions won’t be realistic, and Understanding single-handed mobile device
the knowledge of being observed will affect the user’s be- interaction. Handbook of research on user interface
havior. Those issues are difficult to solve, even if recent design and evaluation for mobile technology, pages
technological advances may help to overcome some of those 86–101, 2006.
problems in the coming years. [14] J. Kjeldskov and C. Graham. A review of mobile hci
research methods. In Human-computer interaction
8. REFERENCES with mobile devices and services, pages 317–335.
[1] C. Ardito, P. Buono, M. F. Costabile, R. Lanzilotti, Springer, 2003.
and A. Piccinno. A tool for wizard of oz studies of [15] J. Kjeldskov and J. Stage. New techniques for usability
multimodal mobile systems. In Proceedings of Human evaluation of mobile systems. International journal of
System Interaction (HSI 2009), pages 344–347. IEEE human-computer studies, 60(5):599–620, 2004.
Computer Society, 2009. [16] K. Leichtenstern, E. André, and M. Rehm. Using the
[2] C. Baber. Evaluating mobile human-computer hybrid simulation for early user evaluations of
interaction. Handbook of Research on User Interface pervasive interactions. In Proceedings of the 6th
Design and Evaluation for Mobile Technology, Nordic Conference on Human-Computer Interaction:
1:731–744, 2008. Extending Boundaries, pages 315–324. ACM, 2010.
[3] J. J. Barton and V. Vijaraghavan. Ubiwise: A [17] K. Z. Li and U. Lindenberger. Relations between
ubiquitous wireless infrastructure simulation aging sensory/sensorimotor and cognitive functions.
environment, hp technical repports. HP Labs 2003-93, Neuroscience & Biobehavioral Reviews, 26(7):777–783,
2003. 2002.
[4] D. A. Bowmabown, J. L. Gabbard, and D. Hix. A [18] Y. Li, J. I. Hong, and J. A. Landay. Topiary: a tool
survey of usability evaluation in virtual environments: for prototyping location-enhanced applications. In
classification and comparison of methods. Presence: Proceedings of the 17th annual ACM symposium on
Teleoperators and Virtual Environments, User interface software and technology, pages 217–226.
11(4):404–424, 2002. ACM, 2004.
[5] N. Dahlbäck, A. Jönsson, and L. Ahrenberg. Wizard [19] M. Lövdén, M. Schellenbach, B. Grossman-Hutter,
of oz studies - why and how. Intelligent User A. Krüger, and U. Lindenberger. Environmental
Interfaces ’93, pages 193–200, 1993. topography and postural control demands shape
[6] I. Delikostidis, T. Fechner, H. Fritze, A. M. aging-associated decrements in spatial navigation
AbdelMouty, and C. Kray. Evaluating mobile performance. Psychology and aging, 20(4):683, 2005.
[20] A. Möller, M. Kranz, S. Diewald, L. Roalter, R. Huitl,
T. Stockinger, M. Koelle, and P. Lindemann.
Experimental evaluation of user interfaces for visual
indoor navigation. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems,
CHI ’14, pages 3607–3616, New York, NY, USA, 2014.
ACM.
[21] Y. Nakanishi. Virtual prototyping using miniature
model and visualization for interactive public displays.
In Proceedings of the Designing Interactive Systems
Conference, pages 458–467. ACM, 2012.
[22] C. Nielsen. Testing in the field. In Computer Human
Interaction, 1998. Proceedings. 3rd Asia Pacific, pages
285–290. IEEE, 1998.
[23] M. Ostkamp and C. Kray. Prototyping mobile ar in
immersive video environments.
[24] J. Pascoe, N. Ryan, and D. Morse. Using while
moving: Hci issues in fieldwork environments. ACM
Transactions on Computer-Human Interaction
(TOCHI), 7(3):417–437, 2000.
[25] N. Sawhney and C. Schmandt. Nomadic radio: speech
and audio interaction for contextual messaging in
nomadic environments. ACM transactions on
Computer-Human interaction (TOCHI), 7(3):353–383,
2000.
[26] M. Schellenbach, A. Krüger, M. Lövdén, and
U. Lindenberger. A laboratory evaluation framework
for pedestrian navigation devices. In Proceedings of the
4th international conference on mobile technology,
applications, and systems and the 1st international
symposium on Computer human interaction in mobile
technology, pages 495–502. ACM, 2007.
[27] P. Singh, H. N. Ha, Z. Kuang, P. Olivier, C. Kray,
P. Blythe, and P. James. Immersive video as a rapid
prototyping and evaluation tool for mobile and
ambient applications. In Proceedings of the 8th
conference on Human-computer interaction with
mobile devices and services, pages 264–264. ACM,
2006.
[28] K. M. Stanney, M. Mollaghasemi, L. Reeves,
R. Breaux, and D. A. Graeber. Usability engineering
of virtual environments (ves): identifying multiple
criteria that drive effective ve system design.
Interacting with computers, 16:831–849, 2004.
[29] A. Sutcliffe and B. Gault. Heuristic evaluation of
virtual reality applications. Interacting with
computers, 16:831–849, 2004.
[30] K. Vadas, K. Lyons, D. Ashbrook, J. S. Yi,
T. Starner, and J. Jacko. Reading on the go: An
evaluation of three mobile display technologies.
[31] H. A. van Veen, H. K. Distler, S. J. Braun, and H. H.
Bülthoff. Navigating through a virtual city: Using
virtual reality technology to study human action and
perception. Future Generation Computer Systems,
14(3):231–242, 1998.
[32] M. Woollacott and A. Shumway-Cook. Attention and
the control of posture and gait: a review of an
emerging area of research. Gait & posture, 16(1):1–14,
2002.
 Interaction with augmented reality

Pierre Ducher
Universität Passau
[email protected]

ABSTRACT brid UI. I will add Tactile UIs to this list, but they can be
In this paper, I will present the current techniques to interact seen as a subcategory of Haptic UIs, this will be explained
with Augmented Reality. Indeed, the interaction part in later in part 2.3.
AR is mandatory as per definition AR makes the added
information interactive. First, the introduction will tell you 2. OVERVIEW OF THE DIFFERENT INTER-
about what is different in interacting AR as opposed as usual FACE TYPES
computers. We will then go through the different ways of
To have a better idea of the different possibilities to in-
interacting with Augmented Reality, as many of them are
teract with augmented reality, here is an overview of the
usually not so well known. This goes from Tactile UIs, to
different HID. Table 1 presents different techniques to in-
aura UIs along with Haptic and Tangible UIs or also gaze
teract with virtual reality depending on their mobility and
tracking. The combination between those is also possible,
the way of interacting with them. The references are refer-
creating Hybrid UIs. Two main problems, among others,
ring to the cited solutions in the same order they appear
are introduced by the usual ways of interacting with AR.
on their respective lines. Nowadays, any computer with a
The Dimension Gap when interacting with a 3D content
web-camera or phone with an image sensor can be an AR
through a 2D interface or the screen concealment problem.
device. Especially smart-phones, which have cameras along
We will see why and how to solve those problems with new
with a touchscreen and other sensors such as a gyroscope,
or different kinds of interaction.
compass, GPS or accelerometer. That’s why most of the AR
systems make use of a touch-screen. This area is evolving
Keywords and we can see today projects such as ATAP Project Tango
Augmented Reality (AR), Interaction, Human Interface De- from Google [7] trying to integrate a depth sensor into mo-
vice (HID) bile devices. For Project Tango, the main goal is not direct
interaction from the user (i.e. using it to ”see” his hands)
but to create a map of the environment. Mapping the user’s
1. INTRODUCTION environment in 3D allows for a better integration of virtual
People today are used to the WIMP (Windows Icons Menus object.
Pointing) paradigm to interact with a computer. This is the
conventional desktop UI metaphor that we find in almost all 2.1 Tangible UI and 3D pointing
operating systems. But with augmented reality, we can not The first step toward a more natural user interaction is to
use this paradigm as it wouldn’t make sense to have win- have something both tangible and registered in 3D to manip-
dows floating in the reality. Even though this paradigm is ulate for the user. This way the user is not lost with abstract
not relevant any more, we still need to manipulate objects, concepts and can relate to already existing concepts. The
thus the need for a way to select, drag, resize, remove, add Studierstube project by Szalavári et al. [24] implemented
etc. objects. Indeed, Augmented Reality is per definition in- this principle by having a pen and panel being tracked by
teractive, so we need a way or multiple ways to interact with their system. The user sees the layer of augmented reality
it. User Interfaces and inputs are used to change the system with his see-through glasses. The pen is used to do any 3D
state and thus interact with it. In [14], the researchers are manipulation, with 6 degrees of freedom, and the panel is
categorizing interfaces into 7 different groups. They have here to display information, menus or options that would
Tangible UI and 3D pointing, Haptic UI and gesture recog- other be a problem to integrate with reality. The system
nition, Visual UI and gesture recognition, Gaze tracking, “projects” information on the panel, as if it was a 2D dis-
Aural UI and speech recognition, Text input and finally Hy- play. This system makes collaboration relatively easy. Users
can manipulate a 3D model with their respective pen and
• Pierre Ducher is a master’s student at the University will share the view of this 3D model. Their goal is to have
of Passau, Germany naturally integrated displays, represented by the panel, and
enabling the users to independently control the 3D model
• This research report was written for Advances in Em- from their viewpoint. Their experiment showed that this
bedded Interactive Systems (2014), Volume 2, Issue 4 kind of system is ideal to work collaboratively on synthetic
(October 2014). ISSN: 2198-9494 data, such as 3D curves.
More recently, we have seen this kind of user interface
 Fixed Handheld Wearable references
Tangible Studierstube, PlayStation 3 [24]
Tactile Smartphones (Hürst et al.) [12]
Visual 2D PlayStation 2 and 3 Smartphones (Hürst et al.) [12]
HoloDesk, Studierstube Tango Digits [11], [24], [19],
Visual 3D
MirageTable [3], [7], [13]
Gaze FreeGaze Occulus Mod, Google Glass [20] [5]
Hybrid Sublimate KITE, 3DTouch & Homer-S Google Glass [15] [22] [18]

Table 1: User Interfaces Categories

for instance the buzzing gloves of Buchmann et al. [4]. But

Haptic UI in general is usually bulky and difficult to use for
augmented reality, especially for mobile systems.

2.3 Tactile UI
Touch-screens with vibration could be considered the poor
man’s Haptic UI. This kind of input is only 2D, and the
haptic feedback is determined by whether or not the device is
integrating a vibrator. But this has the advantage of having
the input right in front of the display, making it easier to
interact with virtual elements than with a traditional haptic
device, such as a Phantom. Touch-screens are part of the
broader category of Tactile UIs.
This area could itself be split in several categories, it has
already been done in a survey [2], but we are not going to
spend that much time talking about them in the present
paper. We will only have an overview of what is existing.
Figure 1: SensAble/Geomagic PHANTOM Omni Haptic Human finger tips contain many sensitive nerves, this is only
Device1 , providing force feedback to the user. logical tu use this to give feedback information to the user.
Most of the tactile interfaces use an array small actuators to
“touch” back the user’s finger. Only the mechanical principle
with video games and the Nintendo Wii, where the user changes between the different devices, with motors, electro-
has a “remote” integrating an accelerometer and infra red static or piezoelectric technology, pneumatic, electrical etc.
camera. This system enable the gaming console to track to give sensation to the user’s fingertip. Of course tactile
the gesture of the user precisely while the user still has a interfaces can be used for blind people to read in Braille.
concrete object in his hand. Feedback can be given to the This is a nice way to have augmented reality for blind peo-
user with either vibrations or sound, also integrated in the ple along with Aural UI. Arrays with taller actuators are
controller. also used to produce a plane which can be deformed. This
is the case with MATRIX (A Multipurpose Array of Tactile
2.2 Haptic UI and gesture recognition Rods for Interactive eXpression) [21] (see figure 2) and more
To have a better feedback and a more realistic feeling when recently with Sublimate [15] (see figure 3). The latter com-
interacting with augmented reality, Haptic user interfaces bines an array of actuator with a see 3D see-through display,
have been introduced. The difference with these category thanks to a half-silvered mirror and shutter glasses. They
of interfaces is that the force is not only from the user to also use a “wand”, a Tangible UI, thus making this system
the device, but also from the device to the user. This inter- a Hybrid UI. The actuators are both used to give the user
faces are usually arm robots, for instance the Phantom by a feedback and to be manipulated by the user. In figure 3,
SensAble, see figure 1, which is a popular device for research the user can move the actuators to change the shape of the
purpose. surface. They also have a collaborative work use-case with
Hayward et al. [9] list and review Haptic interfaces and a “Multi-user geospatial data exploration”, where they use
devices. One of them is a joystick by Rosenberg et al. [23], tablets in addition to the rest in order to extend the active
used in their study to rate the user preferences on haptic workspace, interact and add layers of data.
virtual surfaces. To do so they had a joystick with two Tactile UIs are not limited to fingertips, and some devices
degrees of liberty manipulated by the user. They simulated to substitute vision have been developed [17], they are placed
different kind of feedback, either having a damper or a spring on the tongue are allowing disabled people to see again.
wall model. This kind of studies shows that no aspect should
be neglected when working to build a more natural user 2.4 Visual UI and gesture recognition
interface. Indeed, every detail can make the user experience When leaving the Haptic feedback aside, less cumbersome
better. user interface can be achieved. Visual UI for instance, tracks
Gloves have also been used with an added haptic feedback, the hands of the user. It can be done with a camera, when
doing so, the user’s hand are free and don’t have to ma-
1
http://www.bobspaller.com/phantomomni.html nipulate any device. The camera can be placed at different
Figure 2: MATRIX [21], Figure 3: Sublimate [15],
Multipurpose Array of 3D interactive plane with
Tactile Rods for Interac- actuators
tive eXpression, an array
of tactile actuators
Figure 4: Homebrew Oculus Rift Eye Tracker [5]

places. Externally, filming the user, around the neck of the

user, oriented toward his hands or around his wrist, has been the gaze of the user, we can obtain information about what
demonstrated with UbiHand [1]. he wants to achieve. Gaze tracking is done by having tiny
This category can be split in two, the 2D and 3D devices. cameras filming the user’s pupils. The difficulty is to cal-
Smartphones really often use their camera for gesture recog- ibrate it correctly and to filter out involuntary eye move-
nition but are limited to 2D whereas a device such as Ubi- ments. Google has a very simple gaze tracking technology
Hand [1] or Digits [13] can model in 3D the hand of the user. in his Google Glass where the screen activates only when
The latter aims to be used with mobile devices whereas it is the user is looking at it. But this can be much more pow-
not especially the case for UbiHand. However, both aim at erful, as it has been demonstrated with FreeGaze by Ohno
being low cost. Digits wants to be reproducible using only et al. [20], where a non-intrusive, fast calibrating (only two
off-the-shelf components. The advantages against 3D depth points) gaze tracking system is used for an everyday use.
cameras is that depth cameras are not yet precise enough to Their study shows that the system is viable and accurate.
track our finger, but we use our fingers every day to do com- James Darpinian2 recently built his own Oculus Rift Eye
plicated things. For this reason using fingers to do gestures, Tracker [5]. The Occulus Rift is a 3D head mounted dis-
and not only the arm or hand, is viable. Digits is really play usually used for Virtual Reality, but which can also be
precise and permits to do fine work on virtual objects. used for Augmented Reality. The problem when using a de-
Systems such as HoloDesk [11] or MirageTable [3] both vice such as this is that the user face is not visible, making
have demonstrated that the recognition accuracy is much usual eye tracking with a system such as FreeGaze impos-
better in 3D than in 2D. The latter makes use of a depth sible. James Darpinian resolves this problem by cutting a
camera, 3D glasses and a beamer to view the scene in 3D hole on top of the left eye socket in the Occulus Rift and
whereas HoloDesk also makes use of a depth camera but by placing there a PlayStation 3 Eye Camera. He place a
projects the image aligned with reality on a see through hot mirror, a mirror reflecting only infra-red light, in the
mirror. Although aligning the virtual layer with reality can Occulus to reflect the picture of the eye to the camera. The
be an immersive experience for the user, having this layer principle is illustrated in figure 4. This way of integrating an
in 2D can introduces problems, for instance the Dimension eye tracker with a head worn display could also work with
Gap of which we will talk later. For this reason, usage of a see through display usually used in augmented reality. By
3D display, for instance in MirageTable [3], can result in a using OpenCV, the pupil can be extracted in about 6ms per
better accuracy when manipulating virtual objects. This so- frame for a precision of 1/4 of a pixel. The user can stare
lution has the advantage of being gloveless and blurs the line at something displayed by the Occulus Rift with a precision
between the real and virtual world, but the user doesn’t have of 2 pixels. Although the Occulus Rift is fixed to the user’s
any feedback when his hand is touching a virtual object, in a head, James Darpinian still encounters problems with cali-
similar fashion to Haptic UIs. To lessen this inconvenience, bration, as when moving his head rapidly or when changing
Benko et al. have what they call a ”mirror view”, where facial expression, the Occulus slightly changes position and
real object are duplicated in front of the user, replicating the calibration is not good anymore.
the experience of a real mirror. This way, the user doesn’t
have a projected virtual object on his hand when trying to 2.6 Aural UI and speech recognition
manipulate it. Another natural way of interacting, especially between hu-
man, is by talking. Therefore, Aural user interfaces are to-
2.5 Gaze tracking day more and more used, especially with connected object
Gaze tracking can be one of the most powerful user inter- which can process the voice recognition in the cloud. For
face but also one of the most difficult to implement. Our instance, Google and Apple use speech recognition on their
gaze often reflects what we are thinking about in the real
world. When we look at something we certainly have an 2
http://jdarpinian.blogspot.de/2014/06/homebrew-oculus-
intention with this object. For this reason, by following rift-eye-tracker.html
mobile operating systems, the voice of the user is recorded,
sent to servers on the internet and the result comes back to
the smart-phone. On a hardware standpoint, only a cheap
microphone is needed.
Papers often make examples with an industrial applica-
tion. For instance with a maintenance engineer using AR to
work more efficiently. That’s also the case for Goose et al. [8]
who is suggesting to use a Aural UI for maintenance tech-
nicians. This application is a good example for the usage
of speech recognition. Indeed, factory workers or mainte-
nance technicians often have their hands occupied as they
have to work with them. Interacting with their hands is not
a solution. Their example is using a head-worn eye screen
and camera along with a headset with microphone. They
have specialized algorithms to recognize the equipment be- Figure 5: KITE Platform [22], hand-held AR device made
ing maintained, for instance pipes, and the worker can ma- with off-the-shelf hardware
nipulate the software via the Aural UI. The technician can
for instance ask the system to enlarge a specific part of the
picture or to recognize it and display it rotating in 3D. All of capabilities I expect from the project Tango, which has a
those actions can help the technician, without having him gyroscope, accelerometer, magnetometer and depth sensor.
to even use his hands. Although the Razer Magnetic Sensor is much more accurate
Difficulties arise when working in a noisy environment, than a gyroscope and magnetometer combined, this makes
which can happen often when doing maintenance in a fac- it very similar to KITE.
tory. Fortunately, noise cancellation techniques do exist,
using a second microphone to capture only the noise and re-
moving it from the first microphone audio. The main diffi- 3. PROBLEMS WHILE
culty may be when using speech recognition with augmented INTERACTING WITH AR
reality. Indeed, when a human talks about an object, ambi- All those interfaces aim at providing a better control over
guity can arise. AR applications. Unfortunately, three sorts of problems,
among others, can appear when using such a system.
2.7 Hybrid UI
Most important of all, Hybrid UI is not a category in itself 3.1 Dimension Gap
but a combination of categories. Indeed, what can be better One of them is what I call the Dimension Gap, AR is
against ambiguity when verbally describing an object than usually three dimensional and require six degrees of freedom
to point a finger at it? This is the principle of multi-modal while most of the HID are two dimensional. A touch-screen,
interactions. camera or mouse only have X and Y coordinates, while the
This type of UI has been very well illustrated with KITE [22] interaction might need a Z coordinate, for the depth. This
which has three different types of input. Their device is problem has been demonstrated in [12]. In this paper, users
made of a Windows tablet with a touch-screen, a Razer Hy- are asked to grab a virtual object integrated to the scene
dra magnetic tracker and Primesense Carmine short range captured by the camera, as illustrated in figure 6. They have
depth camera, see figure 5. This camera is the same tech- markers on their fingers so that the mobile device can track
nology embedded in the Kinect or Google’s project Tango. them. It appeared to be more difficult to grab the object
This setup is not really viable for a Handled one as it is this way than by touching it on the touch-screen, because of
heavy. User complained about its weight in the study. How- the lack of depth with the camera. Trying to interact with
ever, the goal was to demonstrate that such a Hybrid device a 3D space through a 2D camera is the same as trying to
could be possible to do with consumer devices. If it were to put the cap back on your pen with only one eye open. This
be integrated, it could be better done and weight less. They is frustrating for the user as he has try again multiple times
found out that the magnetic tracker is very accurate with a before succeeding.
less than 1 mm error and less than 1 degree for orientation
when being within 1.5 m from the transmitter. This trans- 3.2 Object selection and screen concealment
mitter is a huge drawback of the solution, as it transforms Another concern when using AR applications with a touch-
a potentially hand-held device to a fixed device. The ben- screen, especially with mobile devices, is that the user usu-
efits of the magnetic tracker is not only the accuracy but ally has to put his hand or finger on the screen thus covering
also a very reasonable processing requirement. This is espe- the object of interest or other important elements. This also
cially true when comparing it to a visual tracking solution, has been demonstrated in [12]. When the object the user
such as the Primesense depth camera. They only achieved wants to manipulate is hidden, it makes it more difficult to
a processing speed of less than 10 fps while working with move it to a desired position, scale it or rotate it for instance.
640x480 pixels, and this is with an Intel Core i5 and 4GB of One loses precision as one has to guess where the object is.
memory. Tablets usually have a lot less processing power,
being equipped with low power ARM CPUs. Although the 3.3 Lack of Haptic Feedback
system is not perfect, the user experience was much better This problem has been clearly identified by Hürst et al. [12].
than when using only one input. This permitted to have a When the user tries to grab a virtual object through a cam-
car racing game taking in the real world. This is the kind era view on the device, either 3D or 2D, in reality there is
 with a gloveless wearable sensor [13]. The advantage of this
solution is a much more accurate tracking of the users fin-
ger as well as the possibility to have his hand wherever he
wants and not in front of a sensor. But the drawback is that
the AR application needs to draw a 3D hand on screen to
represent the user’s hand, as we don’t have a camera.
For head worn displays and working in a somewhat fixed
fashion, Schmalstieg et al. [24] propose an interface with a
pen and a panel. The pen allows the user to do any move-
ment and operation a 3D mouse supports. They project,
with the head worn display, information both on the panel
and everywhere else, blending it with other objects. The
goal of the panel is to have a surface when manipulating
text or other data we usually use with the WIMP paradigm.
This combination of pen and panel makes it ideal for col-
Figure 6: A green marker (thumb) and a red one (index fin- laborative work. The user can interact in 3D and use a
ger) are used to track fingers and manipulate virtual objects paradigm he already knows (the pen and paper). However
on the board game. Source: [12] its limitation shows up when the user wants to manipulate
an object with its own hands, which are no tracked. The
system will not not the user’s intention. To overcome this
limitation, we would need an hybrid interface.
only air. This makes the confirmation of grabbing an object
we usually have in real life non existent here. 4.2 Alternatives
4. SOLVING THE PROBLEMS 4.2.1 Backside touch-panel and dual screen
An answer to the screen concealment problem can also be
4.1 3D Interface interaction a backside touch-panel. By moving the touch surface to the
First of all, the lack of dimension problem can be tack- back of the device, the fingers no longer the screen and the
led with a 3D sensor, instead of using a simple 2D camera, user can see everything that is on screen. This king of usage
some use a depth sensor, demonstrated by Hilliges et al. has been demonstrated by Sony with the PlayStation Vita
with HoloDesk [11]. This camera will generate a cloud of and the mini game ”Rolling Pastures” in ”Little Deviants”3 .
points, spread in the three dimensions. Using this we can In this game, the player can bump the ground in the game
know exactly where the hand of the user is or map the envi- by placing his finger on the rear touchpad and by moving
ronment more precisely. For instance, when the user wants it. This way, the player can still see the whole screen while
to grab a virtual object, like in [12], a depth camera will playing. Nintendo previously implemented another solution
enable the device to draw the virtual object in front of or on the Nintendo DS4 by both having two screen, one touch-
behind the hand of the user. Also, the user will not need to screen and one regular screen, and by using a stylus instead
wear markers on his fingers. With the depth, we can detect of a finger. The stylus being much thinner than a finger,
more reliably the shape of the fingers, thus detecting easily the user can see more of the screen and can also select an
when they are moving. In addition, when displaying a vir- object more precisely. Those two examples may not be used
tual object, the device can integrate it more realistically in for AR, but we can easily imagine it.
the environment with the help of depth information. With
4.2.2 Haptic UI and Tactile UI
conventional cameras, AR software usually require to have
a square marker on the surface we want to “project” the vir- Another type of touchscreen with haptic feedback could
tual objects, to be able to calculate the perspective. This be used to partially solve the problem of screen concealment.
requirement doesn’t exist with depth sensors as the sensor Like some devices described in [2], it could have a different
gives information about the plane the device is looking at. texture when the user has his finger on something interest-
Like demonstrated in [16], this makes blending virtual object ing, for instance, the object he tries to move. This way, even
with real objects much more easy. when the screen is occulted, the user can still feel what is
The usage of a back camera can be a solution to the screen under his fingertip. But the limitation is that the user can
concealment problem, however the depth remains important not read what is under his fingertip and that any part of
even when we don’t need it to grab an object. Indeed, it has the screen that is not visible but also not touch will remain
been shown by a user study in [10] that the user experi- equally occulted as before.
ence is better when displaying the user’s finger on top of Of course, this goes without saying that both Tactile UIs
the virtual button, for which we didn’t specially need depth and Haptic UIs solve the lack of haptic feedback problem.
information. According to the study, it makes the button This is self-explanatory for Haptic UIs and as for Tactile UIs,
pushing experience feel more realistic. We can easily see here the texture of the tactile surface can be changed when an
the use of depth information from a sensor to draw correctly object is selected, confirming the selection to the user.
the finger on top or under a button, thus saving computing
power for other tasks.
4.2.3 Aural UI
The usage of a depth sensor may be nice, but this is not 3
https://www.youtube.com/watch?v=JtaK6mjnTpY -
the only solution for a 3D interface interaction. Experimen- CES 2012 - Sony Playstation Vita Rear Touchpad
4
tations have been made with gloves [6] and more recently Nintendo DS - en.wikipedia.org/wiki/Nintendo ds
 Another alternative to solve the screen concealment and a
few other problems is to have a Aural UI and speech recog-
nition. By giving orders, the users doesn’t need to use his
hands at all, which comes very handy in some situation, for
instance in the maintenance field where a technician needs
his hands [8]. However, the compute power needed to under-
stand speech is much bigger than what is required even for a
depth sensor. Also, human languages are often ambiguous,
making it more difficult for a computer to understand us.
Finally, it may be socially unacceptable to be apparently
speaking alone in a public space and private informations
could end in the wrong ears.

4.2.4 Fixed System

Of course, on fixed AR systems, the problem of screen
concealment is less noticeable as screens are bigger than for
mobile systems. But another problem for fixed systems is
the need to be near the display to interact with it, as those Figure 7: The Homer-S technique for rotating an object [18]
display are not moveable. A solution is also to have a depth
sensor, such as the Kinect 5 , so that the user can interact at
any distance from the display. Also, a lot of fixed system use objects without moving, here the user needs to move around
a half-silvered, for instance Sublimate [15] or HoloDesk [11]. the virtual objects he his manipulating. Indeed, if he wants
The consequence is that they usually have the user manip- to move the object closer to him, he has to place his device
ulate under the half-silvered mirror, resulting in having the on top, under or on next to the virtual object. He can then
augmented reality layer on top of the user’s hand. This can proceed to translate it. This technique is only viable for
be disturbing if the system is not blending the virtual objects small objects and small AR environment, or if the user is
correctly (i.e. if the part of the object under the user’s hand willing to move around a lot.
is displayed). However, this solves the screen concealment To rotate objects, they have a technique called Homer-S
problem. which seems more viable. It is a complicated to describe
gesture which combines touching the screen and moving the
4.2.5 Visual Help device. The gesture is illustrated in figure 7. In their study,
An additional and artificial visual help can improve the user were more efficient when using Homer-S, it took them
user experience in most of the problem. For instance, for less time to complete the same tasks compared to 3DTouch,
the lack of haptic feedback problem, Hürst et al. [12] have but almost only for a specific task, which was to let a barrel
implemented a system indicating the interpenetration of two down an inclined platform. In average, there is no significant
objects and a visual confirmation of the selection. In their difference in performance among a broad type of tasks.
study, with a visual confirmation, the selection of an object
reportedly takes less time than without. 5. CONCLUSIONS
To help resolve the screen concealment problem, some sys-
As we have seen, although this area is being researched
tems use a deported view of what is under the user’s finger.
for a longer time we might think, AR is not a mature do-
This is also used in non AR applications on smart-phones
main yet. We have each year new technologies enabling us
when precision is required.
to provide better integration between virtual object and the
4.2.6 3DTouch and Homer-S real environments. New sensors and their miniaturisation
can help improve the user experience a lot. The most im-
Without adding any hardware to an existing smart-phone
portant thing seems to be providing an experience as close
or tablet, Mossel et al. [18] propose a novel way to interact
as possible from the reality, to have a natural way to interact
with 6 degrees of freedom with AR. This is done while using
with the system. For that, we have seen many categories of
the device one handedly, which can appear impossible at
input, we can control a computer using our hands, but also
first. Solution to manipulate the third dimension with a 2D
our voice or eyes. New inputs unlock new usages and make
touch-screen usually make use of multi touch gestures, but
easier the usage of augmented reality. This has been illus-
those are difficult to do one using the device with only one
trated with the Dimension Gape and Screen Concealment
hand. Here, what is proposed is to take into account the
problems. Using a depth sensor can for instance solve both
current position of the device, changing the meaning of the
of this problems. The user’s hand interact freely behind his
same gesture when the device is not in the same position.
device which can recognize easily his gesture in 3D. Also,
For instance, when the user slides his finger on the screen,
the screen is completely visible to the user as the gesture
it will translate the object along the x or y axis. But if the
are made behind the screen. Another point about having
device is lying on its back, then it will move along the z
a more natural feeling is the Haptic feedback. When inter-
axis, like if the user was watching the virtual object from
acting with objects in real life, we can touch them. Using
the top through a window provided by the device. This has
haptic devices or tactile surfaces we can recreate this feeling
some advantages but it also has drawbacks. For instance,
helping the user experience.
while with the usual AR interfaces the user can manipulate
Other solutions can cover partially those problems, such
5
http://en.wikipedia.org/wiki/Kinect as having a rear touch-panel and using it instead of the front
touch-screen. Visual help, Touch 3D and Homer-S are also limitations. International Journal of Virtual Reality,
a low cost alternative to expensive hardware needs. Indeed 9(2), 2010.
those solutions don’t need additional hardware and can work [15] D. Leithinger, S. Follmer, and A. Olwal. Sublimate:
with the same smart-phone or tablet the user already has. State-changing virtual and physical rendering to
Unfortunately, we have also seen that those solutions are not augment interaction with shape displays. In
perfect, as for instance the user’s hand doing 3D gestures can Proceedings of the SIGCHI Conference on Human
be tired very quickly and not having a feedback can feel not Factors in Computing Systems, pages 1441–1450, 2013.
natural. [16] S. Lieberknecht, A. Huber, S. Ilic, and S. Benhimane.
RGB-D camera-based parallel tracking and meshing.
6. REFERENCES 2011 10th IEEE International Symposium on Mixed
[1] F. Ahmad and P. Musilek. UbiHand : A Wearable and Augmented Reality, pages 147–155, 2011.
Input Device for 3D Interaction. Communications, [17] T. Maucher, J. Schemmel, and K. Meier. The
page 2006, 2006. Heidelberg Tactile Vision Substitution System. In
[2] M. Benali-Khoudja. Tactile interfaces: a International Conference on Computers Helping
state-of-the-art survey. Int. Symposium on Robotics, People with Special Needs (ICCHP2000), number July,
31:23–26, 2004. 2000.
[3] H. Benko, R. Jota, and A. Wilson. Miragetable: [18] A. Mossel, B. Venditti, and H. Kaufmann. 3DTouch
freehand interaction on a projected augmented reality and HOMER-S: intuitive manipulation techniques for
tabletop. In Proceedings of the SIGCHI conference on one-handed handheld augmented reality. Proceedings
human factors in computing systems, pages 199–208, of the Virtual Reality International Conference: Laval
2012. Virtual, page 12, 2013.
[4] V. Buchmann, S. Violich, M. Billinghurst, and [19] R. A. Newcombe, A. J. Davison, S. Izadi, P. Kohli,
A. Cockburn. FingARtips: gesture based direct O. Hilliges, J. Shotton, D. Molyneaux, S. Hodges,
manipulation in Augmented Reality. Computer D. Kim, and A. Fitzgibbon. KinectFusion: Real-time
graphics and interactive techniques in Australasia and dense surface mapping and tracking. 2011 10th IEEE
South East Asia, page 212, 2004. International Symposium on Mixed and Augmented
[5] J. Darpinian. Moon base: Homebrew Oculus Rift Eye Reality, pages 127–136, 2011.
Tracker, 2014. [20] T. Ohno, N. Mukawa, and A. Yoshikawa. FreeGaze: a
[6] L. Dipietro, A. M. Sabatini, and P. Dario. A survey of gaze tracking system for everyday gaze interaction. In
glove-based systems and their applications. IEEE Proceedings of the 2002 symposium on Eye tracking
Transactions on Systems, Man and Cybernetics Part research & applications. ACM, 2002.
C: Applications and Reviews, 38:461–482, 2008. [21] D. Overholt. The MATRIX : A Novel Controller for
[7] Google. ATAP Project Tango, 2014. Musical Expression. In Proceedings of the
[8] S. Goose, S. Sudarsky, and N. Navab. Speech-enabled International Conference on New Interfaces for
augmented reality supporting mobile industrial Musical Expression, pages 38–41, 2001.
maintenance. IEEE Pervasive Computing, 2(1):65–70, [22] T. Piumsomboon. KITE: Platform for mobile
Jan. 2003. Augmented Reality gaming and interaction using
[9] V. Hayward, O. R. Astley, M. Cruz-Hernandez, magnetic tracking and depth sensing. Mixed and
D. Grant, and G. Robles-De-La-Torre. Haptic Augmented Reality (ISMAR), 2013 IEEE
interfaces and devices. Sensor Review, 24(1):16–29, International Symposium on, (October):289–290, 2013.
2004. [23] L. Rosenberg and B. Adelstein. Perceptual
[10] M. Higuchi and T. Komuro. AR typing interface for decomposition of virtual haptic surfaces. Virtual
mobile devices. Proceedings of the 12th International Reality, 1993. Proceedings., IEEE 1993 Symposium on
Conference on Mobile and Ubiquitous Multimedia - Research Frontiers in, (October):46–53, 1993.
MUM ’13, pages 1–8, 2013. [24] Z. Szalavri, D. Schmalstieg, A. Fuhrmann, and
[11] O. Hilliges, D. Kim, S. Izadi, M. Weiss, and M. Gervautz. ”Studierstube”: An environment for
A. Wilson. HoloDesk: Direct 3D Interactions with a collaboration in augmented reality. Virtual Reality,
Situated See-Through Display. Proceedings of the 2012 3(1):37–48, Mar. 1998.
ACM annual conference on Human Factors in
Computing Systems - CHI ’12, pages 2421–2430, 2012.
[12] W. Hürst and C. Wezel. Gesture-based interaction via
finger tracking for mobile augmented reality.
Multimedia Tools and Applications, 62(1):233–258,
2013.
[13] D. Kim, O. Hilliges, S. Izadi, A. D. Butler, J. Chen,
I. Oikonomidis, and P. Olivier. Digits: freehand 3D
interactions anywhere using a wrist-worn gloveless
sensor. In Proceedings of the 25th annual ACM
symposium on User interface software and technology
- UIST ’12, pages 167–176, 2012.
[14] D. V. Krevelen and R. Poelman. A survey of
augmented reality technologies, applications and
Copyright Notes
Permission to make digital or hard copies of all or parts of this technical report
for personal use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage. The copyright remains with the
individual authors of the manuscripts.

30