2015 48th Hawaii International Conference on System Sciences

Enhanced Human-Computer Interaction for Business Applications on Mobile

Devices: A Design-Oriented Development of a Usability Evaluation Questionnaire

Maximilian Wich Tommi Kramer

University of Mannheim University of Mannheim
[email protected] [email protected]

Abstract—Over the last few years mobile technology has user needs and goals in the focus of the engineering process
gained enormous growth in the field of human-computer and enables us to develop satisfying applications for the user
interaction as it became an essential part of our society’s every- [4]. An essential determinant in that process is the regular
day life. Specific mobile characteristics such as availability,
computational power, or high-resolution displays make these
feedback of future users. In order to make this involvement
devices very useful, even in a business environment. However, efficient it is necessary to have valuable evaluation methods.
usability plays a crucial role when it comes to software design For mobile applications and especially for mobile business
for human-computer interaction with mobile devices. In order applications, there is a lack of proven methods supporting
to achieve a high level of usability in user interfaces of the usability engineering process in order to improve the
application software, it is essential to ensure usability quality
during the development process. Therefore, we have developed
human-computer interaction.
a specific questionnaire for evaluating the usability of mobile A widely used method for evaluating usability of in-
business apps as well as a corresponding web-based software formation systems is the satisfaction questionnaire, which
tool for simplifying the assessment. In this paper, we follow a is filled in by prospective users. Over the course of time
design science research approach and evaluate our designed a wide range of such questionnaires are developed, for
artifacts in expert interviews where we demonstrate the utility
and applicability of the questionnaire and tool.
instance the System Usability Scale [5] or the Usability
Metric for User Experience by [6]. However, these methods
Keywords-Usability Engineering, Human-Computer Interac- assess the usability of a system in a generic way and do
tion, Mobile, Design Science
not focus on particular components of a user interface, like
typography, aesthetic, layout, or terminology. Consequently,
I. I NTRODUCTION it is difficult to draw inferences from the result of the
When Steve Jobs introduced Apple’s first generation of evaluation to the components of the user interface that have
iPhones in 2007, it was impossible to imagine what impact to be improved. In addition, they are designed for infor-
this innovation would have on daily life in society. Six mation systems in general and not specifically for mobile
years later, smartphones are every-day objects and it is hard applications. Hence, the objective of the paper is to develop
to think of a life without them. From 2010 to 2012 the an evaluation questionnaire for assessing single components
sales figures increased from 305 million devices to 722.5 of a mobile user interface regarding their usability in order
million and for 2017 1,733.9 million sold smartphones are to meet the requirements of the current mobile business
forecasted by IDC [1]. However, such an increase will not technology. Such a specific questionnaire helps to identify
remain without consequences for desktop computers and usability problems faster. Moreover, it will improve the user
laptops. Considering the new generation of mobile and smart experience during the software development process.
devices, 87% of all connected devices (desktop computers, This research is embedded in the design science research
laptops, tablets, and smartphones) sold in 2017 will be methodology according to [7]. Section 2 provides the theo-
tablets or smartphones [2]. retical background of usability evaluation questionnaires by
The huge success of the new technology can be mainly revealing characteristics of mobile devices and mobile busi-
ascribed to the devices’ high degree of usability and user ness applications, existing usability engineering processes,
friendliness [3]. Seeing that comprehensive functionality and as well as state-of-the-art evaluation methods in this do-
user-friendly human-computer interaction are no contradic- main. The requirements for a user-friendly design of mobile
tion , people began to use mobile devices more and more business applications are derived from usability theories and
extensively in their everyday life. Such an enormous trend user interface guidelines in section 3. After assembling the
had an impact on the development process of the products requirements, section 4 describes the development of the
as well. To reach a high level of usability it is inevitable to questionnaire and the corresponding software tool based on
involve future users as soon as possible in the development these insights. In section 5, the results of the empirical
process. Additionally, the user-centred design approach puts evaluation of the questionnaire and the software tool are

1530-1605/15 $31.00 © 2015 IEEE 472

DOI 10.1109/HICSS.2015.63
presented and discussed. Section 6 concludes the paper with [17], [18], Integration [19], Context Awareness [18], Secu-
a summary of the results and future work prospects. Hence, rity [17], [20].
the structure of this paper addresses the rigorous develop-
ment and evaluation of two artifacts (the questionnaire and B. Usability Engineering
the corresponding software tool) according to the principles We already know that the success of mobile devices and
of design science. The evaluation is conducted with user mobile applications is strongly related to their high degree
interface experts. of usability [3]. But there is still the question how to reach
II. L ITERATURE R EVIEW this objective.
One solution is described by ISO 9241-210: ”Ergonomics
The following section will provide a definition of mobile
of human-system interaction Part 210: Human-centred
devices and mobile business applications in order to get a
design (HCD) for interactive systems” (successor of ISO
better understanding of the unit of analysis. Additionally, the
13407). The standard based on the centred design paradigm
existing usability engineering process and related work are
from the 1980s provides a guidance on how to manage
presented and discussed.
the development process to create user interfaces with a
A. Mobile Technology comfortable user experience and a high degree of usability
[21].
In the context of the paper ”mobile devices are those
A crucial part of ISO 9241-210 is the specification of the
devices that are used to connect to mobile services” [8, p. 42]
iterative development cycle. The standard defines five main
like laptops, personal digital assistants (PDAs), smartphones,
activities which are conducted sequentially to achieve the
tablets, and mobile phones [9]. These types of devices
goals of HCD. Four of them form a cycle being repeated
distinguish themselves through the following characteristics:
until the product fulfils the required usability objectives.
Performance [10], [11], Data Input [10], [12], Connectivity
[10], [11], Mobility [10], [11], [13], Context Awareness [14], The development process starts with the planning activity.
and User Interface [10]. The goal of this part is to integrate the HCD processes
into the overall system development process. The next
As a consequence of the tremendous progress in the
activity, which is also considered the first one of the cycle,
field of mobile technology in the last couple of years, the
deals with understanding and specifying the context of use.
capabilities of smartphones and tablets have increased vastly.
Following this, the user requirements are specified. After the
They offer more and more functionalities that were only
identification of the requirements, the insights are used to
available on desktop computers or laptops a few years ago.
produce design solutions. Subsequently, the user-based as-
Furthermore, they provide several advantages of the classical
sessment is carried out to evaluate the developed prototypes
mobile phone (feature phone) and an intuitive handling [15].
against the user and organizational objectives. The early
The new generation of mobile devices smartphones and
and regular evaluation offers two essential advantages. On
tablets strongly differs from laptops and feature phones,
the one hand, changes are implemented before it becomes
although their functionalities considerably overlap. There-
disproportionally expensive. On the other hand, the demands
fore, it is crucial to consider their characteristics during the
and needs of the users will be under better consideration
usability engineering process in order to create a satisfying
because of the substantial and regular feedback [22]. The
user experience.
results of the evaluation provide a refactored basis and new
Before dealing with mobile business applications, it is
insights for a further iteration of the development cycle. This
necessary to define ”mobile business” (m-business). Unfor-
procedure is repeated until the developed system ”meets the
tunately, the literature does not provide an internationally
user requirements” [23, p. 11].
accepted definition of this term [16]. However, Koenigstorfer
was able to identify the main characteristics of mobile The evaluation activity is the part of the HCD that this
business by analyzing published definitions. Firstly, the paper deals with. In the following subsection an overview
services provided in the context of mobile business are of the different evaluation methods is provided and it is
available everywhere the user has access to his device. explained why a demand for a questionnaire assessing the
Secondly, these services contain business processes between usability of mobile devices exists.
all market participants, who could be consumer, businesses
C. Evaluation Methods for UI Design
or governments [16].
After defining mobile business we are able to determine As mentioned above, ISO 9241-210 focuses on how to
the unique features of mobile business applications. Differ- manage the HCD processes, but it does not contain methods
ent terms and classifications are discussed for these features or techniques for the activities. To fill this gap and to
in the literature. However, if you consider them as a group, more easily apply the HCD approach, Magurie published
they can be described as the following: Ubiquity [17], [18], a paper with a comprehensive collection of methods which
Accessibility [17]–[19], Convenience [18], Personalization can be used during the development cycle [22]. Furthermore,

473
he suggests the following methods: ”Participatory evalua- since 2007. Therefore, they no longer satisfy all require-
tion”, ”assisted evaluation”, ”heuristic or expert evaluation”, ments of the current generation of mobile business applica-
”controlled user testing”, ”satisfaction questionnaires”, ”as- tions. Nevertheless, the current state of research creates a
sessing cognitive workload”, ”critical incidents”, and ”post- good basis for the development of such a questionnaire.
experience interviews” [22, p. 590]. According to Maguire,
it is not necessary to apply all methods during development. III. UI D ESIGN R EQUIREMENTS FOR M OBILE D EVICES
The methods should be selected on the basis of project goals, The goal of this section is to determine the requirements
available resources and progress of the development. Since for the user interface design of mobile business applications.
the goal of the paper is to develop a questionnaire to assess They serve as a foundation for the development of the
the usability of mobile devices, we will have a closer look usability evaluation questionnaire, described in the next
at the satisfaction questionnaires. section. The requirements analysis has proceeded as follows.
This kind of questionnaire ”captures the subjective im- Starting with an analysis and comparison of the Android
pressions formed by users based on their experiences with and iOS design guidelines, we identify user interface com-
a deployed system or new prototype” [22, p. 681]. The ponents that are characteristic for the current generation
advantage of the method is that it can be carried out in of mobile applications and that are essential for usability
an easy and inexpensive way compared to other methods. evaluation [30], [31]. These insights are extended by addi-
Over the course of time, many questionnaires were carefully tional guidelines and theories from the literature that address
developed for assessing the usability of information sys- user interface design and development. In the second step,
tems, like the Questionnaire for User Interface Satisfaction the requirements for the component design are formulated
(QUIS) [24], Software Usability Measurement Inventory with the aid of the literature. To consider all aspects of
(SUMI) [25], the After-Scenario Questionnaire (ASQ), the usability, we ensure that the set of identified components and
Post-Study System Usability Questionnaire (PSSUQ), the their requirements cover all usability principles described
Computer System Usability Questionnaire (CSUQ) (all three by [27]. Additionally, the characteristics of mobile devices
by [26]), the System Usability Scale (SUS) [5], and the Us- and the unique features of mobile business applications are
ability Metric for User Experience [6]. The common factor included to obtain a valid result. Some of the guidelines
of these questionnaires that they are generic and therefore and theories consider older generations of mobile devices
they assess the usability of a system on an abstract level. and applications. Therefore, it is necessary to prove their
The advantage of a generic approach is that it makes the validity for our context. In the following subsections a short
questionnaires applicable for almost all information systems. summary of the requirements is provided.
However, it is difficult to identify usability problems and
errors in the user interface design, due to the generality of the A. Navigation and Organization
questions. That is one reason why usability questionnaires The first category of design requirements is ”Navigation
for specific systems are developed, like Website Analysis and Organization” dealing with how a mobile business
and Measurement Inventory (WAMMI). With the arrival of application should be structured and how the screens of the
mobile technology a demand for such questionnaires also application should be organized to create a pleasant user
emerged in this area. In 2006 Ji et al. as well as Ryu and experience. The application has to be designed so that the
Smith-Jackson published independently from each other a user can get the required information with minimal effort
questionnaire to evaluate the usability of mobile phone user [31]–[34]. To reach this, the navigation concept has to be
interfaces. The specification to the device type allows them intuitive, the differences between navigation and content
to ask questions being more specific and closer to technology elements have to be clear and the information has to be
than the generic approaches. As a result, the success rate of presented in a way supporting the recognition and processing
identifying usability problems could be increased in both of the information [30], [31], [34]. Also, the user should
cases [27], [28]. Since both questionnaires mainly address always be able to undo the last steps [30]–[35].
consumer products and applications, and not business appli-
cations, [29] developed MoBiS-Q one year later. Eventhough B. Terminology and Wording
if MoBiS-Q was designed for mobile business applications, Another crucial part of the design requirements is ”Ter-
the degree of abstraction in the questions is similar to that minology and Wording”, since a large proportion of the
of the generic questionnaire. Thus, it is more difficult to communication between user and application takes place via
draw inferences about considering the weaknesses in the texts. In this way, for example, the application gives instruc-
usability of the user interface compared to the other two tions to the user, informs him about its status or provides
questionnaires for mobile phones. content. As a general rule, the language should be similar
To summarize, the benefits of specific questionnaires are to the one used by the target user group. All words or texts
obvious for the development of user interfaces, but there used in a mobile context should be specific and precise, but
have been enormous strides in the field of mobile technology still concise because of the small screen size and the external

474
distractions. Furthermore, spelling and grammar should be applications [10], [33], [34]. Besides that, developers have
correct, the terminology consistent, and the abbreviations to consider the behavior of the application during changes in
clear [30], [31]. In terms of instructions, error messages, the device orientation and the start and stop behavior of the
help, and tutorial the texts should be informative and helpful application. Both are components influencing the usability
but not excessive or even offensive [30], [34], [36]. [31], [33], [38].

C. Interaction and Feedback F. Performance and Reliability

In this subsection the focus is on the elements which are The last category covers the performance components
necessary or helpful for the interaction between the user considering the computational power and the connectivity.
and the mobile business application. Especially in context of Even though if the mobile devices have performance lim-
mobile devices, data input is a critical success factor because itations, short reaction times are still essential for a good
the form factor aggravates the manual data input [33], [34]. usability [30], [31], [33], [39]. Another aspect which should
Due to the technological progress, mobile application can be automatically handled by the application is connectivity
make use of elaborate animations and transitions. However, issues. The user should be informed about changes and
they should only be used to support the user’s interaction possible restrictions, but he or she should not have to
with the device [31], [33]. This is also the case for gestures. become active to solve the problem [33], [35], [39]. In
They have to be easy to learn and intuitive to use [34], [37]. addition, an application should provide a synchronization to
Furthermore, the user should always be informed on what allow a convenient usage, if the service is used on different
the application is doing: is it running? Is it downloading devices or platforms [31], [34]. Security and privacy are
or similar? It gives the user the feeling of controlling the the last topics that have to be addressed in this category.
application. [30], [31], [35], [38], [39] Another component in Implemented poorly, like too complex security mechanisms,
this category is messages interrupting the normal workflow these requirements can cause a bad user experience and
of the user, like notifications and modal windows. They consequently discourage the user from using the application
should be well-considered before being used, otherwise, the [10], [38].
user recognizes them with reduced attention [30], [31], [34], The numerous requirements show that many aspects exist
[39]. Summing up, all these elements serve to facilitate the that have to be considered by the user interface designer
interaction between user and device. and developer of mobile business applications. As a conse-
quence, it is useful to develop a comprehensive and clear
D. Visual Design and Aesthetics questionnaire covering all these aspects. With the manual
According to [33], [30], and [31] an aesthetic appearance evaluation of paper-based surveys being time-consuming and
strongly improves the usability of the system. To create an expensive, it is a logical consequence to provide a software
aesthetic user interface, several aspects have to be consid- tool to conduct and analyze surveys using this questionaire.
ered. First of all, the design should be consistent to appear
professional and to provide the user additional orientation IV. I MPLEMENTATION D ETAILS
in the application [10], [31], [33], [39]. In addition, the This section considers the development of both artifacts:
typography and color scheme should be adapted to the the questionnaire and the corresponding software tool. In
content of the application and the characteristics of mobile the first part the derivation of the questions from the design
devices [30], [31], [40], [41]. Due to the limited screen, size requirements is elaborated. Afterwards, an insight into the
images and icons are often used to communicate with user. development of the software tool, called mugram (mobile
But it has to be ensured that the images and icons are well usability histogram), is provided.
discernible and easy to understand [30], [31], [34]. In the
context of business application, branding is an important and A. Development of the Questionnaire
critical topic. Too much or too intrusive branding strongly The questionnaire design is heavily oriented towards the
impairs the usability [31]. requirements described in section 3. It is structured in the
same manner and the questions are directly derived from the
E. Adaptability and Integration aspects addressed by these guidelines. Table I contains six
This category deals with the customizability of the appli- questions from the questionnaire. 100 of the 104 questions
cation and how it interacts with other applications and the are closed-ended questions with a five point Likert scale.
operating system. The usability of the initialisation process This means that a question contains a statement and a scale
already plays an important role. If it is too elaborate, the which allows the user to state how far he or she agrees or
user will close the application and is most likely not to disagrees with the statement. One advantage of this question
use it again [34]. Furthermore, the user should be able to design is to make the results comparable in contrast to
personalize the application to enhance his or her workflow open questions. Furthermore, the scale offers the user more
and to share information easily with other users via similar selection options than a simple yes-no question. The user

475
 Table I
E XTRACTION OF QUESTIONS FROM CATEGORY NAVIGATION AND O RGANISATION

Question strongly disagree strongly agree

I have to touch too many buttons to access to the required information.     
It is often difficult to distinguish between navigation and content elements.     
The gestures differ strongly from what I am used to.     
The combination of colors conveys a feeling of harmony to me.     
The images support the recognition of the content.     
The latency times are too long and disturb the work flow.     

also has the possibility to give a neutral answer and thus degree of usability and that it is as platform independent as
the result is not distorted by false answers, if he or she possible in order to facilitate its usage.
is indifferent. To get a meaningful result from this type 1) Technical Aspects: Due to the requirements mentioned
of questions it is necessary to stick to some rules. First above, mugram was implemented as a web application. In
of all, simple and unambiguous words have to be used comparison to desktop applications or native applications
to make the questions readily understandable for the end for mobile devices, a web application has the advantage of
user [42]. Secondly, the word ”not” should be avoided in the platform independence. It can be used on every device
a statement because the inversion aggravates the answering with a modern browser without developing platform specific
of the question for the user [42]. However, the developed client software. Furthermore, the centralized concept of
questionnaire contains two questions with a ”not”. But it was web applications enables the conduct of parallel evaluations
only used to emphasize the main statement of the question and supports the collaboration in the development team
which does not include a ”not”. Furthermore, research has compared to standalone solutions.
shown that users rather tend to agree with a statement than While HTML 5 and JavaScript are used for the presenta-
to disagree with it. To compensate this effect it is recom- tion tier of the application, the logic tier was implemented
mended to address an aspect with a positively and negatively with PHP. For the persistence of the data a MySQL database
formulated question [42]. Since the objective was to create a server is deployed on the data tier. It should be mentioned
lightweight questionnaire, we avoided such phrases. Instead, that the separation between the logic and data is not clear.
the problem is handled during the processing of the results, Due to performance aspects and the scalability of the system,
which is explicitly explained in context of the development the processing of the survey data is mainly executed by the
of the software artifact. However, it was ensured that the database server.
number of positive and negative statements is equal within To create a comfortable user experience on the different
the categories to obtain a valid mean value. Besides the supported device types, the user interface of the application
closed-ended questions the questionnaire contains a few is based on the Bootstrap 3 framework developed by Twitter.
open questions to give the users the possibility to express This framework, written in HTML, CSS, and JavaScript, fa-
their perception, impressions, and criticism. cilitates the development of responsive and touch-optimized
Additionally, the ten questions from the SUS were in- web applications. To make use of the entire functionality
corporated into the questionnaire to provide a clue for the of Bootstrap, it was necessary to incorporate the JavaScript
analysis of its results. These questions allow calculating a framework jQuery. In addition, Bootstrap was extended by
score which states how usable the entire system is. Thus, bootstrap-datetimepicker, a plugin which provides a more
the development team does not only obtain an individual user-friendly method to select dates and times. To simplify
assessment for the user interface components, but also an the validation of form input and to display static overlay
aggregated measurement for the entire system. There are two elements, the jQuery plugins Validation and Clingify were
reasons why the SUS and not another scale were selected integrated. Since none of these libraries provides chart plot-
for this purpose. It is simple and concise. Beyond that, [43] ting functionalities, the charting library Highcharts, which
validated its reliability and validity by an empirical analysis is written in HTML5 and JavaScript, is used in version 3.0.
evaluating data from nearly one decade. 2) Implementation Aspects: The mugram tool was imple-
mented in order to meet the requirements of an evaluation
B. Development of the Software Artifact tool for the development of mobile business applications and
Since the conduct and analysis of paper-based surveys is to make the defined questionnaire practically usable.
expensive, the software artifact mugram was developed for In order to allow an intuitive operation and to manage nu-
the usability evaluation questionnaire, in order to support merous survey rounds, the backend of mugram is organized
the evaluation process and to reduce the inhibition level of in projects and rounds. For every mobile application that
using it. Therefore, it is crucial that the tool exhibits a high is evaluated with mugram the user creates a new project.

476
 Figure 3. Backend: (a) detailed results and (b) comparison of two rounds
Figure 1. Frontend of mugram: questionnaire

the distribution of the answers and a grade (cf. Figure 3a).

Although the grade (between A and F) is only calculated
from the average value, its usage can be ascribed to two
reasons. In the first place, there are positively and negatively
formulated questions, which means that sometimes 1 is the
best result and sometimes 5. To facilitate the interpretation
of the results, the average values are normalized by the
usage of grades. In the second place, the grades allow
compensation for the acquiescence effect that user rather
tend to agree with a statement than to disagree with it. [42]
suggests a value of 10% to describe this effect. Therefore,
the interval lengths for the grades are not commensurate,
but the interval covering ”strongly agree” is 10% shorter
than the one covering ”strongly disagree”. The lengths of
the other three intervals are also adjusted according to the
Figure 2. Backend: dashboard of mugram harmonic sequence. However, the acquiescence effect has to
be regarded only in context of the questions. For the grades
of sub and main categories, the lengths of the intervals do not
During this process the user is able to adapt the scope have to be adjusted, because the same number of positively
of the questionnaire according to the functionalities of the and negatively formulated questions already compensate the
application. If an application, for instance, provides gestures effect. The answers for open questions are hidden in a
for the interaction, the user can enable this function to collapsible container.
include the corresponding question to the questionnaire. Since the development process of user interfaces is an
Within a project there are rounds which represent survey iterative one, it can be interesting for the development
rounds. It is envisaged to create an individual round for team to see the alterations between different versions of
every version of the application. The reason for that becomes the mobile application. Therefore, mugram allows the user
obvious, if the compare feature is explained. For every to compare up to three rounds within a project with each
round there is a specific link which forwards someone to other (cf. Figure 3b). The feature can be also used for A/B
the questionnaire. testing - an evaluation method that allows you to compare
After the conduct of a survey via a form depicted in two versions of a design and to see which performs better
Figure 1 the backend provides the possibility to analyse the [44].
results. The first tab of the analysis provides an overview
of the results in the form of a dashboard (cf. Figure 2). It C. Usage of the Questionnaire and the Software Tool
informs the user about the SUS score, the five best and worst The target group of the questionnaire and the software
questions and a summary of the result on the main category tool mugram are user interface designers and developers
level. The other tabs contain a detailed analysis for each of mobile business applications. They are supposed to use
question. For the closed-ended question the analysis consists these tools to improve the usability of their applications.
of the average value, the median, the standard deviation, To achieve this, the participants of the surveys based on

477
the questionnaire have to be potential users of the applica- In terms of completeness, the questionnaire has an average
tions. In addition, it is necessary to have a fully-functional degree of agreement of 3.8. According to the experts, the
prototype. Since the questionnaire is comprehensive and questionnaire is very comprehensive, but there are some
it is necessary to test an application extensively before usability aspects which are not addressed yet. These include
completing the questionnaire, it is primarily suitable to use questions dealing with the emotional perception of the user
it in observed field, laboratory tests or with paid testers of interface, ”issues around the device itself” and the quality
the company itself. In such situations the testers are willing of translations within the application.
to invest the effort to carefully complete the questionnaire. With regard to usefulness and validity, whether the results
Due to the amount of questions, it is not recommended to were helpful for the development of mobile business applica-
use the questionnaire in form of a classical online survey tion, the item obtains an average degree of agreement of 4.0
which is conducted without supervision. and the item, whether the results could improve the usability
V. E VALUATION AND I MPLICATIONS of such an application, receive 4.2. Besides this positive
feedback, some experts mentioned that the questionnaire is
This section illustrates the evaluation of both developed
also an extensive item pool which can already be used as a
artifacts. The goal is to evaluate the extent to which the
guideline during the early stages of development.
objectives for the questionnaire and the software tool are
fulfilled. For this purpose the questionnaire is checked for Despite of the usefulness and the validity, the experts did
its completeness, validity, and usefulness and the software not agree on the accelerating impact on the development
tool for its usability and usefulness. process by the questionnaire (average degree of agreement
2.8). This can be ascribed to two reasons. Firstly, it was
A. Evaluation Design mentioned that the completion of the questionnaire costs
As an evaluation method, expert interviews with user too much time. Secondly, one expert explained that such
interface designers and mobile application developers were a questionnaire could not be used until a fully-functional
chosen. It was taken into account to apply the evaluation prototype is available. Therefore, it would not accelerate
procedure which has been used for the usability checklist the actual development process. Nevertheless, the entire life
of [27]. They conducted a conventional usability testing to- cycle of a mobile business application could benefit from
gether with their developed usability checklist and compared the results of the questionnaire. The statement, whether the
the results in relation to the identified usability problems. In experts would use the questionnaire for the development,
this way, the checklist could be evaluated in the context in obtains an average degree of agreement of 3.6. However,
which the tool was used later. But there were some reasons it has to be mentioned that both experts who gave a 3 for
against this kind of procedure and for expert interviews. First the question justified their answer with usability aspects not
of all, the design of the questions and the applicability of being addressed. The result for the question concerning the
the answers should be assessed by usability experts. Sec- level of difficulty is not meaningful. Its average degree of
ondly, the corresponding software tool should be evaluated. agreement is 3.4 (in this case less is better). However, the
Since the users of the tool are user interface designers and standard deviation has a value of 1.34. Therefore, the result
developers, it could not be tested by conventional users of cannot be considered statistically convincing.
mobile business applications. For these reasons, the decision Beyond the aspects mentioned above, there are some
was taken in favor of the expert interviews. further aspects which the experts especially liked about the
The expert interview consists of two parts. The first one
questionnaire. Three of these are its appropriate structure,
deals with the questionnaire, the second one with the soft-
its comprehensiveness, and the question design. In addition,
ware tool. Every part contains two sections: A quantitative
two experts commended the end-user oriented language of
section with closed-ended questions to make the interviews
the question. Another expert considered the integration of the
comparable and a qualitative one with open questions to
SUS positively, since it would increase the reliability and
collect the objections and the improvement suggestions
validity of the results. Concerning possible improvements,
of the experts. For the closed-ended question a five-point
some experts suggested reducing the amount of questions,
Likert scale was used (1: ”strongly disagree” to 5: ”strongly
respectively developing a short version of the questionnaire
agree”). Overall, five interviews were conducted with user
for brief surveys. Furthermore, some of them wished for
interface designers and mobile application developers of
more free text answers. But this would increase the effort of
two start-ups and a large company of the service sector in
completing the questionnaire and impair the comparability.
Germany.
In summary, the evaluation shows that the questionnaire
B. Results for the Questionnaire is a helpful tool for the development of mobile business
The summarized results of the closed-ended questions for applications and it can positively influence the usability.
the first part of the evaluation are shown in the upper part Moreover, it provides some interesting insights for the
of Table II. improvement of the questionnaire.

478
 Table II
E VALUATION RESULTS FOR THE CLOSED - END QUESTIONS

Questionnaire (1 = strongly disagree, 5 = strongly agree ) AVG SD

Q1: I think that the questionnaire addresses all aspects being relevant for the usability of a mobile application. 3.80 1.30
Q2: In my opinion the results of the questionnaire are helpful for the development of mobile applications. 4.00 0.71
Q3: I think that the results of the questionnaire can accelerate the development process of mobile applications. 2.80 0.84
Q4: In my opinion the results of the questionnaire can improve the usability of a mobile application. 4.20 0.45
Q5: The questionnaire is too sophisticated to be answered by the end user. 3.40 1.34
Q6: I would use the questionnaire for the development of a mobile application. 3.60 0.55
Software Tool - mugram (1 = strongly disagree, 5 = strongly agree )
Q7: I thought the system was easy to use. 4.40 0.49
Q8: I found the system unnecessarily complex. 1.60 0.80
Q9: Mugram facilitates the conduct and analysis of a usability survey. 4.00 0
Q10: The analysis of the results is sufficient. 4.00 0.89
Q11: The comparison feature of the results is helpful. 4.60 0.80
Q12: I would use the tool for the development of a mobile application. 3.80 0.40
AVG: average; SD: standard deviation

C. Results for the Software Tool since the numbers 1 to 5 are associated with grade in
many countries and accordingly influence the user in his
The second part of the evaluation focuses on the software
or her answer. Another improvement suggestion is to make
tool mugram. The second part of Table II shows the results
the questionnaire more customizable. It should be possible
for the quantitative questions.
to enable and disable entire sub categories. Overall, the
After the adjustment of Q7s and Q8s scale, the average questionnaire should be adaptable for every round but not
rating for the system’s usability obtains a value of 4.4 on a for every project. The purpose behind that is to customize
scale between 1.0 and 5.0, which is a very good result. The the rounds according to the changes in versions of an appli-
adjustment is necessary since the item Q8 is reversed. cation. Beyond that, the experts required more interfaces:
With regard to the range of features, mugram achieved a REST API for the automated conduct of surveys and
similarly good results. All experts agreed (4.0) on the item for the integration into management dashboards, a sharing
that mugram facilitates the conduct and analysis of a usabil- function of the results for social networks like Google Plus
ity survey. The item considering the scope of the analysis or Yammer, and an export function of the raw data to conduct
obtained an average degree of agreement of 4.2. However, further analyses with SPSS, Stata or R.
it has a relatively large standard deviation of 0.89, which In short, the software tool mugram received very positive
indicates that the experts have different requirements for the feedback from the experts. The ease of use, the visual design,
analysis of the results. A feature which was commended by and the provided functionality have convinced most of the
four experts is the comparison feature of the rounds. The experts that they would use it for the development of their
corresponding quantitative item has an average degree of mobile business applications, if some smaller features were
agreement of 4.7. The item ”I would use the tool for the implemented.
development of a mobile application.” obtained an average
agreement of 3.8. While almost all experts gave an ”agree” VI. C ONCLUSION
(4.0) for this item, one of them gave a ”neither agree
nor disagree” which she warranted with the missing export A. Summary
function for the raw data. The objective of this paper is the development of an
In addition to the compare feature, there are some more evaluation questionnaire to assess single components of
aspects or features which were commended by the experts. a mobile user interface regarding their usability in order
One of them is the platform independence which allows to meet the requirements of the current mobile business
using mugram on every device with a modern web browser. technology. The intention behind that was to create a tool
Furthermore, the clean and aesthetic design of the user which facilitates the development process, to increase the
interface was pleasing to the experts; above all, the visual usability of mobile business applications.
presentation of the results with the dashboard and different According to design science research, a literature review
types of diagrams. But there were also some points of was conducted in advance, followed by the development
criticism, respectively for feature requests. Firstly, the radio of two artifacts and their evaluation. The literature re-
buttons of the questionnaire should not contain numbers, view exposed that there is no comparable questionnaire

479
for evaluating the usability of mobile business applications. [3] H. B.-L. Duh, G. C. Tan, and V. H.-h. Chen, “Usability
Nevertheless, it seems to be a conventional technique to use evaluation for mobile device: a comparison of laboratory
questionnaires during the software development to collect and field tests,” in Proceedings of the 8th conference on
Human-computer interaction with mobile devices and ser-
feedback from future users. Based on the insights gained vices. ACM, 2006, pp. 181–186.
from other usability questionnaires and guidelines for the
design of mobile user interfaces, the questionnaire and the [4] D. Saffer, Designing for interaction : creating innovative
corresponding software mugram were developed. To eval- applications and devices. Berkeley, Calif. : New Riders,
uate both artifacts, five usability experts were interviewed. 2010.
According to the results of this evaluation, both the question-
[5] J. Brooke, “SUS-A quick and dirty usability scale,” Usability
naire and mugram are helpful for the development of mobile evaluation in industry, vol. 189, p. 194, Sep. 1996.
business applications and can improve the usability of these
applications, which are the objectives of this paper. This [6] K. Finstad, “The Usability Metric for User Experience,”
statement is emphasized by the fact that most of the experts Interacting with Computers, vol. 22, no. 5, pp. 323–327,
would use mugram for the development of their mobile ap- Sep. 2010.
plications. Overall, the experts exposed small weaknesses of
[7] A. Hevner, S. March, J. Park, and S. Ram, “Design science
the artifacts and contributed some improvement suggestions in information systems research,” MIS quarterly, vol. 28,
for further development. no. 1, pp. 75–105, 2004.

[8] P. Tarasewich, “Issues in Mobile E-Commerce,”

B. Future Work
Communications of the Association for Information Systems,
The chosen method of expert interviews for evaluating our 2002.
questionnaire and tool lacks validity since it has not been
[9] L. Gorlenko and R. Merrick, “No wires attached: Usability
based on a real application and real end-users. But this is challenges in the connected mobile world,” IBM Systems
an issue that could be addressed in future research, since Journal, vol. 42, no. 4, pp. 639–651, 2003.
it would further support the questionnaire’s reliability and
validity. In addition, the questionnaire should be expanded to [10] B. Adipat and D. Zhang, “Interface Design for Mobile
include the missing usability aspects, which were mentioned Applications,” AMCIS, p. 494, 2005.
during the evaluation, and other mobile platforms besides
[11] M. Satyanarayanan, “Fundamental challenges in mobile
Android and iOS should be integrated. However, the number computing,” Proceedings of the fifteenth annual ACM
of questions should be reduced. One possibility is to make symposium on Principles of distributed computing - PODC
recourse to statistical theories, like the classical test theory ’96, pp. 1–7, 1996.
or the item response theory, to identify correlating questions
or questions with a conspicuously low variance. The latter [12] H. Heitkötter, T. Majchrzak, U. Wolffgang, and H. Kuchen,
“Business Apps: Grundlagen und Status quo,” 2012.
aspect points to a useless question. Moreover, those methods
allow determining the weight of the different main and sub [13] P. Abrahamsson, A. Hanhineva, H. Hulkko, T. Ihme,
categories. One specific method, for instance, could be the J. Jäälinoja, M. Korkala, J. Koskela, P. Kyllönen, and O. Salo,
multivariate regression analysis. To apply such statistical “Mobile-D: an agile approach for mobile application develop-
methods it is necessary to collect numerous survey records. ment,” in Companion to the 19th annual ACM SIGPLAN con-
That is why mugram should be developed further. Offered ference on Object-oriented programming systems, languages,
and applications. ACM, 2004, pp. 174–175.
as software as a service solution, it facilitates the collection
of survey data which could provide helpful insights for the [14] T. Hofer, W. Schwinger, M. Pichler, G. Leonhartsberger,
development of the questionnaire. J. Altmann, and W. Retschitzegger, “Context-awareness on
mobile devices - the hydrogen approach,” in 36th Annual
Hawaii International Conference on System Sciences, 2003.
R EFERENCES Proceedings of the. IEEE, 2003, p. 10 pp.

[1] IDC. (2014) Global smartphone ship- [15] T. Goetz, N. Feldmann, and S. Schmidt, “Smarter apps
ments forecast 2010-2017. [Online] Available: motor fr geschftsmodellinnovationen,” in Smart Mobile Apps,
http://www.statista.com/statistics/263441/global-smartphone- ser. Xpert.press, S. Verclas and C. Linnhoff-Popien, Eds.
shipments-forecast/ [Last accessed on 2014-06-01]. Springer Berlin Heidelberg, 2012, pp. 507–518.

[2] IDC. (2013) Tablet Shipments Forecast to Top Total [16] J. Königstorfer, Akzeptanz von technologischen Innovationen.
PC Shipments in the Fourth Quarter of 2013 and Wiesbaden: Gabler, 2008.
Annually by 2015, According to IDC . [Online] Available:
http://www.idc.com/getdoc.jsp?containerId=prUS24314413 [17] P. Alahuhta, J. Ahola, and H. Hakala, “Mobilizing business
[Last accessed on 2014-06-01]. applications,” Technology Review, vol. 167, no. 2005, 2005.

480
[18] D. Xiaojun, I. Junichi, and H. O. Sho, “Unique Features [32] D. Norman, The design of everyday things. Basic Books,
of Mobile Commerce,” Journal of Electronic Science and 2002.
Technology of China, vol. 2, no. 3, 2004.
[33] J. Gong and P. Tarasewich, “Guidelines for handheld mobile
[19] F. F.-H. Nah, K. Siau, and H. Sheng, “The value of mobile device interface design,” in Proceedings of DSI 2004 Annual
applications,” Communications of the ACM, vol. 48, no. 2, Meeting, 2004, pp. 3751–3756.
pp. 85–90, Feb. 2005.
[34] J. Nielsen and R. Budiu, Mobile Usability. Pearson
[20] F. Aloul, S. Zahidi, and W. El-Hajj, “Two factor Education, 2012.
authentication using mobile phones,” in 2009 IEEE/ACS
International Conference on Computer Systems and [35] S. S. Chan, X. Fang, and J. Brzezinski, “Usability for
Applications. IEEE, 2009, pp. 641–644. Mobile Commerce Across Multiple Form Factors,” Journal
of Electronic Commerce Research, vol. 3, no. 3, p. 187,
[21] D. A. Norman and S. W. Draper, “User Centered System 2002.
Design; New Perspectives on Human-Computer Interaction,”
Jan. 1986. [36] B. Shneiderman and C. Plaisant, Designing the User
Interface: Strategies for Effective Human-Computer
[22] M. Magurie, “Methods to support human-centred design,” Interaction. Addison-Wesley/Pearson, 2010.
International Journal of Human-Computer Studies, vol. 55,
no. 4, pp. 587–634, Oct. 2001. [37] D. Saffer, Designing Gestural Interfaces. O’Reilly Media,
2008.
[23] ISO 9241-210, “Ergonomics of human-system interaction –
Part 210: Human-centred design for interactive systems,” [38] J. Häkkilä and J. Mäntyjärvi, “Developing design guidelines
2010. for context-aware mobile applications,” in Proceedings of
the 3rd international conference on Mobile technology,
[24] J. P. Chin, V. A. Diehl, and K. L. Norman, “Development applications & systems - Mobility ’06. New York, New
of an instrument measuring user satisfaction of the human- York, USA: ACM Press, Oct. 2006, p. 24.
computer interface,” in Proceedings of the SIGCHI conference
on Human factors in computing systems. ACM, 1988, pp. [39] N. Z. B. Ayob, A. R. C. Hussin, and H. M. Dahlan,
213–218. “Three Layers Design Guideline for Mobile Application,”
2009 International Conference on Information Management
[25] J. Kirakowski and M. Corbett, “SUMI: the Software Usability and Engineering, pp. 427–431, 2009.
Measurement Inventory,” British Journal of Educational
Technology, vol. 24, no. 3, pp. 210–212, Sep. 1993. [40] J. Benson, K. Olewiler, and N. Broden, “Typography for
Mobile Phone Devices: The Design of the QUALCOMM
[26] J. R. Lewis, “IBM computer usability satisfaction question- Sans Font Family,” in Proceedings of the 2005 Conference
naires: psychometric evaluation and instructions for use,” on Designing for User eXperience, ser. DUX ’05. New
International Journal of Human-Computer Interaction, vol. 7, York, NY, USA: AIGA: American Institute of Graphic Arts,
no. 1, pp. 57–78, 1995. 2005, pp. 2–18.

[27] Y. G. Ji, J. H. Park, C. Lee, and M. H. Yun, “A Usability [41] J. Tantillo, J. Di Lorenzo-Aiss, and R. E. Mathisen,
Checklist for the Usability Evaluation of Mobile Phone “Quantifying Perceived Differences in Type Styles: An
User Interface,” International Journal of Human-Computer Exploratory Study.” Psychology & Marketing, vol. 12, no. 5,
Interaction, vol. 20, no. 3, pp. 207–231, Jul. 2006. pp. 447–457, 1995.
[28] Y. S. Ryu and T. L. Smith-Jackson, “Reliability and Valid- [42] J. A. Krosnick and S. Presser, “Question and questionnaire
ity of the Mobile Phone Usability Questionnaire (MPUQ),” design,” Handbook of survey research, vol. 2, pp. 263–314,
Journal of Usability Studies, vol. 2, no. 1, pp. 39–53, 2006. 2010.
[29] M. Markova, A. Aula, T. Vainio, H. Wigelius, and M. Kulju, [43] A. Bangor, P. Kortum, and J. Miller, “Determining what
“MoBiS-Q: a tool for evaluating the success of mobile individual SUS scores mean: Adding an adjective rating
business services,” in Proceedings of the 9th international scale,” Journal of usability studies, vol. 4, no. 3, pp. 114–
conference on Human computer interaction with mobile de- 123, 2009.
vices and services. ACM, 2007, pp. 238–245.
[44] B. Martin, B. Hanington, and B. Hanington, Universal
[30] Android Developers. (2013) Design An- Methods of Design: 100 Ways to Research Complex
droid Developers. [Online] Available: Problems, Develop Innovative Ideas, and Design Effective
https://developer.android.com/design/index.html [Last Solutions. Rockport Publishers, 2012.
accessed on 2014-06-01].

[31] Apple Inc. (2014) iOS Human Interface Guide-

lines: Designing for iOS 7. [Online] Available:
https://developer.apple.com/library/ios/documentation/
userexperience/conceptual/MobileHIG/ [Last accessed on
2014-06-01].

481