*&&&"$.

*OUFSOBUJPOBM
$POGFSFODF
PO
4PGUXBSF
BOE
4ZTUFN
1SPDFTTFT
*$441


 
 

 



Padmalata Nistala Kesav Vithal Nori Raghu Reddy
TCS Research Software Engineering Research Centre Software Engineering Research Centre
Tata Consultancy Services Ltd. IIIT Hyderabad IIIT Hyderabad
Hyderabad, India Hyderabad, India Hyderabad, India
[email protected] [email protected] [email protected]

Abstract Quality Models play a critical role in assuring engineering problem[45]. 
  

 

quality and have evolved over 40+ years. They provide support  



 

discusses the support
for defining quality attributes, building and measuring the provided by standards for engineering quality. He points out
quality of the resulting product. Each quality model adopts a that requirement analysis phase, and verification and
critical view on quality in terms of a set of model elements and validation phases are well supported by ISO standards and
relationships between them. This study aims to provide an models, and architecture and design phase, and
overview of the state-of-the-art research on quality models with implementation phase have very limited support from the
a focus on encompassing model elements and their support to published standards and models [46]. Finne discusses the need
architecting quality. The study was conducted using systematic
for quality driven development and argues that quality design
mapping as the research methodology. A total of 238 primary
papers were classified based on the type of research, standards
and implementation must truly be an integral part of software
usage, and publication trends. We identified that 17% (40) of development [30].
papers belong to quality models. These 40 models were analyzed There are evolved quality models such as ISO 25000 [49],
for the underlying meta-model elements and their support for a Quamoco[37] that help in defining granular quality
quality architecture using 
rence architecture characteristics and measurements but it is not evident as to
framework. The architecture phase mapping analysis shows what extent do they support building software quality in a
that quality planning phase is 100% supported, quality systematic manner. There are multiple studies and
assessment is 75% supported, quality documentation is included
comparisons on quality models [52][53][54][55], but they
in 40% models and quality realization aspect is barely
considered in 13% models. Quality realization happens through
largely focus on comparison of individual quality attributes,
software processes and patterns, and it is necessary to evolve and do not discuss other model elements, nor provide a picture
quality models and software process architectures that correlate on the support provided for architecting quality end to end in
quality definitions and quality realization mechanisms. Future a systematic manner. This is the focus of our study.
research is expected in this direction. Our study aims to provide an overview of the state-of-the-
art research on quality models with a focus on encompassing
Keywords—Software Quality, Product Quality, Software
model elements and their support to architecting quality in a
Process, Quality Model, Quality Meta Model, Quality
Characteristic, Quality Pattern, Quality Architecture, Reference
systematic manner. We conducted systematic mapping study
architecture, Quality Realization following the guidelines provided by Kitchenham [43]. We
have mapped out the research on software quality models and
I. INTRODUCTION further analyzed the models through a meta-lens in terms of
encompassing meta-elements and the support offered towards
Software quality is the degree to which a software product
architecting quality 
 
reference architecture
satisfies stated and implied needs. Each quality model adopts
framework [41]. Bayer has proposed an architecture
a critical view on quality in terms of a set of model elements
paradigm that can be used for baselining architecture support.
and relationships between them. Quality models have
It comprises four phases: Planning, Realization,
evolved over a period of 40+ years from basic quality
Documentation, and Assessment. Planning phase defines the
attributes definitions as in Boehm[1], McCall[2] to
scope and scenarios to be used, during Realization phase
exhaustive quality characteristic definitions. Widely
solutions are selected and design decisions are taken to fulfil
accepted international standards such as ISO 9126 [48] and
the scenarios, Documentation is to document the results of the
ISO 25000 square series [49] have been published with
realization and Assessment phase is to analyze and evaluate
elaborate quality definitions and measurement criteria.
using defined assessment criteria.
Software process is the vehicle for building quality and it is
necessary to establish a correlation between software We analyzed 1255 research articles and selected 238
processes and quality definitions. relevant papers for the mapping study and classification.
Based on the research approach classification [42], we
Latest industry reports on world quality indicate the need
identified 40 key quality models which were further analyzed
for increased focus on overall product quality and adopting a
for architecture support. The study shows that quality
back to basics approach on achieving product quality [47]. In
characteristic and quality metric are the most common model
the emerging context of innovations, Breu et al. perceive an
elements, and software process and pattern/ practice elements
increasing role for quality and outline the new perspectives on
are barely addressed. Architecture phase mapping indicates a
software quality [44]. They stress on the end-to-end aspect of
significant gap in addressing the quality realization phase. The
software quality management and outline knowledge
results of this study will help researchers to understand the
management as a key aspect for fulfilling quality related tasks
software quality models area, the research trends, and types of
and decision. Tian argues that we need to move beyond just
research including standards related work, and the architecture
performing quality assurance activities towards quality
phases supported by quality models and the research gaps that
engineering by managing these quality expectations as an
could be addressed.


¥
*&&& 
%0*
*$441

Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 The rest of the paper is organized as follows: Section II requirements (ISO 25030) and Quality evaluation (ISO
analyzes related work, Section III describes the research 25040) divisions.
method used to conduct the mapping study, Section IV
presents results of the SMS, Section V discusses the findings, Dromey proposes a software product quality model to
Section VI discusses the study limitations, and Section VII support the building of quality into software [3]. He points out
states the conclusion and future work. that software does not directly manifest quality attributes, but
exhibits properties that contribute to quality attributes. He
II. RELATED WORK proposes a key link between quality attributes and product
 

 


  
 
! 

In this section, we briefly introduce some of the common carrying product property"#

The elements in Dromey's model

quality model terms and discuss few well-known quality are quality attribute, product component, and quality carrying
models to provide background on their encompassing meta- product property. This model considers quality definition and
model elements, along with literature reviews, systematic also focusses on the product properties to be built, but does
studies on quality models. not discuss how to realize those product properties.
‡ Software quality: Capability of a software product to Kitchenham et al. propose a Software Quality In
satisfy stated and implied needs when used under Development (SQUID) approach to define software quality
specified conditions. [49] requirements for a project [4]. They distinguish between the
‡ Quality model: Defined set of characteristics and of two components of a quality model: a structure model that
relationships between them, which provides a framework defines the model elements and their relationships, and a
for specifying quality requirements and evaluating content model that identifies a set of entities linked in
quality [49] accordance with that structure. The key elements in the
SQUID model are product behaviour, statement of quality
‡ Quality meta-model: A structure for organizing quality requirements, quality characteristic, quality sub characteristic,
related concepts and describing the complex structures of internal software property, and measure. The distinction of
quality models. It is a structure model that defines this model is in addition to the quality characteristics, it
quality model elements and their interactions. [4] [37] considers the operational behaviour of the product to derive
‡ Quality characteristic (QC): Category of software quality the quality requirements.
attributes that bears on software quality [49]. It is also Ortega adopts the view that qualities relate to each other
referred to as a quality attribute, quality factor and so on. and must be integrated for a systemic global quality design
A. Quality Models [11]. He argues that the following four dimensions are
required in order to make quality systemic: product efficiency,
One of the earliest quality models was McCall’s quality product effectiveness, process efficiency, and process
model [2]. He conceives product quality under three effectiveness. The elements in Ortega's systemic model are
perspectives: Product operation, Product revision, and quality characteristic, sub-quality characteristic, and quality
Product transition. Under each perspective, quality attributes metrics for product and process. This model looks at both
are defined as a hierarchy of quality factors, quality criteria, product and process dimensions from an evaluation
and quality metrics. Factors are user-oriented and Criteria are perspective but does not consider the aspect of building or
software oriented. McCall does not include the system or designing quality.
design elements into the model.
Marinescu and Ratiu highlight the limitations of well-
Boehm’s model proposes a framework for software known quality models based on Factor $ Criteria $ Metric
quality in terms of a well-defined hierarchy of quality (FCM) approach in addressing the quality design aspect and
characteristics [1]. The quality characteristics are based on propose a Factor-Strategy approach by introducing the
three primary dimensions for general utility of any software concept of detection strategies in the quality model to
package: As is utility, maintainability, and portability. These formulate good-design rules and heuristics in a quantifiable
quality characteristics are decomposed into multiple levels, up manner [12]. The elements of the model are quality
to a primitive characteristic. The elements in Boehm's model characteristic, quality sub characteristic, detection strategy,
are quality characteristic (QC), sub-quality characteristic (Sub and quality score. The uniqueness of the model is in the
QC), primitive quality characteristic, and quality metric. The association of quality factor to the strategy and quantification
definition of the QCs is granular in Boehm's model along with of this association using a computational approach.
measurement criteria, but the model does not state how the
defined quality characteristics have to be built. Khomh and Gueheneuc propose a model for Design
Enhanced QUALITy Evaluation (DEQUALITE) of object-
ISO 9126 standard is one of the early standards on oriented systems, taking into account the design aspects for
product quality and has been defined to standardize the evaluating quality in addition to quality attributes [18]. They
quality characteristics [6]. ISO 9126 view of quality is explore design patterns and leverage them in their method to
  


 

! 

 

build a quality model to measure the quality systems. The key
characteristics of an entity that bear on its ability to satisfy elements in DEQUALITE model are quality attribute, quality
 

  
 "#
It distinguishes three types of metric, design pattern/ design rule, and product or system.
quality: internal quality, external quality and quality in use. It The uniqueness of this model is the consideration of design
is a layered model and comprises quality characteristics, sub elements in evaluation and learning the design rules.
quality characteristics and metrics but does not discuss any Lifecycle process of the system is not considered here.
aspects related to the actual realization of QCs into software
systems. ISO 9126 is revised as ISO 25000 series of Finne looks at the abstraction of concepts in a quality
standards [49], that has four parts: Quality model (ISO model and proposes a quality meta-model that can be
25010), Quality measurement (ISO 25020), Quality instantiated and used for system development [30]. Key



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 elements of Finne’s model are quality attribute, use case, compare 40 quality models in terms of their encompassing
information system, contributor and metric. A three level model elements.
view of quality modelling is proposed $ comprising the meta
level, general and instance level that can be varied and III. RESEARCH METHOD
generalized. We carried out the systematic mapping study on software
Wagner et. al. propose Quamoco, a quality meta-model quality models following the guidelines recommended by
that is highly generic and attempt to bridge the gap between Kitchenham et al. [43]. Mapping studies are useful as a means
the abstract description of quality characteristics and concrete of analyzing how research in a given topic has evolved over a
quality measurement by operationalized quality models [37]. period of time. The research methodology is described here.
They introduce a key concept for operationalization called A. Research Questions
Product f%
 

& 
 

 #
The main research goal is to develop comprehensive
The key model elements of this meta-model are quality aspect, quality models and process architectures that can
product factor, entity, and evaluation (measure). Quamoco systematically build product quality at each process step.
helps operationalization for evaluation but still does not Towards that goal, we explored the space of quality models
address any aspects of quality design or process. and the support they provide for architecting quality. The aim
B. Previous Literature Reviews of this study is to provide an overview of the state-of-the-art
Multiple literature reviews were carried out on software research on quality models with focus on encompassing model
quality models. Cote et al. compare quality models to identify elements and their support to architecting quality.
a quality model that can be used as a foundation for software In order to achieve that, the selected papers were reviewed
quality engineering [51]. Four quality models (MacCall, in two phases. In the first phase, papers were classified by
Boehm, Dromey, ISO 9126) are compared on three criteria: type of research and publication trends. The type of research
five different perspectives of quality, usable from the top to classified on multiple parameters: standards research, models/
the bottom of the lifecycle, and from the bottom to the top of meta-models, research approach. * 
 
is
the lifecycle. They conclude that ISO 9126 seems well suited further modified to breakdown primary papers into model
for software quality engineering. In an analytical and papers and other research papers. The research questions are:
comparative study, Al-Qutaish compares five quality models
and the quality characteristics present in each model [52]. RQ1. What are the publication trends in software quality
They conclude that ISO 9126 model is the most useful quality models research area? What is the relevance of this
model. Tomar and Thakare in a systematic study on quality research area?
models, select 70 papers and analyze the published journals, RQ2. What types of research has been reported in software
investigated topics, and research approaches [53]. This paper quality models area? What are the main research
summarizes the findings but does not provide the trends and approaches reported?
frequency/ distribution of the data for the research questions.
Singh et al. discuss in their review on software quality models, In the second phase, the selected model papers from the
five popular quality models and compare the common first phase are further analyzed in terms of model elements and
attributes of various models and discuss the importance of their support to architecting quality. The meta-model
some key attributes such as reliability, usability, efficiency, elements were extracted from models and mapped to  

maintainability and portability [54]. reference architecture phases: planning, realization,

documentation and assessment. The support each quality
Miguel et al. in their review of quality models, discuss 14 model offers with respect to an architecture phase is
main models $ 6 basic, 4 tailored, and 4 open source [55]. determined and research gaps are identified.
They carry out a comparison of basic and tailored models
based on quality characteristics. They conclude that ISO 9126/ RQ3. What are the common meta-model elements considered
ISO 25010 will be the main reference model and need to in software quality models?
incorporate communications as a quality factor in the model. RQ4. What is the support quality models provide for
Davuluru et al. review five product quality models and two architecting quality from a reference architecture
process quality models in a study [56]. The conclude that ISO perspective?
9126 has many important characteristics and is a reliable
model. They recommend that an appropriate model has to be The search process has been developed to answer these
chosen according to the needs. Suman and Wadhwa review 17 questions. The overall selection process is depicted in Figure1.
quality models and conduct a comparative analysis based on B. Search Strategy
their quality characteristics [57].
Quality model papers have been searched in the following
We can observe from the existing literature reviews that four reference databases: IEEE explore, ACM digital library.
all the existing reviews and systematic studies limit to a Springer, and Elsevier. The main keywords used were
comparison of five/ six well-known models and the focus of "Quality model", "quality meta-model", "software quality",
their studies is on a comparison of a single element $ quality and "product quality". Search string has two groups of terms:
characteristic. As we can note from the previous discussion the first set of terms on quality models and meta-models, and
on quality models, the composition of model elements varies the second set on standards related models. As ISO standards
from quality characteristic, metric, goals to process/ activity are widely used for product quality models, we expanded the
and pattern/ practice and so on. None of the studies takes a search string for specific quality models of ISO standards as
comprehensive look in terms of their encompassing model the abstract and title of such papers sometimes do not contain
elements and the extent to which they support building quality the first set of terms. The search string is formulated and
in a systematic manner. This is the focus of our study and we refined based on retrieved results across all databases.
identified 238 papers for overall research area mapping and



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 Database fields title, abstract, and keywords were considered o Phase1: Eliminating duplicates, non-English and non-
in the string formulation. The search string formulated was: research papers. Applying these criteria, we selected 982
papers.
("software quality model") OR ("product quality model") OR
("ISO 9126 quality model") OR ("ISO 25010 quality model") o Phase2: Paper title and abstract were read and inclusion/
OR ("COTS quality model") OR ("quality meta model") exclusion criteria applied to filter topic and domain of the
This string has been coded as per the syntax format for each paper. Through these criteria, we selected 282 papers.
database. The cutoff date for the search was Jan 2019. o Phase 3: Full-text reading of the paper. This reading of
C. Selection Criteria papers resulted in 238 primary set of papers.
We used the following inclusion and exclusion criteria The overall selection procedure retrieved 238 primary
for reviewing and selecting the papers. papers for the systematic mapping study.
The inclusion criteria are: E. Data Extraction
- Articles written in English language
- Published in Journals, Conference or Book sections To answer the research questions, we extracted the
- Topic of paper is related to software quality or following information for each paper.
product quality 1. Title: Title of publication
- Content: the complete text must be available 2. Year: Year of publication
The exclusion criteria are: 3. Publication Channel: Channel in which the article was
- Duplicates published - Conference, Journal, or Book Section
- Articles written in non-English languages 4. Publication Venue: The venue in which the paper was
- Non-research papers such as sessions, tutorials published $ conference name or journal name.
workshop, panel, corrigendum, keynotes and so on 5. Research Approach: We adopted the classification of
- Model is for a specific quality characteristic or research types as outlined by Wieringa et al. [42] -
domain such as performance model, reliability evaluation, validation, solution proposal, opinion,
model, testing model, inspection model and so on. experience, and philosophical papers. Solution proposal
- Domain of the paper is non-software such as papers were further classified as Model papers, which
energy, manufacturing and so on. outline a new quality model, and Solution papers, which
describe an approach/ framework/ tool.
D. Selection procedure
6. ISO Standards Use: If the paper abstract or body contains
As depicted in Figure 1, the search on the reference reference or application of the quality standard ISO 9126
databases using the search string retrieved the following or ISO 25010, it is marked as an ISO standards use paper.
records: ACM-249, IEEE-549, Science direct-144 and
Springer-274. To identify additional relevant quality model The following additional information is extracted for papers
literature that is not discovered in the automatic database classified as Model papers based on the research approach.
search, we conducted backward snowball sampling on the 7. Model: Name of the quality model
model papers classified in the first phase analysis as per
guidelines of Kitchenham et al [43]. Snowballing involves 8. Model Type: whether the model is a Model/ Meta Model
gathering the references from the identified papers and 9. Meta-Model elements included: For meta-models, the
applying the listed inclusion and exclusion criteria. We meta-model elements were extracted. For models, the
identified 38 papers from this step, marked as other sources. abstract meta element was extracted.
The initial set of papers in total were 1255. This set of papers
were filtered in three phases applying the selection criteria. 10. Reference architecture phases supported by model: We

 
   
 

for
mapping the architecture phases supported by the model.
D /ŶŝƚŝĂů^Ğƚ ^ĞĐŽŶĚ^Ğƚ
<ĞLJǁŽƌĚƐ ;ϭϮϱϱͿ
;ϮϰϵͿ ;ϵϴϮͿ
ϯƌĚ WŚĂƐĞ
&ƵůůdĞdžƚ
/ ϮŶĚWŚĂƐĞ
^ĞĂƌĐŚ
;ϱϰϵͿ dŝƚůĞΘ
^ƚƌŝŶŐƐ ϭƐƚ WŚĂƐĞ
ďƐƚƌĂĐƚ
^ĐŝĞŶĐĞ WƌŝŵĂƌLJWĂƉĞƌƐ
ůŝŵŝŶĂƚĞ
ŝƌĞĐƚ ;ϮϯϴͿ
ƵƉůŝĐĂƚĞƐ dŚŝƌĚ^Ğƚ
;ϭϰϰͿ ;ϮϴϮͿ
EŽŶͲŶŐůŝƐŚ DŽĚĞůƉĂƉĞƌƐ
^ƉƌŝŶŐĞƌ EŽŶͲŵŽĚĞů
;ϰϬͿ WĂƉĞƌƐ;ϭϵϴͿ
;ϮϳϰͿ
EŽŶͲZĞƐĞĂƌĐŚ
/ŶͲĚĞƉƚŚĂŶĂůLJƐŝƐŽĨ
KƚŚĞƌ
ƋƵĂůŝƚLJŵŽĚĞůƐĨŽƌ
^ŽƵƌĐĞƐ
ĂƌĐŚŝƚĞĐƚƵƌĞƐƵƉƉŽƌƚ
;ϯϵͿ

Figure 1 Selection Process



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 IV. MAPPING RESULTS 20% journal papers. There is good publication% in top
Overall 238 papers are selected for carrying out systematic software engineering (SE) journals such as Information and
mapping. This high number of articles reflect the importance software technology (IST). IEEE transactions, IEE software,
ACM transactions and ACM SE journals. Top conference
of the area and the attention received from researchers.
venues are WoSQ- international conference on software
A. Research Area Relevance and Publication Trends (RQ1) quality, ICSE- internal conference on software engineering,
To understand the research area structure and publication and QSIC- international conference on quality software.
trends, we classified the 238 primary papers based on These publication trends indicate that quality models is a
publication year/ time period, type of publication channel, relevant and most researched area in SE.
venue of publication and standards usage. Figure 2 shows the
publication trend across the time period. dŽƉ:ŽƵƌŶĂů&ŽƌƵŵƐ
Ϯϱй

^ƚƵĚLJŝƐƚƌŝďƵƚŝŽŶWĞƌzĞĂƌ ϮϬй ^Y:

ϮϬй
ϴϬ
ϳϬ /ddžŶƐ͕^t͕^D
ϭϱй
ϭϯ ϭϰй
ϲϬ ϭϮ
/^d DddžŶƐ͕^
ϱϬ ϭϭ ϭϬй ϭϬй ϭϬй
ϰϬ
ϲй :^^͕W^
ϯϬ ϳ ϱй
ϳ ϱϳ ϱϯ
ϮϬ Ϯ ϰϯ
Ϯ Ϯϵ Ϭй
ϭϬ ϭϴ Ϯϯ
ϭϱ Ϭ ϭ Ϯ ϯ ϰ ϱ ϲ
Ϭ
ƵƉƚŽϮϬϬϬ ϮϬϬϭͲϬϯ ϮϬϬϰͲϬϲ ϮϬϬϳͲϬϵ ϮϬϭϬͲϭϮ ϮϬϭϯͲϭϱ хϮϬϭϲ
Figure 3 Top Journal Publication Forums
WƵďůŝĐĂƚŝŽŶƐ /^KƌĞůĂƚĞĚ

B. Types of Research (RQ2)

Figure 2 Paper Distribution per Year
To answer RQ2, we classified the primary papers on three
As the number of papers is high, paper counts are grouped dimensions: research approach, standards research and model
into 3-year periods. The first bucket is from 1976 to 2000, type. For research approach classification, we adopted the
during which period, the number of papers are low, which classification scheme by Wieringa et al [42] - evaluation,
increased from 2001, hitting the peak during 2010-2015. The validation, solution proposal, opinion, experience, and
increase in papers indicates a growing interest in the area, and philosophical papers. Solution proposal type is further
also coincides with the work on international standards ISO   

! "

! "# Table II and Figure 4
9126 and ISO 25000. There is a decrease in the number of depict the quality model study distribution as per types of
papers from 2016, which can possibly be due to the maturity
research.
of the current paradigm of quality models centered around
quality definition and measurement. TABLE II. TYPES OF RESEARCH IN QUALITY MODELS

TABLE I. PUBLICATION CHANNEL Research No of Standards

% Papers Standards%
Approach Papers Research
Publication Channel No of Papers Standards%
Model 40 17% 2 1%
Conference Papers 126 53%
Solution 103 43% 23 14%
Journal Papers 81 34%
Validation 23 10% 4 2%
Book Section 31 13%
Evaluation 30 13% 10 4%
There is a good spread of publications across various
publication channels as shown in Table 1: conferences, Experience 22 9% 4 2%
journals and book chapters. We further analyzed the top Opinion 20 8% 1 0%
journal forums and depicted the data in Figure 3. Topmost
journal publications are in Software quality journal (SQJ) with
'ĞŶĞƌĂůsƐ^ƚĂŶĚĂƌĚƐZĞƐĞĂƌĐŚ

KƉŝŶŝŽŶ

džƉĞƌŝĞŶĐĞ

ǀĂůƵĂƚŝŽŶ

sĂůŝĚĂƚŝŽŶ

^ŽůƵƚŝŽŶ

DŽĚĞů

Ϭй ϭϬй ϮϬй ϯϬй ϰϬй ϱϬй

/^K^ƚĂŶĚĂƌĚ 'ĞŶĞƌĂůZĞƐĞĂƌĐŚ

Figure 4 Research Classification by a) Research approach b) Standards Research c) Model Type



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 In Table 1 and Figure 4-a)%
! "
 

 

 
 %

! "
 

=
dLJƉĞƐŽĨDŽĚĞůůĞŵĞŶƚƐŝŶYƵĂůŝƚLJDŽĚĞůƐ
framework/ tool. We found that 17% of the papers, i.e., 40 ϰϬ
ϯϱ

DŽĚĞůŽǀĞƌĂŐĞ
papers propose new quality models. These 40 quality models ϯϬ
Ϯϱ
from 40 papers are taken up for further analysis in RQ3 and ϮϬ
RQ4. The biggest set of papers 43% belong to the solution ϭϱ ϯϯ ϯϬ
ϭϬ
category on various frameworks, tools, approaches using ϱ
ϲ ϳ ϱ ϴ ϰ ϱ
quality models. Empirical papers are on three categories: Ϭ ϯ ϯ

experience $ industry/ personal (13%), evaluation - case

studies, surveys, field studies (9%), validation $ laboratory,
simulations (10%) and personal opinion based (8%). There
фͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲDĞƚĂͲDŽĚĞů ůĞŵĞŶƚĂƚĞŐŽƌLJͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲх
were no philosophical papers.
ISO standards are an important contribution to quality Figure 5 Types of Meta-Model Elements
models and much work is done related to those standards. We
classified the papers further if the paper has referred/ As depicted in Figure 5, quality attribute and quality
customized ISO 9126 or ISO 25000 series of standards. We metric are the most common elements in quality models.
can see from Figure 4-b) that 33% solution papers and 33% Quality models support quality definition through attributes
evaluation papers use ISO standards. Overall 23% of papers and quality measurement through various metrics. However,
refer to ISO standards in their research, indicating the in order to build quality, these quality attributes need to be
prominence of standards research in this area. converted into software properties and software processes,
The 40 quality model papers were classified as shown in and associated with patterns and practices, which are barely
Figure 4-c). As discussed in section II, there are quality supported in models. Quality dimensions such as software
models that define individual quality characteristics and process, pattern knowledge, and information system are
measures, and also models that model the quality concepts barely addressed in quality models and quality artefacts such
from a meta-model perspective. A meta-model provides more as requirements, design need to embed the quality realization,
generality and applicability for the model as it can be which again is a gap in many models.
instantiated for the specific product. 40% of the models were
D. Model Support for Architecting Quality (RQ4)
meta-models and 60% were quality models.
For this research question on quality model support for
C. Types of Model Elements in Quality Models (RQ3) architecting quality, we have mapped the extracted meta-
To address this research question, we have adopted a model elements to  
>XZ\
reference architecture
meta-lens to analyze the 40 quality models identified in RQ2 paradigm as per the following guidelines.
analysis. Model elements were extracted for each quality
‡ Planning phase defines the scope of architecture and
model. If it is a meta-model, directly the element was
includes a selection of scenarios and a definition of views
extracted. For specific quality models, the underlying meta-
to be used for system architecture. Model elements
model element was identified and extracted. For example, for
related to the quality definition such as goal, attribute,
elements such as security, performance, usability and so on,
criteria, and so on are mapped to the planning phase.
meta-elements is quality characteristic. Model category
groups related model elements. The frequency of each model ‡ In the realization phase, solutions are selected and design
element was computed based on the number of models that decisions are taken in order to fulfil the requirements
include the specific model element. Table III lists the model given by the scenarios. Model elements related to design
category, model elements and their frequency. and implementation such as design pattern, practice and
so on are mapped to the realization phase.
‡ In the documentation phase, the results of the realization
TABLE III. TYPES OF MODEL ELEMENTS IN QUALITY MODELS
phase are documented. Model elements related to
Model Distinct Meta-Model product artefacts and deliverables such as requirements,
Category # Models use cases, system components, entities and so on are
Elements
Q. Goal Quality goal 3 mapped to the documentation phase.
Quality characteristic/ Sub ‡ The goal of the assessment phase is to analyze and
Q. Attr Quality characteristic/ Quality 33
attribute/ Quality factor
evaluate the resulting architecture using defined
Quality criteria/ Primitive assessment criteria. Model elements related to
Q.Criteria 6
characteristic measurement such as measure, metric, score and so on
product property/ Software are mapped to the assessment phase.
SW property 7
property/ design property
SW Process Process/ SDLC/ activity 3 Table IV maps the distinct model elements to each reference
architecture phase and also lists the contributing quality
Pattern Pattern/ practice 5
model name and reference. Based on Table IV data, count
Entity Entity/ component/ element 8 of meta-model elements for an architecture phase is mapped
Req. requirement / use case 4 in Figure 6. Figure 7 draws the ArchPhaseCoverage. In
Q. Artefact Artefact/ system 5 which if a model has elements corresponding to a phase,
ArchPhaseCoverage is taken as 1, else as 0.
Q. Metric Q metric/ Q measure 30



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 x Quality planning phase: We can see from Figure 6 and
Figure 7 that planning phase has been explored well in DŽĚĞůůĞŵĞŶƚƐŝƐƚƌŝďƵƚŝŽŶĂĐƌŽƐƐƌĐŚWŚĂƐĞƐ
quality models as various aspects related to planning are ϰϬ

modelled as distinct model elements, totaling 34. Some ϯϱ ϯϰ

of these elements are QC, Sub QC, criteria, quality goal, ϯϬ
dimension, product property and so on. All the 40 Ϯϱ
Ϯϯ
models (100%) address the planning phase through ϮϬ
including one or other planning related model elements. ϭϴ
ϭϱ

x Quality realization phase: This is a poorly addressed ϭϬ

area in quality models. From Figure 6, it is evident that ϱ ϰ

this area has the least number of model elements (4). Ϭ
Patterns and practices are some of the aspects related to Ϭ ϭ Ϯ ϯ
YƵĂůŝƚLJWůĂŶŶŝŶŐZĞĂůŝnjĂƚŝŽŶŽĐƵŵĞŶƚĂƚŝŽŶ ϰ
ƐƐĞƐƐŵĞŶƚ ϱ
Ͳϱ
this phase. As depicted in Figure 7, most models include
planning elements and assessment elements, and Figure 6 Meta-model elements across Arch Phases
realization aspects are barely considered. 5 models out
of 40 (13%) have some elements for realization.
YƵĂůŝƚLJDŽĚĞů^ƵƉƉŽƌƚĨŽƌƌĐŚ͘WŚĂƐĞƐ
However, even in these models, a holistic consideration
ϰ
of various aspects of quality realization is missing.

ƌĐŚŝ͘WŚĂƐĞŽǀĞƌĂŐĞ
ϯ
x Quality documentation phase: Documentation is a
partially addressed area in quality models. Product Ϯ
manifestation of quality realization is documented here.
Some models consider artefact manifestations such as ϭ

requirements, use cases, system components and so on.

Ϭ
As per Figure 7, 18 models (45%) consider some quality ϭ ϯ ϱ ϳ ϵ ϭϭ ϭϯ ϭϱ ϭϳ ϭϵ Ϯϭ Ϯϯ Ϯϱ Ϯϳ Ϯϵ ϯϭ ϯϯ ϯϱ ϯϳ ϯϵ ϰϭ
documentation elements but no model considers quality фͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲYƵĂůŝƚLJDŽĚĞůƐͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲͲх
attribute manifestation across all product artefacts YƵĂůŝƚLJWůĂŶŶŝŶŐ ZĞĂůŝnjĂƚŝŽŶ ŽĐƵŵĞŶƚĂƚŝŽŶ ƐƐĞƐƐŵĞŶƚ
resulting from software development.
Figure 7 Model Coverage across Arch Phases
x Quality Assessment phase: Assessment is a strongly
supported phase by quality models. As shown in Overall, we can infer from this analysis that quality planning
Figure 7, 30 models (75%) include assessment related and assessment phases are well supported by various quality
elements such as quality metric, measurement proeprty. models, quality documentation is moderately supported and
It can be seen from Table IV that the assessment quality realization is least supported by quality models.
dimension is exhaustively elaborated $ 23 distinct There are gaps and a need for research in quality models in
assessment elements could be identified. terms of quality realization and documentation activities.

TABLE IV. QUALITY MODEL COVERAGE FOR ARCHITECTURE PHASES

Reference
Meta-Model Elements Quality Model Coverage
Arch. Phase
Boehm [1], McCall [2], Dromey [3], SQUID[4],
QC, SubQC, primitive QC, quality criteria, Process, quality Datawarehouse [5], ISO9126 [6], ProcessQuality [7], QMOOD
attribute, quality carrying property,software property, quality [8], Bertoa [9], GEQUAMO [10], Systemic [11], Factor-
goal, quality dimension, process factor, sub process factor, Strategy [12], Alvaro [13], Lee_CBD [14], COTS [15], ABQM
design quality attribute, OO design property, SubQC runtime, [16], DevApproach [17], DEQUALITE [18], MDE_QM[19],
Quality
SubQC lifetime, userQC,devQC, sponsorQC, stakeholder, SQP [20], 2D [21], Squale [22], AOSQUAMO [23], SaaS
Planning
integratorQC, activity, development approach, purpose, target, [24],TechTopic [25], QuEF [26], QFacto [27],CMQuality [28],
viewpoint, SDLC, quality type, environment, quantitative SOA [29], Finne [30], QUIMERA[31], Unified [32],
property, view, phase, architecture QC objective, archtecture QUALGEN [33], Mobile [34], AOSQ [35], CBQM [36],
QC subjective Quamoco [37], ISO25000 [38], Khammal [39], SQuAP[40]

DEQUALITE [18], MDE_QM[19], CMQuality [28], Finne

Quality
design pattern, practice, quality pattern, contributor [30], QUIMERA[31]
Realization
Dromey [3], SQUID[4], Datawarehouse [5], QMOOD [8],
product behviour, component, query, OO component, entity,
ABQM [16], DEQUALITE [18], SaaS [24],TechTopic [25],
Quality system, SaasFeature, designTopic, code topic, run topic,
CMQuality [28], SOA [29], Finne [30], QUIMERA[31],
Documentation model elment, tranform rule, use case, information system,
Unified [32], QUALGEN [33], ISO25000 [38], Khammal [39]
artefact, function, method, requirement
quality metric, measurement property, quality measure, Boehm [1], McCall [2], Dromey [3], SQUID[4],
quality metric-internal, quality metric-external, quality metric- Datawarehouse [5], ISO9126 [6], QMOOD [8], Bertoa [9],
inuse, OO metric, measurement attribute, process quality Systemic [11], Factor-Strategy [12], Alvaro [13], Lee_CBD
Quality metric, detection strategy, quality score, fact, impact, [14], ABQM [16], DEQUALITE [18], MDE_QM[19], SQP
Assessment evaluation method, quality factor value, result, assessment [20], 2D [21], SaaS [24],TechTopic [25], QuEF [26], QFacto
model, quality indicator, quality measure, measurement [27],CMQuality [28], SOA [29], Finne[30], QUIMERA[31],
element, measurement function, measurement method, Unified [32] QUALGEN [33], Quamoco [37], ISO25000 [38],
measurement tool Khammal [39]



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 V. DISCUSSION quality models except in few models. Limitations of FCM
This section discusses the main findings of the study. We models in detecting the design flaws are discussed by
have identified 238 primary papers on software quality models Marensecu [12]. Pattern/ practice is modelled in 13% of the
for the mapping study. quality models. Context/ environment is another significant
aspect of quality and determines the quality scope and
With regard to the publication trend (RQ1), the high scenarios. This element is also barely addressed. There is a
number of primary papers indicate a good interest in the lack of harmonization across various quality dimensions in
quality models area. Conferences (53%) and journals (34%) quality models. Russo points out a clear gap in research in
both have a good number of publications. Software quality terms of a mutual relationship between the three dimensions:
journal (SQJ) and Information and software technology (IST) software quality, process, and architecture and states that no
are prominent journal venues. WoSQ, ICSE are top comprehensive model exists studying this triple interaction
conference venues. The publications trend increased from [40]. There is a need for research to formulate and harmonize
2000 onwards, reaching a peak during 2007- 2015. This rise the various dimensions of quality and their relationships in
in publications coincides with the rise in the number of ISO terms of a comprehensive model.
standards related research during this period. The maximum
number of model papers were also published in this time- Architecture support of quality models is the next
frame. There is some drop in recent years indicating a research question (RQ4) and the study deals with this question
maturity in the current paradigm of quality model definitions, through exhaustive model element analysis and mapping to
which are largely focused around quality and metrics architec
 #

 
   
 


definitions. Research needs to move beyond the current with its four phases is considered as a baseline framework for
quality definitions to the next level of quality paradigm analyzing architecture support of quality models. Architecture
towards quality design/ realization. support starts with a definition of the relevant quality views
that is addressed through the planning phase. Quality planning
For the types of research (RQ2), we classified on three is a well-addressed area from an overall quality architecture
dimensions: research approach, standards research, model/ perspective. All quality models have one or more planning
meta-model. We identified based on Wieringas classification related model elements. As listed in Table IV, 34 unique
scheme [42] that 17% of the papers, 40 were new quality model elements for planning can be identified across the 40
models, 43% were solutions, 31% were empirical papers quality models. This amply demonstrates the coverage of
(validation + evaluation + experience) and opinion papers planning phase support by quality models. The next phase is
were 8%. ISO 9126 [48] and ISO 25000 [49] are the most the quality realization which is a key aspect of any
referred quality model standards, 23% of papers are on architecture. The realization aspect is barely addressed in the
standards related work. 40% of the models are meta-models models. Only very few models look at this aspect.
and 60% are specific quality models. Some of the early models Realization related elements found in models are only four.
[3] [4] discuss meta-models and approaches, and there has Quality documentation is partially considered in models.
been an increasing trend of quality attribute specific models Components and requirements/ use cases are discussed in
during 2000-2010. In recent years post-ISO 25000 standards, some models. Beyond requirements, other design decisions
there are meta-models that explored the missing aspects of and documentation are not included in models. Quality
ISO quality standards in terms of quality operationalization, assessment is well addressed in models and 23 model
documentation and so on [37] [30] [39]. There are many more elements were identified. Overall the quality models support
desired aspects or dimensions to quality model and research is for architecting quality is 100% for quality planning phase,
to be carried out to investigate and address them. 13% for quality realization phase, 40% for quality
40 key quality model papers identified were taken up for documentation phase and 75% for quality assessment.
further analysis with respect to encompassed model elements Quality realization happens through processes and patterns,
(RQ3). The first set of model elements can be identified as and it is necessary to evolve software processes that establish
related to the quality definition: QC/ SubQC/ QA/ QF/ a correlation with the product quality definitions and quality
architecture QC, runtime QC, lifecycle QC and so on. realization mechanisms, i.e., patterns and practices. Nistala et
Similarly, there are model elements defined around quality al. outline process- product correlation and building quality
measurement: Quality metric $ internal, external, in use, OO, using pattern as key principles for product quality engineering
Quality measure, measurement attribute, element, evaluation [50]. There are research gaps and a strong need to develop
method, tool, quality score and so on. These two categories, comprehensive quality models and process architectures for
Quality attribute (83%) and Quality metric (75%) are the most addressing the important quality realization aspect.
common model elements defined across models. While the VI. STUDY LIMITATIONS
product quality dimension is endorsed in every model through
QCs, the process dimension is not correlated with the product We conducted the systematic mapping study following the
view. Diessenboeck discusses the problem of omitting guidelines of Kitchenham [43]. The main limitations of this
activities from quality model definitions as a serious flaw [16]. study are: bias in paper selection, data extraction, and
Some models consider the process dimension, but purely from classification schemes.
an evaluation perspective [11] [17]. Activity-based quality The search process was conducted by the first author and
models propose 2-dimensional views to integrate process and data selections were reviewed in detail by the second and third
product dimensions [16]. The software process is the vehicle authors. In order to mitigate the selection bias, the selection
for building quality and process is considered only in 8% of protocol $ search databases, inclusion, exclusion criterion
the models. It is necessary to reconcile the quality definition were elaborately discussed and refined. In case of any
with the software process. The knowledge related elements uncertainty in the selection, the article is reviewed by other
such as practice/ patterns are an essential connect for fulfilling authors. Search string was first piloted and has been refined in
the quality attributes. This dimension is not addressed in multiple iterations based on results and manual search



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 verification. Keywords in the search string have been updated For researchers and practitioners, this study provides an
after review and results verification. To control the publication understanding of the state of the art on quality models
bias, the SMS included the four scientific databases for research, as the first step for future studies. The set of 40
software engineering: IEEE, ACM, Science direct and model papers can serve as a reference catalogue of quality
Springer. All item types: conferences, journals and book model papers for researchers. This study is a first in carrying
sections were included. The authors have carefully studied the out a comprehensive meta-model analysis of quality models
title, keywords and abstract to take a decision on selecting the and mapping out their support for the reference architecture
paper. phases. This study identifies a need to move beyond the
current quality definitions to the next level of quality and
To ensure the accuracy of data extraction, a template was process paradigms towards quality design/ realization.
developed with required data elements and Zotero tool was
configured to extract and update the template data. The papers REFERENCES
downloaded from all databases were loaded into the tool for
[1] #
*#
 %
`#
{#
%

#
|%
!}~€€‚

consistency in data extraction. First four data elements were ‚|}€ƒ~

ƒ„
ƒ„*{‚
}|€…%"
#
ZX%
Z†‡ˆ#
automatically extracted and we configured the custom tool [2] J. A. McCall, P. ‰#
{%

Š#
„#
*  %
!„



fields for additional model related data fields. Verbatim data  #
 
€#
‹ 

& 


 %"

from the paper was extracted for the model information. This Fort Belvoir, VA: Defense Technical Information Center, 1977.
helped in extracting data in a consistent manner, which was [3] {#
Š#
& %
!




 %"
IEEE
used to carry out the mapping study. Transactions on Software Engineering, vol. 21, no. 2, pp. 146$162,
Feb. 1995.
To have consistency in classification schemes, we have [4] #
‰ 

#
|Œ%
!
}€&


 


adopted reference research frameworks. To classify the  

 %"
Software Quality Journal, p. 23, 1997.
  
 %


 
*  
[5] #
#
`  %
‹#
%

#
`Π%
!& 

 

 

classification scheme [42] and to classify architecture support, Inf


&
*  %"

Conceptual Modeling – ER ’98,


  
 
eference architecture framework, vol. 1507, T.-W. Ling, S. Ram, and M. Li Lee, Eds. Berlin, Heidelberg:
Springer Berlin Heidelberg, 1998, pp. 349$362.
Planning- Realization- Documentation- Assessment phases
[6] ~#
 %
! 

 
 
 
 

 %"
Journal of
[41]. Systems and Software, vol. 49, no. 1, pp. 89$96, Dec. 1999.
The search process, search strings, selection criteria, [7] #
%
{#
Ž%
‹#
Œ%

#
 %
!
Š  


database fields were first piloted and refined in multiple 

 
 
 %"

New Approaches in Software
Measurement, vol. 2006, R. Dumke and A. Abran, Eds. Berlin,
iterations. After each iteration, for any updates, the whole Heidelberg: Springer Berlin Heidelberg, 2001, pp. 94$110.
process was repeated again. [8] `#


‹#
Š#
&%
!
 


‘ -oriented
 
 
  %"
IEEE Transactions on Software
VII. CONCLUSION Engineering, vol. 28, no. 1, pp. 4$17, Jan. 2002.
This systematic mapping study focusses on software [9] #
„#
 %
! 
 

‹ƒ
‹ %"
’““’#
quality models and their encompassed model elements from a [10] ‚#
Š %
!Š‚}ƒ”A Generic, Multilayered,
meta perspective. It provides an overview of the quality ‹ %

 
 %"
Software Quality Journal, p.
models research area, maps the publication trend over 40+ 11, 2003.
years and determines the support offered by models for [11] #
ƒ %
!‹


 
 


‚ 



%"
Software Quality Journal, p. 24, 2003.
architecting quality. We included 238 primary papers in the
[12] {#
 

&#
{%
!

 

‘ -oriented
study, which were classified based on the type of research: design: the factor- 
 %"

11th Working Conference on
model (17%), solution (43%), empirical (31%) and opinion Reverse Engineering, Delft, Netherlands, 2004, pp. 192$201.
(8%) papers. Usage of popular ISO standards ISO 9126 and [13] A. Alvaro, E. S. de Almeida%

{#

|#
 %
!



ISO 25000 series was computed. 23% of overall papers ‹ 

 
 %"
#
ZZ%
’““•#
referred to ISO standards. [14] ‰ 
|


`
| %
!


 

Evaluation Model for the Artifacts of Component Based

From the primary set, 40 key quality model papers were &   %"

Sixth International Conference on Software
identified. These are further analyzed for the distribution of Engineering, Artificial Intelligence, Networking and
model elements and the support offered for architecture. Parallel/Distributed Computing and First ACIS International
Quality attribute (83%) and Quality metric (75%) are the most Workshop on Self-Assembling Wireless Networks (SNPD/SAWN’05),
Towson, MD, USA, 2005, pp. 20$25.
common model elements defined across models. Other
quality dimensions such as software process, knowledge, [15] #
{ 

#
 Œ%
!
~ 
 Quality Model for
‚ 
‹ƒ
‹ %"
Journal of Computer Science, vol. 2,
system, context are barely addressed in quality models: 8% of no. 4, pp. 373$381, Apr. 2006.
models consider process and 13% consider pattern/ practice. [16] F. Deissenboeck, S. Wagner, M. Pizka, S. Teuchert, and J.-F. Girard,
For mapping the model support for architecting quality, !
- 
 


 %"

2007 IEEE
International Conference on Software Maintenance, Paris, France,
model elements were mapped to the four reference 2007, pp. 184$193.show that
architecture phases: planning, realization, documentation, and [17] K. Domínguez, M. Pérez, A. C. Grimán, M. Ortega, and L. E.
assessment. It is identified that quality planning (100%) and  –%
!ƒ„*{‚
}|€…
ƒ&‚|
‚&
ƒ~

quality assessment (75%) are well supported by the quality ƒ„*{‚

&‚‚|ƒ‚~
{ƒ‹Ž‚%"
In Proceedings of
models, quality documentation (40%) is supported to some the 11th IASTED International Conference on Software Engineering y
ApplicationsACTA, p. 7, 2007.
extent and quality realization (13%) is barely supported. The
software process is the vehicle for building quality and it is [18] F. Khomh and Y.-Š#
Š—— %
!&‚}|€‚
 
 -
 

 
 %"

Proceedings of the 15th Conference
necessary to evolve quality models and processes that on Pattern Languages of Programs - PLoP ’08, Nashville, Tennessee,
establish a correlation of processes with the product quality 2008, p. 1.
definitions and quality realization mechanisms, patterns and
practices.



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.
 [19] #
 

#
&  %
!
 

 
 
and Future Directions), 2015 4th International Conference on (pp. 1-
Models in Model- & 
‚ %"
Proceedings of the Nordic 5). IEEE., p. 6, 2013.
Workshop on Model Driven Engineering, p. 15, 2008. [37] S. Wagner, A. Goeb, L. Heinemann, M. Kläs, C. Lampasona, K.
[20] Z. A. Rana, S. Shamail, and M. M. Awais, !

  

Lochmann, & A. Trendowicz%
!ƒ   

 
 



 
 %"

Proceedings of the 6th international 
  


%"
Information and Software
workshop on Systems development in SOA environments - WoSQ ’08, Technology, vol. 62, pp. 101$123, Jun. 2015.
Leipzig, Germany, 2008, p. 35. [38] `#
Š%
`#
|%

|#
‹%
!
€


&   
f
[21] |#
˜%
|#
|%

Ž#
Š%
!’-D Software Quality Model and Case 
 

%"

2016 Third International



„  
{  %"

2008 International Conference on Trustworthy Systems and their Applications (TSA),
Conference on Computational Intelligence for Modelling Control & Wuhan, China, 2016, pp. 117$122.
Automation, Vienna, Austria, 2008, pp. 1147$1152. [39] #
‰ %
…#
Œ%
#
#
Ž %

#
–Œ%
!Š  

[22] K. Mordal-Manet, F. Balmas, S. Denier, S. Ducasse, H. Wertz, J. Meta Model of Software Quality%"
 #
‡%
#
‡%
’“Zˆ#
Laval, F. Bellingard, and P. Vaillergues, !


- A [40] &#
{%
#
‹%
#
„ %

#
%
!
 -Model
practice-based industrial qualit
 %"

2009 IEEE International for Information Systems Quality: A Mixed Study of the Financial
Conference on Software Maintenance, Edmonton, AB, Canada, 2009,  %"
ACM Transactions on Management Information Systems, vol.
pp. 531$534. 9, no. 3, pp. 1$38, Sep. 2018.
[23] #
‰%
#
#
Š %

{#
‰%
!

 

[41] J. Bayer, T. Forster, D. Ganesan, J. F. Girard, I. John, J. Knodel, R.
aspect-  

 

™ƒ}ƒš%"
ACM Kolb, and D. Muthig, !& 

{   
  
 


SIGSOFT Software Engineering Notes, vol. 34, no. 5, p. 1, Oct. 2009. ‚
 %"
#
Z‡Ÿ%
’““X#
[24] `#
…#
| %
`#
*#
| %
&#
*#
‹ %

#
&#
‰%
!
 

[42] {#
* %
~#
 %
~#
 %

‹#
{ %
!{   

for Evaluating Software-as-a- 


‹ 
‹%"

2009 engineering paper classification and evaluation criteria: a proposal and
Seventh ACIS International Conference on Software Engineering 
%"
Requirements Engineering, vol. 11, no. 1, pp. 102$107,
Research, Management and Applications, Haikou, China, 2009, pp. Mar. 2006.
261$266.
[43] B. A. Kitchenham, D. Budgen, and #
  %
!Evidence-based
[25] {#
 ›%
Ž#
Š %
‹#
‰› %
Š#
  %

#
 %
!
software engineering and systematic reviews%" CRC Pres, vol. 4, 2015
Proposal for a Quality Model Based on a Technical Topic
[44] R. Breu, A. Kuntzmann-‹ %

#
„   %
!~ 

‹ %"
#
Z“%
’““†#
Perspectives 

 %"
IEEE Software, p. 7, 2014.
[26] F. J. Dominguez-Mayo, M. J. Escalona, M. Mejias, and A. H. Torres,
[45] J.Tian, !Software quality engineering: testing, quality assurance, and
!
 



 
‚ 
„ Œ

&*‚


  "% John Wiley & Sons, 2005.
   %"

2010 Fourth International Conference on Research
Challenges in Information Science (RCIS), Nice, France, 2010, pp. [46] W. Suryn, Software Quality Engineering. John Wiley & Sons, 2014.
495$506. [47] !* 
 
{ port 2017-Z %"
~
‚%
’“Z‡#
[27] S. Kalaimagal and R. Srini%
!„
Z’

 
 
[48] €ƒ
†Z’ˆ
’““Z%
!
 
$ 
 "%
Z




‹ƒ
 %"
ACM SIGSOFT Software Engineering Part 4
Notes, vol. 35, no. 2, p. 1, Mar. 2010. [49] ISO 25000
’“ZX%
! 


‚ ”Systems and
[28] K. Mehmood, S. S.-S. Cherfi, and I. Comyn-*%
!‹-Quality: A Software Quality Requirements and Evaluation (SQuaRE) series of
Pattern-Based Method and Tool for Conceptual Modeling Evaluation standards - Quality   
&"

€  %"

Advances in Databases and Information [50] P. V. Nistala, K. V. Nori, S. Natarajan, N. R. Zope, and A. Kumar,
Systems%
 #
ˆ’†•%
#
‹%
#
€œ%

#
  %
‚#
! 
  



 
‚ %"


Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 406$420. Software Quality Assurance, Elsevier, 2016, pp. 133$150.
[29] #
Š 

‰#
|%
!

 


ƒ%"

[51] M.-#
‹¡—%
*#
%

‚#
Š %
!€ search for a widely
WoSQ ’11 Proceedings of the 8th international workshop on Software applicable and accepted software quality model for software quality
quality - WoSQ ’11, Szeged, Hungary, 2011, p. 18.  %"
Software Quality Journal, vol. 15, no. 4, pp. 401$416,
[30] #
„ %
!

 
 -


 %"
Nov. 2007.
Software Quality Journal, vol. 19, no. 4, pp. 663$688, Dec. 2011. [52] R. E. Al-%
! 
 


‚ 
|  

[31] A. García Frey, E. Céret, S. Dupuy-Chessa, and G. Calvary, An Analytical and ‹
%"
Journal of American Science,
!}€‚{

 
 


 
 %"

p. 10, 2010.
Proceedings of the 3rd ACM SIGCHI symposium on Engineering [53] #
#


#
#
Π%
!
 

ƒ


interactive computing systems - EICS ’11, Pisa, Italy, 2011, p. 265.  
 %"
International Journal of Software Engineering &
[32] ‰#
|

#
Š %
!




 %"
Applications, vol. 2, no. 4, pp. 61$70, Oct. 2011.
in WoSQ ’11 Proceedings of the 8th international workshop on [54] B. Singh and S#
#
‰‘%
!
{  


 
 %"

Software quality - WoSQ ’11, Szeged, Hungary, 2011, p. 3. in 2013 International Conference on Communication Systems and
[33] T. Mens, L. Doctors, N. Habra, B. Vanderose, and F. Kamseu, Network Technologies, Gwalior, 2013, pp. 801$806.
!}|Š‚N: Modeling and Analysing the Quality of Evolving [55] `#
#
 %
&#
%

Š#
{¢ –%
!
{  




 %"

2011 15th European Conference on Software Quality Models for the ‚ 


%"
International
Maintenance and Reengineering, Oldenburg, Germany, 2011, pp. 351$ Journal of Software Engineering & Applications, vol. 5, no. 6, pp. 31$
354. 53, Nov. 2014.
[34] &#
„Œ %
#
‰ Œ%

‹#
*  %
!


 
[56] #
& %
`#
 %

#
#
#
{ %
!



 

Model%"

2012 12th International Conference on Quality Software,  %"

2014 International Conference on Advances in
%
%
‹%
’“Z’%
#
Z•X$157. Engineering & Technology Research (ICAETR - 2014), Unnao, India,
[35] #
‰%
! -Oriented Software Quality Model: The AOSQ 2014, pp. 1$8.
 %"
Advanced Computing: An International Journal, vol. 3, no. 2, [57] 

‘
*%
!
‹




pp. 105$118, Mar. 2012.  

 %"
 #
•%
#
•%
’“ZX#
[36] &#
#
&%
!‹
‹ 
 
 
 %"
€

Reliability, Infocom Technologies and Optimization (ICRITO)(Trends



Authorized licensed use limited to: Universidade Tecnologica Federal do Parana. Downloaded on February 21,2021 at 19:48:00 UTC from IEEE Xplore. Restrictions apply.