D e s i g n #or

TestabiMty
Object-Oriented Systems
.

2n

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
estability is the relative ease
and expense of revealing soft-
ware faults. This article maps
the testability terrain for
object-oriented development to assist
the reader in finding relatively shorter
and cheaper paths to high reliability.
Software testing adds value by reveal-
ing faults. It is fundamentally an eco-
nomic problem characterized by a
continuum between two goals. A reli-
ability-dtiven process uses testing to
produce evidence that a pre-release
reliability goal has been met. Time
and money are expended on testing
until the reliability goal is attained.
This view of testing is typically associ-
ated with stringent, quantifiable reli-
ability requirements. Other things
being equal, a more testable system
will reduce the time and cost needed
to meet reliability goals.
A resource-limited process views test-
ing as a way to remove as many rough
edges from a system as time or money
permits. Testing continues until avail-
able test resources have been expend-
ed. Measurement of test adequacy or
system reliability are incidental to the
decision to release the system. This is
the typical view of testing. Other
things being equal, a more testable
system provides increased reliability
for a fixed testing budget.
 Regardless ofwhich goal is empha- the scope of this article) in control sys- variables in sequential circuits. Test-
sized, testability is imporrant for both rem state-space models [20]. There ing a sequential circuit (a state ma-
ad hoc developers and organizations are many obstacles to controllability chine) is considerably more difficult
with a high level of process maturity. ancl observability. In general, they than testing a combinational circuit (a
Testability reduces cost in a reliabil- result from the fact that a component decision table). For sequential cir-
iry-driven process and incrcascs relia- under test must be embedded in an- cuits, these techniques aim to reduce
bility in a resoiirce-lilnil~~l process. other system. Removing obstacles to the test problem to one of generating
lIesign li,r testabilily (1)1’1‘) i s a controllable input and ohservable combinational inputs. 3) Bud/-in TP.TC
strategy t o a l i g n tli~ tlcvelopment output is the primary concern of de- (BIT) adds standard test circuits. The
pr~jcess so that Ic’st ing is maximally sign for testability. As a practical mat- density of logic gates on chips has
clIticrive i~ntlcr cillic,i- ;i reliability- ter, testability cannot be considered steadily increased (about an order of
tlri\,cn o r i.c.soilt.c,c,-lilnitrtl r e g i m e . apart from the software process. Pro- magnitude in the last 10 years), while
Ol3jecr-oric9iretl systems present cess cnp~tbilily is equally important. the feasible number of pins (external
SOIIIC’ (Inicllic ol~z~;~c.lcs to testability as Design for testability in VLSI. The interf&es) remained about the same.
well as sll;lrillg many with conven- concept of testability has played an With present-day VLSI, on-chip
tional illll)l~.l]l~tlr;llio]ls. The cost and increasingly important role in the testing capabilities are a necessity.
tlil’lictlll~~ 01. testing object-oriented design and manufacture of inte- Without these capabilities, it is impos-

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
S~SICIIIS C;III Ix- reduced by following grated circuits (chips or ICs). Chip sible to provide adequate controlla-
50111c l);isi< &sign p r i n c i p l e s a n d resting is performed to identify de- bility and observability. ANSI/IEEE
I)l;lllllitlg li)r Icsr. Broadly conceived, fects resulting from chip manufactur- Standard 1 149.1 defines a boundary-
soliwarc restability is a result of six ing or physical failures due to a wide scan architecture for BIT fLmctions
lilc~tol~s: range of causes (vibration, heat, cos- [ 141. Individual chips (embedded in a
mic rays). DLgn -cwifh/mn i s the single package) may be accessed via
l (:harac~eristics of the representa-
engineering development process fi)ur to six additional external test
lion
that assures ftlnctional correctness. lines and a few standardized test cir-
l (:haracteristics of the implementa-
Chip testing assumes complc‘tc hnc- cuits on every IC. This provides the
tion
tional correctness. In contrast, soli- ability to place an embedded chip 01
l Built-in test capabilities ware testing ;ISSII~C'S pci.iic1 replica- board into test mode, transmit (01
l The test suite (test cases and asso-
tion and tire presence 01’ liinctional generate) test bit strings to specific
ciated information)
13lrllls. embedded components, and capture
l The test support environment
(hips iI1.e tested by applying input output bit strings. The basic BIT ca-
l The software process in which
bit strings and observing output bit pabilities can be extended to provide
testing is conducted
strings. For example, with a simple complex, automatically initiated test
These six factors are tllc spine 01. the AN I) gate, we expect that a logical 1 is suites called Built-in,-Sdf’ Te.st (BIST).
testability fishbone (Figure I). We’ll produced only when both input lines Since approval in 1990, ANSI/IEEE
consider each in turn. Betijre exam- are also logical 1. There are three Standard I 149.1 has been rapidly
ining the major antI minor bones of main approaches to integrated circuit adopted by 1C manufacturers. Inter-
object-orienred tc’stability, we review design for test.’ 1) Ad hoc solutions operability and reliability of compo-
I’elated app~~~~dws. Although re- are component-specific. These in- nents and systems have improved.
search and pra(.lic~c~ in software test- clude the ability to disconnect parts, Software testability has received lit-
ing has Iiatl rcl;lli\~cly little to say adding more test points, and reduc- tle consideration relative to the large
about restabilir>, it 11~s been given ing a network to smaller, more tracta- number of books and articles pub-
considcr;~blc attcnlion i n the engi- ble components. 2) The sl~~c?zrt~rl lished on other aspects of software
neering antI inaiiuf,lctlii-ing of VLSI appronc/~ refers to design rules that testing. Definitions of testability from
(very large-scale integration) devices. reduce or make controllable state two U.S. software engineering stan-
dards appear in Figure 2.
The Concept of Testability Some notion of testability is im-
Testability has two key facets: control- plicit in most testing strategies. Struc-
Inbilit~ ancl o/w~7d~i/ily. To test a com- tural and functional test case design
ponent, you must be able to control methods are typically based on a pro-
its input (and internal state) and ob- gram or system model and a related
serve its output. If you cannot control fault hypothesis. For example, data
the input, you cannot be sure what flow testing assumes the paths
has caused a given output. Ifyou can- formed by definitions and uses of
not observe the output of a compo- variables are a good place to look fol
nent under test, you cannot be sure errors. These models imply a kind of
excellent summary of DC-r appI-oaches and an
how a given input has been pro- integrated process model 1,~ clevelr)pment and
general testability criteria, since cer-
cessed. With controllable input and manutacturiw of testable multichip modules in tain programs may be easier or
observable output, compliance can be harder to test according to these
through known-good’ICs and elfective test strat-
decided. These concepts also have an egies.” Pmwh,q\ 01. I/W IEEE. X0, I 2 ( D e c . models. For example, it may be pro-
analogous, formal meaning (beyond 1992). 1965-1994. hibitively difficult and expensive to
 devise patl! test cases for a program the domain of the output variables. A less sensitive statement is less like11
with spaghetti code or self-modifying The number of changes that must be to produce different output. Lower
control. Qualitative discussions of made to obtain perfect observability sensitivity implies that faults are more
testability appear in [ 151 and [22]. In and controllability is an index of test- likely to be hidden, and therefore rel-
general, testable software is small, ability. atively more testing will be required
simple, and explicit. An observability metric fi)r every to reveal them. Sensitivity is com-
A pragmatic analysis of unit-level statement in a program is presentctl puted for each statement. Program
testability is presented in [15]. When i n [24]. The merric i s lx~srtl OII Ills. sensitivity (testability) is the minimum
a module is embedded in an applica-
tion system, it may be difficult to fully
exercise it (lack of controllability) and
the immediate outputs may be ob- Representation Implementation Built-in Test
scured (lack of observability). User
interface bindings may reduce test-
ability. Accessing and tracing motlulc Testability
input and output may require slxxial
tools. Several strategies are suggested
to increase controllability ancl obs~rv-
ability. Information hidilig ;IIICI SC’IW Test Suite Test Tools Test Process

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
ration of concerns improves tcst;tl)il- L

ity. Test scaIl6lding (test tllivcl.5 atic

stubs) improves contl~oll;~l~ili~~ ;IIICI
4.3.4 Traceability of requirements to test cases. The contractor shall document
observability. Even wit11 sc;ll’li)ltlillg. iI
the traceability of the requirements in the Software Requirements Specifications
is often us&i1 I0 xltl c.sl)lkil l)IIiII-ii1 (SRS) and Interface Requirements Specifications (IRS) that are satisfied or partially
test liinctions 10 1~c’l~01~l iillc.t.il;tl 21;11~’ satisfied by each test case identified in the Software Test Description (STD). The
0 1 1 tlemantl. “\Yitli siliil)lc. III~~~II~~~~. contractor shall document this traceabiltty in the STD for each Computer Software
this separation ol‘~~o~~~~c~~~t~~ i\ IIW.I’III- Configuration Item (CSCI).
with complcs 0~1~~5. il is iii\.‘ilil;il)l~..”
10.3.2 Testability of requirements. A requirement is considered to be testable
Asscl-riot15 illc,rc;tsc, ;IIICI rc*l;tilt OIIM.I.\ -
if an objective and feasible test can be designed to determine whether the
ability wlicn 1lic i110~11i1~~ IIII~I~I~ I(‘51 i5
requirement is met by the software.
subsccluct1ll\ iiitcgr;~~c(l \+illl 111~. rc’al DOD 2167A
I>
of an ;il~l)li(.;ilioii s~sl~~iii.-
A fi)rnial iiivcslii:.;llioli 01’1~~s~;tl~ility Testability. (1) The degree to which a system or component facilitates the
based 011 ;tl)slI.;I(.I tl;il;t 01x3 ;iritl Ilivii. establishment of test criteria and the performance of tests to determine whether
intcrliiccs is givcli iii ( l,I]. ‘l’wo pix- those criteria have been met. (2) The degree to which a requirement is stated in
terms that permit establishment of test criteria and performance of tests to
tices arc’ itlcnlilietl tlial reduce
determine whether those criteria have been met.
testability. Global or local state vari-
IEEE 610.12
ables reduce observability of proce-
durc or function output, since re-
sponse is not determined by input
explicitly passed to the unit under observable effect of program muta- Figure 1. The testability
test. A fimction or procedure that tions. “A program’s testability is ;I fishbone: Main facets
camlot produce every possible value prediction of its ability to hide faults
in the domain of its associated type when the program is black-box tested Figure 2. Testability definitions
lacks controllability; part of the out- with inputs selected randomly from a
p u t d o m a i n cannot be produced particular input distribution.” Test-
under any circumstance. Modules ability is defined as sensitivity to faults.
can be made perfectly observable by Sensitivity is determined by systemat- of all statements sensitivities. Hence,
adding explicit arguments to replace ically mutating a program and its highly sensitive programs are highly
access to global or local state vari- data, running the mutated program, testable.
ables. Similarly, they can be made and comparing the output of the
perfectly controllable by defining a mutant to the output of the original Testability Factors
data type that exactly corresponds to program. Given some input set, state- The preceding definitions of software
1nent sensitivity is a product of three testability focus on technical aspects.
‘An asset-tion consists of’s condition and an asso- probabilities: frequency of statement As a practical matter, testability is as
ciated exception routine. This may be imple- execution, probability that a fault will much a process issue as it is a techni-
mented by the assert () macro in C and C++.
The assertion statement is simply a Boolean ex- result in an incorrect data state, and cal problem. There are six primary
pl-ession that is evaluated when the statement is the probability that a fault will result testability factors (see Figure 1). Each
reached during program execumn. If the con- in an incorrect output. A highly sensi- may facilitate or hinder testing in
dition evaluates to true, nothing hilppens. If the
condition is lake, the proqam is terminated 01 tive statement is very likely to pro- many ways. A usable representation is
an exception is activated. duce different output when mutated. necessary to develop test cases. Im-
 oriented repi.escilt;itions, generically ANSI standard 1016 provides a defi-
linoI4 I1 ;ib ol?jcc-I-oi.irnretl analysis nition of a complete and usable soft-
(()()I\) illltl ol?j~c.i-oi.ic.il~e~~ d e s i g n ware design description (SDD) [13].
(001)). )cor ~~~;iiii~h~. II 51 outlines a An SDD should define the organiza-
s\stc.~tl;llic ;I~~IIO~IC~~I IO OOA/D which tion of software components, depen-
(~1~11ic ill\ citltl~c~s~s development of dencies, interfaces, and detail of algo-
I(‘41 ( 1~5~‘5 I I OIII Ihc representation. rithms and data structures.
I,igtlIc. :i \IIo\vlr Ic5lability facets of Completeness means the specification
,‘c~j,rc’~c’rll.lllC,II\. covers all aspects of the system and
l(c.cltiil (‘IIIC’III\, itlclude both narra- that there are no “to be determined”
li\t. ( .II).IJ)llil\ statements and abstract items.
,I SI~‘III Itlo(l<4b. ‘I Ire standards shown Truceability is a simple, common
III IRIS”“’ L’ itlclllili essential charac- sense notion that can prevent a wide
1c.1 ihlic \ foi lc~hlahility. Objectlur re- range of problems. It is simply a
Representation. ‘I )I(. ~II’~W’II( (’ 01 .I (III~IC.IIK.III~ AI<’ phi-ased or modeled bookkeeping operation that records
~-eprchct~l;~l~c)r~
.rrltl iI5 II~(.I~IIII(.\\ ii1 ~II \II~ II .I \\.I\ 11m compliance in the the antecedent of any work product.
test developnlcllt is ‘I c rili( ;I) t(.\t.1l)il- L~\\IC’III IIIIO(.I ICYI (SUI‘) is not subject In testing, we are interested in being
i t y taclor. S~slctii I~~~~I~~~~~II~~I~I~II~ IO 1tl10\\ I C r;blic iiiterpretation. Frtlsi- able to find which software compo-

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
lmgr fl~om ll;lllll~;ll l;lllgll;l~:(~ \1.llC’- /j/t, c .I~)llOillll~. III;I~ be developed and nent implements a given specifica-
merits about desired c.lpal)ililic,h I O IIIII)I(.III(.III(.(I \\.itll existing technol- tion, and which specification imple-
detailed finmal qwcificat~o~~s. 1 CMIII~ o,q\. \\‘111lc, 1101 a11 capabilities benefit ments a given requirement. Without
without a represrirtatioii i5 siiili)l\ IIOIII ,I (lll.ltllil;llivc definition, it ce~-- this information, it is practically im-
experimental pi-otolyping-it ~~~ltli101 t;tirll) C’.IZC’~ lc.\tillg to quantify those possible to develop complete and ac-
be decided that a test has pamu OI Illal C;~II. I:IJI c3;imple, “average re- curate test plans for even a small sys-
fbikd without an explicit description ~XIIISC’ IIIIIC n II) Ix I second or less,” t e m . Currency m e a n s t h a t t h e
of the expected result. The best that inslcad 01 “ I ~~l)on.se time will be specification mirrors the system as
can be said of “testing” without a rep- quit-h.” I I:I:I.,‘.\NSI standard 830 de- programmed. If specifications are not
resentation is that it may force pro- fines 111~ (lc3il;1111c aspects of software current, they result in incorrect test
duction of a partial representation as rc’citlil C’IIIC’III, \~KX ilication: unambig- plans.
part of the test plan. There are many IlOlllr. cc~lllJlI~~I~~. verifiable, consis- Sepwution of’concem.5 is a first prin-
approaches to developing ot+c I - 1~‘1ll, r~lotlili.tl~l~~. traceable, feasible, ciple of software engineering that is
;ititl 11~~~11il 101 m a i n t e n a n c e [l 11. directly related to controllability and
l~k11 01 I IIc.\c. alcributes contributes to observability. A component that can
Ic.\l.ll)ilil\. act independently of others is more
Figure 3. Testability: St)c.( il;, .Iliollh describe the detail readily controllable. Separation of
Representation tlcGgu1 Ior AII illl~)lelnentation. IEEE/ concerns is application- and environ-
ment-specific, but we suggest a few
key issues here. If user inte$ce capa-
Requirements Specification bilities are entwined with basic func-
tions it will be more difficult to test
each function. The control strateg
Completeness
(mechanism for allowable sequence of
4Feasible component activation) of object-
oriented programming tends to be
Quantifiable IEEE 1016 less explicit (but no less important)
IEEE 830 than conventional systems. If control
is embedded in the structure of the
Representation interfaces, it is more difficult to devise
t explicit tests than if control is an ex-
plicit function of a control object. Col-
User interface
luboration Puckaging refers to the
structure of classes participating in a
responsibility. Architectural Packaging
refers to the way classes are allocated
Collaboration Packaging
to tasks in the target environment
SUT/Test Suite and how their runtime interfaces are
4 composed. To the extent that packag-
Currency
ing and capabilities are orthogonal,
-/ they will be more testable. In a dis-
Traceability Separation of Concerns tributed target environment, for ex-
ample, a single responsibility imple-
 Structure Fault Sensitivity

Input Distribution

Demeter Compliance

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
Multiprogramming

Control Flow Non-deterministic Control

Time Sensitivity Shared Resources

External Interface Determinism Exception Handling

mented by objects rcsitlillg OII M’I.C’I ;II IIII(~(.I l(~l ((11 I ). IIIII\ r(xlIl(.itlg lest- Figure4. Testability:
hosts will probably, 1)~ IIIOIX’ (lillic 1111 ;Il)ilit\. (~oo~l tlc.sigll 11511;1ll! illrproves Implementation
to test than the S;IIIIC I~C~~~~OII~IIIIIII\ OII Ic.sl;il;ilil\ I)111 1’1 (‘II wc.ll-tlcsig1icd
a single host. ;lpl)lic-alioits III;~\ lx. illllci-c71tly difli-
Implementation. I;iglll-c .I SIIOWS c11lt IO 1~51. ‘l‘llc~~c 111ct 1.ic.s 1)rovide in-
.
the Implementation tcst;tl)ilit\ lisll- li)ivliatioil tiselill 101. rc*sol\~iirg tlesign to test application server classes with-
b o n e . A n object-orielltc.tl l>~‘~~granl ~r;itlf3)l’l~ and aswssiil~ ixhtive cliffi- out using the GUI. However, the
that complies with genc~Al~~ acwl~ted cdty mid cost ol’testiiip,; tlq, are not GUI may decrease both controllabil-
principles of objecr-oricnretl p r o - sonic kind of‘ score lo 1)~ tllasiniizecl ity and observability of a CUT. Deter-
gramming poses the fewest obstacles or minimizecl. minism is the extent to which the CUT
to testing. Structud testability can IX ‘l‘hc /l/ill 0/ L)P)Wlf~) constrailis uses does not require asynchronous coop-
assessed by a few simple metrics. (message se;ids) to “preltrred sup- eration with other tasks. A high de-
These are summarized in Tables I plier classes,” in effect superclasses, gree of concurrency poses obstacles
through 4. A metric may indicate classes of‘ instance variables, and to repeatable tests. It may be difficult
complexity, scope of testing, or both. classes of arguments passed to the to reproduce and isolate the cause of
The effect of all complexity metrics is class uncler test [18]. Compliance re- a failure: the SUT, the environment,
the same: a relatively high value of cluces the number of interfaces, the or a particular input pattern. Testable
the metric indicates decreased test- number of‘ stubs and drivers that may exception handling requires consistent
ability; a relatively low value indicates be needed, and the number of inte- usage of language-supported features
increased testability. Scope metrics gration test interfaces. Fault sensitivity and a related design strategy [20].
indicate the quantity of tests; the [24] was discussed previously. Low Exceptions are typically less control-
number of tests is proportional to the sensitivity corresponds with low test- lable than other application functions
value of the metric. High scope and ability. Performance tweaks tend to re- and may require simulation of failure
low complexity suggests many simple duce the correspondence of the im- modes. This may be difficult and time
tests are indicated; low scope and plementation and a specification and consuming.
high complexity suggests a few diffi- can interfere with both controllability Built-in test. Built-in test (BIT)
cult tests. and observability. External Interfaces capability provides explicit separation
For example, with high coupling often create difficult testability prob- of test and application functionality.
among classes (CBO) it is typically lems. For example, with an “event- The systematic addition of set/reset,
more difficult to control the class driven” GUI, it may be very difficult reporters, and assertions to a class is a
Table 1. Testability and enCaPSUlatiOn

LCOM (Lack of Cohesion in Methods). Number of groups of instance J

variables used by one method only [lo]. High LCOM means more states must
be tested to prove absence of side effects among methods.

PAP (Percent Public And Proiet led). I’rrcen~age of data members in a (:+ + J
class thal are public 0~. pro~c~-Id. Intlicatcs ploporlion of class data visible to
other objects [2 11. lligh PAP means more’ opporf\tnities fin. side efiects among
classes

PAD (Public Access to Data members). The number of external accesses to a J

class’s public or protected data members. Measures violations of
encapsulation [21]. High PAD means more states to test to prove absence of
side effects among classes.

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
Table2. Testability and inheritance

NOR (Number of Root Classes, RootCtn.) The number of distinct class

hierarchies employed by a program [21]. NOR is one dimension of number of
components to test.

FIN (Fan I II). ‘l‘he nun~be~ o1 classes I’IXWI which a class is derived. FIN > 1
is only possible with multiple inhrritancc [?I]. High FIN increases rhe
possibility of’ incorrect bindings. III most case>, all of the inherited methods
will need to be retested in the Cli’l‘.

NOC (Number of Children). The number of classes derived from a specific J

parent class. It indicates the number of classes which will be directly
impacted by a change to the parent (class fan-out) [lo]. In most cases, all of
the parent methods will need to be retested in the child CUT.

DIT (Depth of Inheritanc-e ‘1.r~). Indic~atrs the level oi a class within its
hierarchy [IO]. In IIIOS~ cases all inherited mrlhods will need 10 be retested in
the CUT.

Table 3. Testability and polymorphism

OVR (Percent of non-overloaded calls, Pctcall.) Percent of calls throughout

the SUT that are not made to overload modules [21]. Overloading may
result in unanticipated binding. High OVR indicates more opportunities for
faults.

DYN (Pel-cent of dynamic calls.) Percent of messages throughout the SUT

whose target is deter-mined at WI) time. Dynamic binding may result in
unanticipated binding. High DYN means that many lest cases will be
needed to exercise all the bindings of a method.*

Bounce-C. Count of the number of yo-yo paths visible to a CUT. A yo-yo J

path is a path that traverses several supplier class hierarchies due to
dynamic binding. A bounce may result in unanticipated binding. High
bounce indicates more opportunities for fau1ts.S

Bounce-S. (Iount of the number- of yo-yo paths in SUT. A yo-yo path is a

path that travel-ses several supplier class hierarchies due IO dynamic
binding. A bounce may result in unanticipated binding. High bounce
indicates more opportunities f& l>ults.$

$Under investigation by the author.

simple way to provide effective con- cation is treated as a no-op or an ex- ment of test cases [8]. A comprehen-
trol and observation. Attempts to ception. In test mode, BIT services sive list of test suite components is of-
approximate BIT with application are enabled. Password control requires fered in IEEE/ANSI standard 829 [2].
methods is a partial solution at best. If a password of some kind to activate It defines the contents of a test plan,
a standard test interface is included in BIT functions. The driver must sup- test design specification, test case
all classes, additional development ply a password value in a parameter specification, test procedures, test
overhead is minimal and the poten- defined as part of the BIT interface. item transmittal report, test log, test
tial payback is great. The BIT fish- A BIT request without the password incident report, and test summary. A
bone is shown in Figure 5. would be treated as a no-op or an test suite lacking in these features
Without set/reset, effective state- exception. There are many possible may pose practical obstacles to test-
based testing is a practical impossibil- saftity schemes. Regardless of the BIT ability. A test case schema for object-
ity. A set/reset u~cthotl allows an ob- ~I~~~~l~OilCll, xome lorm o f encapsula- oriented programs is presented in
ject to be set to a prcclcfilletl iutertr;ll tiorl salCty should be provided. [71.
state, regardless of its ~itrrc’t~l sl;itc. .+\.~~iorr.\ can be used for many Expected results are necessary for
Several approaches ;LIC’ possible. ‘1‘1~ p~~q~osc’s: I)l.“-(.otl’litiolls, post-condi- test cases. An oracle is a mechanism
BIT can ofltir ;I IIIC’III~ 01. SI;II(Y \\.illl liorls, (.I;155 iriv;ii.i;ints, and sampling for producing expected results. Typi-
noncontroll;~l~lc itis1;tiit.c. v;iri;il)lc KII-

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
ues, a menu ol~sl;~tcx witli ~~01111~011;11~1~
values, 01‘ dirccl iil;lilil)ill;llioi1 01 iit-
stance variables. Driver Set/Reset Safety
A reporter I.C’IIII.IIS t11c. c OIN I (‘I<’
(internal, p&ilc) 5l;itc 01 ;III OI+Y I. I I
the CUl‘ does IIOI ollc~- cxm~plc~c~ I’(‘-
porting of’ abstr;lc.r (c~tc~r11;11, I)]ll)lic )
state, the reporter IIIII~I pro\~i(lc. il. ’ .\
r e p o r t e r must Ix lriist ~o~~ll~~~--\~~~~
need to be sure its relxms arc ;wI~-
rate. Reporter code should 1)~ SIIII-
jetted t o carel’ul Acrlltin). I;oI- C’S;IIII-
ple, the code could be proved ~OIXXYI . .
and then testecl with low-level tlcl~~g
probes.
Concrete State‘J ln,.‘,,,,e .‘.sult s 7 ‘” ‘St
Useful set/reset and reporter metll-
ods necessarily violate encapsulation. PAbstract
r e State
- C,/o n d i t i o n s 4
BIT services should not be used to
accomplish application requirements. Trustworthy Post-Conditions
A safety provision is advisable to pre- 4
vent inadvertent or willful misuse of / Class Invariants -/
BIT services. Source code control uses
compiler-directives to include or ex-
clude BIT features. Either a BIT or a Reporter Assertions
non-BIT version of the SUT can be
built. This is problematic because the
tested implementation is not neces-
sarily the released implementation. of intermediate results. For example, Figure5. Testability: Built-in test
There are many horror stories about they can monitor long-running algo-
systems that inexplicably fail when an rithms to detect failure to converge.
apparently innocuous code segment After class testing, a class must be
is added or removed. Source code embedded in an application system. tally, expected test results are pre-
control should be used with due dili- Once embedded, assertions continue pared by manual calculation or sim-
gence. Mode control provides a global to provide built-in testing and report- ple simulation. Oracle automation
parameter to toggle test mode and ing. They contribute to testability by and class testing are discussed in [ 161.
normal mode. In normal mode, BIT proving built-in observability. A driver A system for which an oracle is infea-
services are disabled. Any BIT invo- is a special-purpose class that acti- sible is fundamentally untestable. For
‘Abstract state is implementation-independent vates the CUT-drivers are discussed example, the quantity of input or out-
and may be reported via an object’s interface. later in this article. put to consider may be intractably
Concrete state is implementation-dependent
Test suite. A test suite is a collec- large, or there may be no way to ob-
and is typically not reported. For example, sup-
pose a collection is implemented by a linked list. tion of test cases and plans to use tain expected output, n priori. Such
The abstract state of the collection includes the them. The Test Suite fishbone is systems require special verification
contents and number of items in the collection;
concrete state includes pointers to the first, last, shown in Figure 6. There are many strategies [25].
and most recently used nodes [19]. possible approaches to the develop- A test suite represents an asset ac-
 WMC (Weighted Methods per Class) is the turn of the cyclomatic complexit)
of methods implemented within a class [IO]. Higher complexity is often
associated with a greater number of faults; a proportionally greater number
of test cases will be required for decision coverage.

RFC (Kesponse for a (3ass) is a count of methods implrmented within a

class. plus the number of methods used 1,) an object not dw II)
inheriranc-e 1 IO]. High KFC indicatrs a c,lass will either require high numbr~. of
stubs or must he embedded with many other c I;~sses brlor~ it c all bc tested.
Each such interface will need to be tested.

CBO (Coupling Between Objects.) The nmnber of non-inheritance couples to

classes outside the CUT [lo]. High Cl30 indicates a class will either require
high number of stubs or must be embedded with many other classes before
it can be tested. Each such interface will need to he tested.

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
CLC ((Zlass Complexil~ .) 3’hta c-yt-lomatic- c-onil)lcxir\ of the c-olltl.ol,~~-a~)h
fol-med by a union of all mrthod control graphs with a s(atr transltlon graph
for the class.

NOM (Nominal Number of Methods per class.) 20 is suggested as an upper

limit. NNOM = NNOF + NN0P.t

NOP (Nominal Number of Procedul-es per class.) Procedures change state. J

-4

TOM (li)tal Number- of hlc~hods peg- class.) Samt as NNOM, hut includes all
inherited methods. TNOM = ‘I‘NOF + ‘I’NOI’.
More methods require more testillg.$

TOF (Total Number of Functions per class.)+ J

TOP (Total Number of’ I’row(lr~~-cs )WI- class.) If there ate 11 procedu~-es with J
no sequential activatloii (x)ns(riliut, at Icils( nL, lcstb al-e necessary to verifj
state control.$

#Under investigation by the au1hor

Table 5. DFT StrategieS

standard drivel

Self-Test Built-in Self-Test (BIST). On-chip Generic driver, executable

ability to generate and execute tests specification packaged with the
using standard BIT implementation
 quired at considerable cost and fishbone is shown in Figure 8. ~l‘he tool and method training, iuo11\ ,I-
should be treated accordingly. The key process abilities of the defined tion, and appropriate educatiotl at~tl
reuse of a test suite (regression test- level for software product engineer- experience.
ing) provides economic justification ing [23]. .l‘esting is most effective when it is
for careful test development. Over Constunf~ of purpose is necessary foi viewed as an essential component of’s
time, a reusable test suite increases continuous quality improvement system of production. It is neither a
the amount oftesting that can be done, [ 121. Staff and management need to magic filter to remove problems with
and hence increases testability. Effec- be empowered to perform adequate an upstream source nor a dumping
tive test suite reuse requires configura- testing. ‘l‘his requires sufficient fund- ground for hastily developed code.
tion management c.onlrol and tract- ing, ~countability, and an unequivo- An integrated test stratea views testing
ability to success& \,crsions of rht cal. Gnc-cl-e commitment to establish- in context. Vertical integration means
representation and iiilplCilieill;iiioii. ~IIK ;III(I itilproving the process. there is a well-defined path among
Test case develop~~~e~it is LI clial- ‘l’lic lc.c.ll~lical approach must be rj- tesf processes for classes, clusters of
lenging activiry lllill is 5ustcplil)lc lo /fvtliW. I IIc o\c’I all development pro- classes, and application systems. Hor-
errors similai- 10 lliosc. occ Ill.ritig iI1
software tl~~\~c~lo~“ll’~“l. 1“;11111\ I(“\1
cases cati 1)~ pi~ol~l~rii;ilic~. iii 111;11 ;I

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
corrw ilill,l~i1iciil;ilioii III;I\ Ix. I.(‘- Oracle Reusable
jectetl o r iii(oi rv~l iiiil)l~.iii(.iil;ii~(~ll
m a y b e ac.wplc.~l. l+:st;il)lislic~tl ICX II-
niqur5 loi \\()I-k ~IXKIII( I ~.(.\ic.\\ .IIIII
defec1 l)t.c’\(‘“lic)ll w i l l iltll)lo\(. ICYI
suites atltl iiic.l.c;iw, lIlc.ii c 01111 il)iilioil
t o trslill)ilil)
Test tools. ‘lC,s1illg I(YIII~I~Y .IIIIO-
malioil. \\‘ilIio~il ;iiilotii;ilioii It.\ 1~3.
ing will 1)~. tloiic, 01’ gt’c’;11(‘1 (()\I \\ ill 1)~.
inciii.rc~(l IO ;I(.llic\.c ;I gi\,c,ri rc~li;il)ilil\
goal. I IN. ;ll)~.tl(.c 01 Ioola iilliil)ils
testahilil!. ~I‘lic co1i1l~mrn1s 01 ;I 1c’sT
envii o~illlclil Ii;ivc I)c‘(‘Il ;111;11)/(.d
rlsc\ll1c.t.c~ I(il. Basic c‘o~~lpo~leiils iii-
clutlt (~oilli~iii~;ition InallilgenlCnt,
trsl hiiil(. 111;111;1ger, s t a t i c ailaly~er, Test Incident Report
itisli.utllc.titoi., run-time trace, coni-
pal‘atol‘. reporting, and integrated Closed Loop
Assessment
c1eh111. ‘l’est case development is sup-
porlrtl b y i n p u t c a p t u r e s y s t e m s ,
scl-ipt editors, spec-based generator, Verified Documentation
code-based generators, input data
generators, and support for other
developer-defined tests. The test bed
(test environment) needs functions to
initialize a system and its environ- cess must be defined a11tl I c~J~;II;II)Ic Figure 6. Testability: Test suite
ment, execute test scripts, and replay [23]. A rustor/lr,.-o,,rrr/trl/ 1~31 process is
scripts under predefined conditions. driven by customer expectations fi)r
I‘here are many commercial offerings cost and reliability. Systems should
for such tools. These components, not be tested unless a rrrrd/~~ rn.~~.\-
with the necessary changes, are merit has been performecl. .l‘hia pre- izontal integration means that test-
equally useful for object-oriented vents waste of testing resources on ability is considered over all stages of
implementations. Several test tools trivial problems more easily pre- development: representation, (re-
for object-oriented systems have re- vented by design or programming. quirements and design), implementa-
cently become commercially avail- Closed-loop feedback is necessary to as- tion (code), test suite (testing), and
able. Intel-operability is a key con- sess testing effectiveness. For exam- subsequent iterations of reuse and
cern, as it is with CASE in general. ple, if the desired level of post-release maintenance. Verification and valida-
The Test Tool fishbone is shown in reliability is not obtained, a process tion integration means that testing is
Figure 7. improvement cycle should be initi- used in a balanced, optimal mannel
P r o c e s s c a p a b i l i t y . Overall soft- ated to determine and correct the with other quality assurance prac-
ware process capability and maturity cause of the problem. Testing is a tices: prototyping, inspections, and
can significantly facilitate or hinder challenging task requiring high staf’ reviews for all stages and woi-k
testability. The Process Capability capability. This results from adequate products.
Toward High Testability For
Object-Oriented Systems Configuration Management Runtime Trace
Software testing is an economic prob-
lem closely intertwined with nearly all
Test Suite Management
major technical issues in software
engineering. The implication of the
foregoing analysis is hardly startling:
testability results from good SO~~M~;I ~C‘
engineering practice and an elfectivc
software process. Figure !I hllo\vs III<
entire testability fishbonc. WC’\,C~ ~CCII
how these basic li~ccca colllrihlrc IO
testability. HOWCVCI-, ;I III;IIILI;I~ ap-
preach t o testitlg M ill clt~ic~kl~ ~.a(.11
practical limits Ilr;ll ;kIx 1111s;lli~l;l(.lol.)
given the c’sIt’III 01’ Ic3litlg Ilc~c~tl~Yl lo
assure higll i.~il.s;ibilil! ;riitl rc4i;ibilil~~.
Just as high tesl;tbilit) li)~- \‘I.SI l-e-

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
quires an explic,il 1~51 iiil’r;ihli iic Iurc,
so will higll lesr;il)ilil! loi. l;irgc ol?ject-
oriented a~41~~111b. .\I~\;IIIC c,tl IcW 2i110-
mation is ~~~YxI~YI.
A n exlellsioll 01 111~. COIIIII~OI~ n o -
tion of ;I tlii\cl .IIICI III’I’ Iiiiiclions
suggests 50111c it~t) ig:llillg 005sil)ilities
for adv;i~ic~~l I('\1 .illlolli;ilioii. :\n in-
structivc. C'OIII~KII i\ol~ (11 k.v~lb 01‘ ab- Test Bed Test Case Development
straction iii \‘I .SI ;III(I (.I;155 libraries is
provided iri 1I I) ~t.iritl;r~~l~~~ilion is
needed to ac.llic~\c~ ;I Iligll lc~cl ol‘eco-
nomic efticiclic\ .IIICI ~i~I~~~ol~~i~;~l~ility. Constancy of Effectiveness
The "gauge" ih cc.1111.11 IO Illis \,ision. Purpose
A gauge is ;I IX~II.GIIII~. ~~II\\;II.(.(oIII~o- Defined and Repeatable
nent usccl IO I(‘31 ;I \III~I(. (I;155 ;intl is
routinel) ~)IIK~II( (~1 ‘15 ~);ii‘l 01. t h e Customer-oriented
impleniriil;~lioii. .\ii 0iIIliiic. strategy
Test Readiness Assessment
for ilnplcrr~~il~;itic,rr CII tlic, g:;lclge con-
cept fOllows. ‘l’hl~~ .TI ~'~~III~);II‘c's VLSI Adequacy Assessment
DFT stl-;llcgic,s ;III(I ( I.145 I(,slillg. l’os-
sible c.ollli~lll.;l~iolIs IOI. c.orl-~~pond-
i n g ol?j~c.l-c,i.i~rlt[,(i IIF slralcgies
are tlvl)ic-tc,tl iii 1;iglil.c. IO ant1 sum-
marizetl itI ‘l’;il~l~~ .G.
Furlllcr (lisc tls\ioII 01‘ elements in
Table 5 fi)llow~:
l No Ut;7: ‘1’0 the extent that class
testing is done, there is no system-
atic DFI‘. ~l‘he implementation is
developed and the tester copes.
This typically means it is difficult to
control and observe the CUT. This
approach will result in the highest

V and V Integration

Figure 7. Testability:
Test automation

Figures. Testability: Process

capability

96 seprcdxr 1994/V"L3i. N"9 COYM"*,CI~I*WIorr"crcm

 Requirements Specification
Oracle Reusable

b Representation

Test Executwn Log

Traciability Separation Verified Documentation

of Concerns

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
Conhguratum Management Runtme Trace
Structure Fault Sensitivitv l-
Test Sute Management Comparator
I I - I
Stabc Code Analyzer Incident Tracker

hlstrumentor

Standard Mechan,sm

External API

Control Flow Non-determtncmc Control

External Determinism Exce(;tion

Interface Handling
Test Bed Test Case Development

Constancy of Effectiveness
Driver Set/Reset Safety
Defined and Repeatable

Customer-ormled

Test Readiness Assessment

Ad%quacy Assessment

Process Closed Loop Feedback

Capability
Built-in Test

Abstract State

Repdrter Assertions

TESTABILITY
Flguee9. The testability fishbone: All facets
 by developing a driver for each
Thread or use builds class and stubs (as needed) for the
No DFT for the CUT class’s servers. Assertions and other
Build n I
forms of inline instrumentation may
be added. This improves controlla-
bility and observability. However,
the absence of explicit set/reset can
result in lengthy setup sequences
and reliance on debug probes to
verify state. This results in a rela-
tively high test cost or less reliabil-
ity, depending on the basic testing
strategy.
l Structured DFT. The ad hoc strat-
Test Results
Ad Hoc DFT egy is augmented by including a
standard BIT and BIT application
programmer interface (API) in all

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
classes. As discussed previously,
minimal BIT requires state set/reset,
state reporting, and BIT-safe imple-
mentation. This can offer complete
controllability and observability.
However, a single driver must be
developed for each class. To de-
velop the driver, the CUT interface
is hard-coded in the driver. The
t Test Results requisite interface information is
obtained by manual inspection of
the CUT. Test cases must be manu-
Structured/Standardized DFT
ally prepared for each class.
.- l Standardized DFT. The agreement
ma on and adoption of an industrywide
4 VLSI standard for structured DFT
improved IC interoperability and
reliability, as well as reducing costs.
R- Individual IC manufacturers with a
high degree of vertical market inte-
gration practiced structured DFT
long before the adoption of IEEE
1149. As the VLSI industry became
less vertically integrated, however,
interoperability needs provided a
Advanced DFT
win-win economic opportunity,
leading to the rapid and wide-
spread adoption of 1149. An analy-
Class n
sis of how a similar scenario might
Class 2 develop for commercial class librar-
.
Class 1 : ies is beyond the scope of this arti-
. - T-spec cle. However, within a single devel-
Driver * opment organization, there are
*
clear benefits to a standard test API
for class drivers and class BIT: one
API to learn instead of many, more
time for better testing, and consis-
tent test capability over the entire
library.
l Self-Test DFT. The class testing

Figure lo. DFT COtIfigUratiOtIS possible test cost or least reliability, analog to the structured strategy for
depending on which basic testing calls for a driver to be paired with
strategy is followed. every class. This has several nega-
l Ad hoc DFT. Class testing is done tives. With n classes to test there are

98 September 1994/Vd.37, No.9 COI*M”“ICI~O”SOC~. ICY

 Interpretive t I
Driver Driver 4-b

Test Results
___) B I T

Dispatching T-spec
Driver

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
Driver

Test Results

Generated

Symbolic
Driver
T-spec

Driver 1 CUT

Test Results

now n additional interfaces to syn- The use of an implementation as Figure 11. Advanced DFT
chronize, the additional develop- the exclusive basis for automated test- architectures
ment and maintenance effort for ing is essentially tautological and
every class driver, and limited auto- therefore insufficient. There must be
mation of test case generation, exe- a specification and a means to gener-
cution, and evaluation. ate expected results from the specifi-
 by the driver. The scheme is essen-
tially an extension of the technique
used to run programs under a sym-
bolic debugger. Thus, automatic test
generation could be developed by
simply using the same input as a
symbolic debugger.
A suitable test specification lan-
guage with respect to some testing
methodology must be constructed.
Figure 12 provides a sketch of a spec-
ification language to automate the
state-based testing approach in the
FREE methodology [9]. Inclusion of
such a specification would not impose
an onerous coding burden. For ex-
ample, the Eiffel language provides

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
an elegant executable specification
for pre-conditions, post-conditions,
and class invariants [19]. Only state
control information needs to be
Figure 12. FREE test solutions to the interface binding added. Inclusion of an explicit test
specification problem, which are depicted in Fig- specification would also serve other
ure 11. Further discussion of ele- important software engineering
ments of Figure 11 follows: goals:
l Interpreter. The test specification
cation. Ideally, a single generic driver l Provide an explicit, specification-
exported from the CUT contains
would automatically generate a test based test plan for every class.
the names necessary to create a bin-
suite from an embedded test specifi- l Prevent fragmentation of specifi-
dable CUT message. The driver
cation (t-s+), activate the CUT with cation information and the
uses these names to generate a pro-
this test suite, and evaluate the re- implementation.
gram passed to the interpreter. The
sults. The need for a driver paired l Provide an explicit, readable spec-
interpreter accomplishes the bind-
with every class is obviated. The self- ification embedded with the imple-
ing in the normal manner.
test procedure would consist of the mentation. This would facilitate
l Test dispatcher. Instead of activat-
following steps: reuse and understandability.
ing features in the CUT directly,
the driver activates test methods in The BIST (built-in self test) ap-
1. The driver requests the t-spec
the CUT which in turn activate proach has limitations-it is not a
from the CUT.
application methods in the CUT. replacement for all forms of testing.
2. The driver generates a test pat-
The same test method interface is Unless a general solution for an auto-
tern from the t-spec.
used throughout the application mated oracle is found (this is not
3. The driver generates test case
system. The dispatcher selects the likely), manual preparation of test
values by splattering the input pa-
rameter domains (random sampling actual message interface to use and cases will continue to be necessary.
then sends the message. Control is The test specification language in
of from off, on, and in regions in
returned to the BIT dispatcher, Figure 12 can define the bounds of
the domain).
and then back to the driver. acceptable input and output, but
4. The driver uses set/reset to con-
l Driver generator. Instead of at- does not permit determination of the
trol the state of the CUT.
tempting to drive the CUT directly exact correctness of a given input and
5. The driver sends the test case
message(s) to the CUT. from a generic driver, the generic output. For example, given only the
driver reads the CUT source code pre- and post-conditions for a square
6. The driver evaluates the re-
sponse of the CUT. and generates an intermediate root function, we could not decide
driver to activate the CUT under whether any given value was indeed
7. The driver evaluates the state of
control of the primary driver. This the square root of the argument (e.g.,
the CUT with the reporter.
scheme has been investigated in the is 4.25 the square root of 18.0625?),
The feasibility of this scheme de- ASTOOT system [13]. but we could decide that was within
pends on solving two problems. First, l Language extension. The lan- the bounds (post-conditions) of a
rhe test pattern method names and guage translator is modified to ac- square root function. The t-spec
arguments must be bound to mes- cept syntax extensions for test speci- should be consistent with all other
sages accepted by the CUT. The man- fications. After translation, portions representations. A full discussion of
ual step of inspecting the CUT to pre- of the symbol table and the test considerations and implementation
pare the requisite interface must be specification are saved as a separate of automatic consistence support is
alltomated. There are several general file that can be directly processed beyond the scope this article.

100 Srptember 1994Wol.37, No.9 COYI”“ICIWONS~~~..~~

 sign. IEEE Trans. Softw. Eng. 20, 6
Conclusion (Jun. 1994), 476-493.
Testability reduces total test cost in a ll. Cox, B.J. Planning the software in-
reliability-driven process and in- dustrial revolution. IEEE Softw. Nov.
c r e a s e s r e l i a b i l i t y i n a resource- 1990, 25-33.
limited test process. Thus, regardless 1 2 . Deming, W.E. Out of the Crisis. MIT
of the testing strategy employed, it is Center for Advanced Engineering
in the interest of a development orga- Study, Cambridge, Mass., 1986.
nization to improve testability. Test- 1 3 . Doong, R.K. and Frankl, P. The AS-
ability of an object-oriented system is TOOT approach to testing object-oriented
programs. Tech. Rep. PUCS-104-91.
a result of six primary factors: repre-
Polytechnic University, Brooklyn,
sentation, implementation, built-in
N.Y, 1991.
test, the te st suite, the test support
1 4 . Freedman, R.S. Testability of software
environment, and process capability. components. IEEE Trans. Softw. Eng.
Each influences controllability and 17, 6, tJune 1991), pp 553-64.
observability of the implementation, 1 5 . Hoffman, D. Hardware testing and
or basic operational effectiveness of software 1Cs. Proceedings, Northwest
Software QuaSty Conference (Portland

Downloaded from the ACM Digital Library by Yuan-Ze University on April 10, 2025.
the test process. Nearly all the tech-
niques and technology for achieving Ore., Sept. 1989), pp 234-22.
high testability are well established, 1 6 . Hoffman, D. and Strooper, P. A case
study in class testing. In Proceedings of
but require financial commitment,
CASCON 93. IBM Toronto Labora-
planning, and conscious effort. The
tory, Oct. 1993, pp 472-482.
advanced built-in test capabilities
1 7 .Jacobson, 1, Christerson, M., Jonsson,
sketched here do not yet exist, but are I’. a n d Overgaard, G. Oblect-Onentrd
feasible with existing technology. 0 .%@a~-r Eqizc~&g. Addison-Wesley,
Reading, Mass., 1992.
References 1 8 . Lieberherr, K.J. and Holland, 1.M.
1. ANSI/IEEE Standard 830-l 984: Stan- Assuring good s t y l e f o r object-
dard for Software Requirements Spfcifjca- oriented programs. IEEE .So/iw. (Sept.
Lions. The Institute of Electrical and 19X9), pp. 38-49.
Electronic Engineers, New York, 19. Meyer, R. Obj&-Or&ted Softural-r Con-
1984. struction. Prentice-Hall, Englewood
2. ANSI/IEEE Standard 829-1983: IEEE Cliffs, N.J., 1988.
Standard for Software Test Documenta- 20. Ogata, K. Modern Control Engineering.
tion. The Institute of Electrical and Second ed. Prentice-Hall, Englewood
Electronic Engineers, New York, Cliffs, N.J., 1990.
1987. 2 1. 00 tool aids software testing. The Out-
3. ANSI/IEEE Standard 1016-l 987: IEEE look. Fall 1993, McCabe & Associates,
Recommended Practice for Software De- Columbia, Maryland.
sign Descriptiolw. The Institute of Elec- 22. Ould, M.A., and Unwin, C. Testing in
trical and Electronic Engineers, New Software Development. Cambridge Uni-
York, 1987. versity Press, Cambridge, UK, 1986.
4. ANSI/IEEE Standard 1149.1-l 990: 23. Paulk, M.C., Curtis, B., Chrissis, M.B.,
IEEE Standard Test Access Port and and Weber, C.V. Capabilio Maturity
Boundary-Scan Architecture. The Insti- Model for Software, Version 1.1. CMUJ
tute of Electrical and Electronic Engi- SEI-93-TR-24. Software Engineering
neers, New York, 1990. Institute, Pittsburgh, Pa., 1993.
5. Beizer, B. Software System Testing and 24. Voas, J., Morell, L., and Miller, K.
Quality Assurance. Van Nostrand Rein- Predicting where faults can hide from
hold, New York, 1984. testing. IEEE Softw. (March 1991), pp
6. Beizer, B. Software Testing Techniques, 41-8.
Second ed. Van Nostrand Reinhold, 25. Weyuker, E.J. On testing non-testable
New York, 1990. programs. Comput. J. 1982 25, 4, pp
7. Berard, E. Essays on Object-Oriented 465-70.
Software Engineering. Prentice-Hall, About the Author:
Englewood Cliffs, N.J., 1993. ROBERT V. BINDER is president of
8. Binder, R.V. Testing Object-Oriented RBSC, Inc. He is researching state-based
Programs: A Survey. RBSC-94-002. testing for reliability engineering in ob-
Robert Binder Systems Consulting, ject-oriented systems and a systems engi-
Inc., Chicago, 1994. neering methodology for client-server
9. Binder, R.V. The FREE A#roach to development. Author’s Present Address:
Testing Object-Oriented Systems. RBSC- RBSC, Inc., 3 First National Plaza, Suite
94-003, Robert Binder Systems Con- 1400, Chicago, IL 60602; email: rbinder
sulting, Inc., Chicago, 1994. @chinet.com
10. Chidamber, S.R. and Kemerer, C.F. A
metrics suite for object-oriented de- 0 ACM OOOZ-0782/94/0900 $3.50