A Forrester Consulting

Thought Leadership Paper

Commissioned By Tricentis

July 2018

The Definitive Software

Quality Metrics For
Agile+DevOps
Measuring The Risk Of A Release Candidate
 Table Of Contents
1 Executive Summary
2 Agile+DevOps Leaders Understand
They Must Achieve Quality At
Speed
6 The Unseen Barrier To Quality At
Speed: Risk
8 Manage Business Risk In
Agile+DevOps With Effective
Metrics
11 Key Recommendations
12 Appendix

ABOUT FORRESTER CONSULTING

Project Director: Forrester Consulting provides independent and objective research-based consulting
Andrew Magarie, to help leaders succeed in their organizations. Ranging in scope from a short
Senior Market Impact Consultant strategy session to custom projects, Forrester’s Consulting services connect you
directly with research analysts who apply expert insight to your specific business
Contributing Research:
challenges. For more information, visit forrester.com/consulting.
Forrester’s Infrastructure &
Operations research group © 2018, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is
strictly prohibited. Information is based on best available resources. Opinions reflect
judgment at the time and are subject to change. Forrester®, Technographics®,
Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks
of Forrester Research, Inc. All other trademarks are the property of their respective
companies. For additional information, go to forrester.com. [1-169LWT7]
 Executive Summary
Delivering high-quality software fast is no longer a luxury — it is a
prerequisite for digital success and mandatory for customer loyalty and
acquisition. We are in an era where customers are empowered to choose
and switch to the company that can provide them the best experience
quickly and at minimal cost. Organizations that cannot keep up with their
customers’ expectations and adapt to new business paradigms will find
themselves losing customers and business to competitors that can.
How can firms deliver with quality at speed? Implementing Agile+DevOps
best practices and building in quality is the first step — and this is not
trivial. Automation is the lever that development teams pull to go faster.
However, if not properly managed, automation can jeopardize quality and
increase risk. To ensure that quality issues and unacceptable risk do not
negate the velocity benefits of automation, firms must ensure they are
tracking the right metrics to measure quality throughout development-
testing-deployment.
In March 2018, Tricentis commissioned Forrester Consulting to evaluate
current software development and delivery priorities as well as key metrics
To ensure that quality tracked throughout the software development life cycle. To explore this
issues do not negate topic, Forrester conducted an online survey with 603 global enterprise
respondents responsible for their firms’ Agile and/or DevOps strategies.
the velocity benefits
of automation, firms KEY FINDINGS

must track the right ›› Five core practices separate successful DevOps leaders from
laggards. Firms that are more mature in Agile+DevOps do five key
metrics to measure things differently: 1) they allocate proper testing budgets and focus
quality throughout on upgrading their testing skills; 2) they implement continuous testing
to meet the demands of release frequency and support continuous
development. delivery; 3) they include testers as part of their integrated delivery teams;
4) they automate end-to-end functional testing; and 5) they shift-left
testing to earlier in the development life cycle. For firms that follow these
five best practices, testing is a way to gain speed and efficiency, rather
than a bottleneck to delivering faster.
›› Business risk must be accurately and continuously managed.
Automating software delivery without being able to accurately measure
software quality is dangerous. Firms must understand the business
risk — the chance of a negative user experience — that each software
application carries when deciding whether to release. Unfortunately,
most firms today admit they have gaps in their ability to accurately
assess and manage business risk in testing.
›› Leaders ruthlessly drive end-to-end test automation. With a better
indication of risk from their testing metrics, DevOps leaders can
relentlessly pursue automation to develop with quality at speed. They do
this by prioritizing end-to-end automated business process test cases.
They automate test design and test execution, and they orchestrate the
automation of the overall dev-test-deploy process.

1 | The Definitive Software Quality Metrics For Agile+DevOps

Agile+DevOps Leaders Understand
They Must Achieve Quality At Speed
Development teams today must deliver high-quality software at speed
with an eye on cost. To meet this ambitious goal, firms continue to
steadily adopt both Agile and DevOps — preferably together. Forrester
Analytics’ Q3 2017 Global Agile Software Application Development
Online Survey data shows that benefits, such as delivery frequency
(83% vs. 75%), technical quality (64% vs. 29%), and functional quality
(72% vs. 47%), dramatically improve when teams are leveraging both
Agile and DevOps together, leading to faster business value (see Figure
1
1). For this reason, firms serious about digital transformation must
adopt Agile+DevOps practices — and now.

72% of firms agree that testers are critical to continuous

delivery success.

Figure 1
Benefits of Agile+DevOps greatly outweigh separate initiatives
Running Agile transformation Running Agile and DevOps as one
separate from DevOps transformation

75%
Greater frequency of delivery
83%

59%
Firms running
Better business/IT alignment
74% Agile+DevOps as a
47% single transformation
Improved functional quality
72% are 35 percentage
41% points more likely to
Faster business value
69% see improved
Improved technical quality
29% technical quality and
64% 28 percentage
Greater predictability of releases
35% points more likely to
55%
see faster business
Greater predictability of results 22% value in the software
47%
aligned with requirements
they release.
24%
Continuous delivery
47%

Base: 230 respondents at companies doing some form of Agile

Source: Forrester Analytics’ Q3 2017 Global Agile Software Application
Development Online Survey

2 | The Definitive Software Quality Metrics For Agile+DevOps

 Software testing is a key part of the Agile+DevOps life cycle — 72%
of firms say testers are critical to continuous delivery success. To this
end, more mature firms are transforming software testing to become
continuous testing by implementing new testing best practices. These
firms (see Figure 2):
›› Focus on building their skills while budgeting testing sufficiently to
implement Agile and DevOps practices across the organization.
›› Implement continuous testing in response to the demand for faster
releases.
›› Include testers as part of the integrated delivery team.
›› Automate end-to-end functional testing.
›› Implement testing earlier in the development life cycle (shift-left
testing).
While most Agile and/or DevOps firms have implemented some of these
five core testing practices, few have actually implemented all of them —
only about a quarter of respondents say they somewhat or completely
agree with all five best practice statements. This is an important
distinction. While many CXOs and high-level decision makers believe
they are ahead of the pack, our research shows that only the small
fraction of firms that follow these best practices have more advanced
Agile and DevOps practices compared to their peers — and their
behaviors and attitudes support that assertion (see Figure 3).

Figure 2 26%
“How much do you agree or disagree with the following statements Follow best
practices
about your continuous delivery organization and practices?”
(Only “Somewhat/completely agree” are included)

67% Testers are included as part of the integrated delivery team

59% We have sufficient skills and budget to successfully implement

Agile/DevOps practices across the organization

59% We’ve implemented continuous testing in response to the demand

74%
for faster releases
Others

53% We have automated end-to-end functional testing

Only 26% of firms we
surveyed follow all
53% We have shifted all types of testing to earlier in the development
life cycle (shift-left testing) five of these
continuous testing
Base: 603 enterprise Agile and DevOps decision makers in North America, EMEA,
and APAC best practices.
Source: A commissioned study conducted by Forrester Consulting on behalf of
Tricentis, March 2018

3 | The Definitive Software Quality Metrics For Agile+DevOps

 Figure 3
“How mature do you consider your Agile and/or DevOps practices compared to your peers?”

Follow best practices Others

Agile DevOps

35% 22%
Far ahead Far ahead
14% 12%

35% 44%
Slightly ahead Slightly ahead
37% 34%

30% 35%
On par or behind On par or behind
49% 53%

Base: 603 enterprise Agile and DevOps decision makers in North America, EMEA, and APAC
Source: A commissioned study conducted by Forrester Consulting on behalf of Tricentis, March 2018

These more advanced firms focus on automation with high obsession.

They:
›› Consider end-to-end automation a business differentiator. Firms
that rely on manual testing are more than twice as likely to report that
testing is a bottleneck (59%), compared to those that don’t (26%).
Automating the software delivery life cycle is a crucial component
of accelerating high-quality software delivery. Automating software
quality streamlines what can be highly manual and time-consuming
processes. A majority of firms (53%) that follow Agile+DevOps best
practices consider automating their software quality processes a
critical differentiator. This is almost twice the number of “other” firms
(27%) that feel it’s critical (see Figure 4).

Figure 4

!
“Compared with other IT processes in your organization, rate the
importance of automating your software quality process.”

Follow best practices (N = 157) Others (N = 446)

Critical — considered as a 53%

Over half (53%) of
business differentiator 27% firms that follow
Important — considered as a 42% Agile+DevOps best
strategic priority 54%
practices consider
Somewhat important — 4%
considered as a tactical initiative 17%
automating the
Not important — considered as an 1%
software quality
afterthought if at all 2% process CRITICAL,
compared to just
Base: 603 enterprise Agile and DevOps decision makers in North America, EMEA,
and APAC 27% of other firms.
Source: A commissioned study conducted by Forrester Consulting on behalf of
Tricentis, March 2018

4 | The Definitive Software Quality Metrics For Agile+DevOps

 ›› Automate more testing automation to increase benefits.
More advanced Agile+DevOps firms have taken the lead with
automating QA and testing processes in the development
pipeline. Our survey shows that more advanced firms are more
likely, by 23 percentage points on average, to have automated
more than 50% of their testing proceses (see Figure 5).
Agile+DevOps leaders understand the major benefits that testing
automation can bring to their development organziation and are
taking steps to implement automation wherever possible in the
development pipeline.

Figure 5
“Using your best estimate, how much do you automate each of the
following?” (Showing top 10 for “Over 50% automated”)

Follow best practices (N = 157) Others (N = 446)

More advanced
61%
Acceptance testing
34%
Agile+DevOps firms
are more likely by 23
61%
Integration testing
36% percentage points
on average to have
58%
Unit testing
36% higher levels of
58%
automation for key
API testing
37% testing and QA
Test data management/ 57%
processes.
provisioning 33%

57%
Test environment provisioning
30% 52% of more
Load/performance testing
57% advanced
43%
Agile+DevOps firms
Mobile application testing
54% have automated more
31%
than half of their
Test case design
53% risk-based testing,
28%
compared to just
Functional test automation (GUI)
53%
27% of other firms.
32%

Base: 603 enterprise Agile and DevOps decision makers in North America, EMEA,
and APAC
Source: A commissioned study conducted by Forrester Consulting on behalf of
Tricentis, March 2018

5 | The Definitive Software Quality Metrics For Agile+DevOps

The Unseen Barrier To Quality At Speed: Business risk starts
Risk accruing early in the
Automating the software development life cycle is an imperative for
software development
accelerating the speed and frequency of releases. However, without life cycle and can
an accurate way to measure and track quality throughout the software snowball throughout
development life cycle, automating the delivery pipeline could increase the
risk of delivering more defects into production. And if organizations cannot development if not
accurately measure business risk, then automating development and testing assessed quickly.
practices can become a huge danger to the business.
For this study, we use the following definition of business risk: any application
shortcoming that impairs the end user’s (or customer’s) expected experience
and ultimately erodes confidence in the business. Business risk is different
for every application and is compounded by the complexity of each
organization’s architecture and transaction dependencies. If firms are unable Figure 6
to measure relative risk for each application as software is being designed, “How important are each of the
developed, integrated, and tested, then, ultimately, they will not know the following to the business when
risk that a release candidate carries. Tolerance for risk is solely dependent on determining the success of a
senior management. No matter where a firm sets the bar for risk, firms must customer-facing software release?”
be able to continuously measure risk to ensure final products are delivered (Only “Very important” responses are
within levels of risk tolerance. included)
Firms looking to use DevOps to automate their software delivery must
understand and accurately measure business risk. But even more advanced 53% Projects or products are of high
Agile+DevOps firms struggle to get an accurate view of risk through the quality
software delivery pipeline. When comparing firms following Agile+DevOps
testing best practices, we see that: 43% Projects or products are
delivered on time
›› Risk relevance is not on par with quality or speed. Most firms have
not yet made the connection between speed, quality, and risk. Overall,
39% Projects or products are
the importance of risk in customer-facing software lags behind the
delivered within budget
established development goals of quality on time and on budget (see
Figure 6). Overall, only about a third of respondents say it’s very important
34% Projects or products are within
for success that customer-facing software be delivered within acceptable
acceptable business risk
business risk. When looking at firms that follow best practices, this
number jumps to 50%, but risk still lags quality, on time, and on budget. Base: 603 enterprise Agile and DevOps
decision makers in North America, EMEA,
›› Firms are confident in their abilities to deliver within acceptable and APAC.
risk . . . Most firms believe they deliver customer-facing products within Source: A commissioned study conducted
acceptable business risk. A staggering 80% of respondents say they can by Forrester Consulting on behalf of
Tricentis, March 2018
often or always do this (see Figure 7).
›› . . . but admit there are gaps in their testing processes . . . Fewer than a
quarter of firms think that their QA and testing processes completely cover 50% of firms following
business risk in all phases of testing. Although those firms that follow
Agile+DevOps best practices do better, fewer than half (38%) cover risk
Agile+DevOps testing
completely in all testing phases. best practices think
›› . . . and acknowledge the shortcomings of their test suites. Most delivering within
firms recognize that their test suites do not always give them a good
indication of business risk. In fact, just 15% of respondents say this is
acceptable business
the case today — and nearly 40% say that they have a good indication risk is very important,
sometimes or less often. Even more advanced Agile+DevOps firms see compared to just 29%
the limitations here: Fewer than one-third of them say their test suite
always gives them a good indication of business risk. of other firms.

6 | The Definitive Software Quality Metrics For Agile+DevOps

 Given that most firms, even the ones following continuous testing best
practices, admit that their software testing processes have risk gaps and
do not always give accurate measures of business risk, it stands to reason
that the 80% who say they always or often deliver within acceptable risk
may be overestimating their capabilities. And, given the critical importance
of automating the software delivery pipeline to deliver faster, being able to
say with certainty that a software release is both high-quality and within
acceptable business risk is a crucial part of not just delivery automation,
but also overall software delivery success.

Figure 7

Most firms believe they deliver within acceptable business risk . . .

We always or often deliver customer-facing products
80%
that are within acceptable business risk

. . . but admit they have risk gaps in QA/testing . . . . . . and recognize their test suite does not always give
a good indication of business risk
“How well do your current QA/testing processes cover “How accurately does your test suite allow you to
business risk today?” measure business risk?”
Follow best practices (N = 157) Others (N = 446)

Covered completely in all phases 38% We always have a good indication of 32%
of testing 18% business risk from our test suite 8%

Covered completely in some phases, 48% We often have a good indication of 48%
and somewhat in the remaining 47% business risk from our test suite 47%

Covered somewhat in all phases We sometimes have a good

12% 16%
of testing indication of business risk from our
28% 34%
test suite
Covered somewhat in some 1% We rarely have a good indication of 2%
phases of testing 6% business risk from our test suite 8%

0% We never have a good indication of 2%

Not at all
1% business risk from our test suite 3%

Only 23% of overall respondents say they cover Just 15% of overall respondents say they always have
business risk completely in all phases of testing. a good indication of business risk from their test suite.

Base: 603 enterprise Agile and DevOps decision makers in North America, EMEA, and APAC
Note: Percentages may not total 100 because of rounding.
Source: A commissioned study conducted by Forrester Consulting on behalf of Tricentis, March 2018

7
Manage Business Risk In Agile+DevOps
With Effective Metrics
To build high-quality software at speed, organizations must build in quality
and automate as much as possible during all phases of the continuous
delivery pipeline. As the saying goes: “you can’t manage what you don’t
measure,” so firms need to be able to define and track the right metrics
that can help enable quality at speed while minimizing risk. Therefore,
metrics must be defined and tracked to determine if the software release
candidate has an acceptable level of risk.

THE KEY SOFTWARE QUALITY METRICS TO MANAGE RISK

While firms today use a wide array of metrics in software development
and testing, Agile+DevOps have undeniably changed the mission of
software testing practices. By examining the metrics that firms following
Agile+DevOps best practices use and find critical to success, firms can
pinpoint the crucial metrics that manage risk and promote DevOps maturity
(see Figure 8).

TOP QUALITY METRICS THAT MINIMIZE RISK

Examining the metrics Agile+DevOps leaders track in the various phases of
development leads to several important considerations (see Figure 9):
›› In the build phase, unit testing done well matters. Counting unit tests
is a waste of time but understanding change impact matters. Tracking
“unit” tests prioritized by risk is the key. As the code base evolves,
developers and testers need immediate feedback about change impact.
This feedback is significantly more actionable if prioritized by level of risk.

Figure 8
Top five most important metrics to manage risk suggested by Agile+DevOps leaders, split by development stage

Build Functional validation Integration testing End-to-end regression

testing
#1 Number of automated Requirements covered Requirements covered Percent of automated
tests prioritized by risk by tests by API tests end-to-end test cases
#2 Successful code builds Count of critical New API defects found Requirements covered
functional defects by tests
#3 Unit pass/fail Pass/fail rate API bug density Total number of defects
identified in tests
#4 Total number of defects Bug density API test pass/fail rate Number of test cases
identified executed
#5 Code coverage Risk coverage API functional code Test case coverage
coverage/API risk
coverage (tie)

Base: 157 enterprise Agile and DevOps decision makers in North America, EMEA, and APAC that use Agile+DevOps best practices
Source: A commissioned study conducted by Forrester Consulting on behalf of Tricentis, March 2018

8 | The Definitive Software Quality Metrics For Agile+DevOps

Figure 9
Analysis of key development metrics for advanced Agile+DevOps firms

Percent of advanced Agile+DevOps firms using Percent of advanced Agile+DevOps firms using metric
metric today today that select it as a top three most critical metric

Build phase: Top five most important metrics

Number of automated tests prioritized by risk 27% 63%
Successful code builds 36% 61%
Unit test pass/fail 30% 60%
Total number of defects identified 32% 59%
Code coverage 28% 58%

Functional validation phase: Top five most important metrics

Requirements covered by tests 40% 69%
Count of critical functional defects 21% 66%
Pass/fail rate 46% 64%
Bug density 33% 62%
Risk coverage 34% 62%

Integration testing phase: Top five most important metrics

Requirements covered by API tests 34% 75%
New API defects found 29% 64%
API bug density 39% 63%
API test pass/fail rate 44% 62%
API functional code coverage/API risk coverage (tie) 31% 62%

Integration testing phase: Top five most important metrics

Percent of automated end-to-end test cases 36% 70%
Requirements covered by tests 40% 70%
Total number of defects identified in test 33% 66%
Number of test cases executed 41% 65%
Test case coverage 48% 61%

Base: 157 enterprise Agile and DevOps decision makers in North America, EMEA, and APAC that use Agile+DevOps best
practices
Source: A commissioned study conducted by Forrester Consulting on behalf of Tricentis, March 2018

9 | The Definitive Software Quality Metrics For Agile+DevOps

 Sixty-three percent of these firms consider the number of unit tests
prioritized by risk as one of their top desired metrics. But far fewer can
actually do so — while 34% of advanced Agile+DevOps firms track the
number of unit tests run, only 27% prioritize by risk. And less advanced
Agile+DevOps firms use it even less — just 15% can track the metric
today. Other important metrics tracked in the build phase focus on
ensuring code quality — like the number of successful code builds (61%),
unit test pass/fail rate (60%), and total number of defects identified (59%).
›› When functional testing kicks in, user story/requirements coverage
gets the focus. The top goal for testers in this stage is to minimize risk
by ensuring that the functionality expressed in user stories works as
expected. Extending this coverage concept to business risk coverage is
an additional metric that leading Agile+DevOps firms execute to advance
release automation.
Running functional tests and checking against covered requirements
(69%), the density of bugs (62%) or number of functional defects found
(66%), and the ratio of tests passed vs. failed (64%) are metrics that
successful teams identify as important to manage risk and track quality
during functional testing.
›› To deal with modern distributed architectures, measuring integration
testing and API tests win all around. With application architectures
becoming more decoupled, decomposed in services and microservices,
API testing increases in relevance. So much is going on beyond the user
interface that it’s impossible to keep quality high and diminish risk without
addressing the API layer. Advanced Agile+DevOps firms in the integration
testing phase prioritize many of the same metrics as the functional testing
phase; this time, the focus is on APIs specifically.
These metrics include tests run against functional requirements (75%),
total number of new API defects found (64%) and API bug density (63%),
API test pass vs. fail rate (62%), and API code coverage (62%). Monitoring
API risk coverage is an important metric as well, with 62% of advanced
Agile+DevOps firms prioritizing it as a top three metric for the stage.
›› End-to-end regression testing also gets a first-class citizen role. In
this category, leading organizations automate end-to-end tests at the
process or transaction level. Automating these types of tests is not easy,
and advanced test automation tools are required because speed matters.
The choice of the testing technology matters because achieving and, more
importantly, maintaining high levels of automation is crucial. Therefore,
the more tests that are automated, the better, and so 70% of leading
Agile+DevOps teams prioritize the percent of automated end-to-end test
cases as a top metric.
Many of the other metrics that leading firms rank as important in this stage
are quantitative and measure coverage of functionality vs. requirements
(70%), number of test cases executed (65%), and total number of defects
identified during testing (66%).

10
 Key Recommendations
Forrester’s in-depth survey of global enterprises using Agile and/or
DevOps about the metrics they track in software development yielded
several important recommendations:
Realistically assess and up-level your continuous testing practices
for Agile+DevOps. While many CXOs believe their firms are ahead of
the curve on DevOps, our research shows that only about a quarter
of firms are following testing best practices. Make sure your firm is
implementing continuous testing as part of its DevOps strategy.
Make business risk the driver for your metrics program. Most firms
acknowledge gaps in their ability to manage business risk with the
right metrics but are nevertheless optimistic on risk coverage. This
is dangerous for extreme automation — and exacerbated as delivery
velocity and volume increase. First, you must clearly define risk.
Second, you must continuously measure your exposure versus that
risk. Next, use that understanding to promote releases. By focusing
test automation efforts on processes and transactions that that carry
higher business risk, you reduce the chance that critical defects slip
into production.
Ruthlessly automate your testing and QA processes, end to end.
If your goal is to deliver high-quality software faster (and it should be),
then you need to automate your software development pipeline. More
mature Agile+DevOps firms understand that automation is critical to
driving release velocity, and they are reaping the benefits. Automating
end-to-end testing is a key step in implementing continuous testing
and a top priority for Agile+DevOps leaders today.
Prioritize test execution predicated around business risk.
Understanding which tests deliver the greatest risk coverage is a
significant advantage when speed matters. The ability to prioritize
test execution depends on the implementation of a model that is
collaboratively developed among the development, testing, and
business stakeholders.

11 | The Definitive Software Quality Metrics For Agile+DevOps

 Appendix A: Methodology
In this study, Forrester conducted an online survey of 603 enterprise
organizations in North America, Europe, and Asia Pacific to evaluate
current software testing practices and metrics tracked during survey
development. Survey participants included decision makers and individual
contributors responsible for their organizations’ Agile and/or DevOps
development strategies. Questions provided to the participants asked
about their firms’ attitudes toward software development automation, risk
management, and testing practices, as well as the metrics they track and
value in the software development life cycle. The study was completed in
March 2018.

Appendix B: Demographics/Data
CA 3%
Industry
(Top ten shown)
UK 9%
DE 9% Technology/software 27%

FR 11% IN 8% Financial services and insurance 15%

US 38% Manufacturing and materials 12%
SG 14% Retail 6%

AU 6% Healthcare 5%
Education and nonprofits 4%
NZ 2% Government 4%
Company size Telecommunications services 4%
51% Transportation and logistics 3%
Consumer product manufacturing 3%

25% 25% Title

Vice president 12%

Director 38%
1,000 to 4,999 5,000 to 19,999 20,000 or more Manager 35%
employees employees employees Project manager 8%
Full-time practitioner 7%
“Which of the following continuous development
practices does your organization use today?” “What is your level of responsibility when it comes to Agile
and/or DevOps development strategy at your organization?”
28%
Agile
I am the final decision-maker 36%
development
51% I am part of a team making decisions 41%
Agile+ I influence decisions 11%
DevOps
I am involved 11%
21%
DevOps

Base: 603 enterprise Agile and DevOps decision makers in North America, EMEA, and APAC
(Note: Percentages may not total 100 because of rounding.)
Source: A commissioned study conducted by Forrester Consulting on behalf of Tricentis, March 2018

12 | The Definitive Software Quality Metrics For Agile+DevOps

Appendix C: Supplemental Material
RELATED FORRESTER RESEARCH
“Faster Software Delivery Will Accelerate Digital Transformation,” Forrester Research, Inc., April 12, 2018
“The State Of Agile 2017: Agile At Scale,” Forrester Research, Inc., December 17, 2017
“Master DevOps For Faster Delivery Of Software Innovation,” Forrester Research, Inc., November 21, 2017
“Build The Right Software Better And Faster With Agile And DevOps Metrics,” Forrester Research, Inc.,
September 18, 2017

Appendix D: Endnotes
1
Source: “Agile Only ? No Thanks ! Agile + DevOps, Please!,” Forrester Research, Inc., December 21, 2017.

13 | The Definitive Software Quality Metrics For Agile+DevOps