12/30/2019 Gartner Reprint

Licensed for Distribution

Market Guide for Vulnerability Assessment

Published 20 November 2019 - ID G00367737 - 39 min read
By Analysts Craig Lawson, Mitchell Schneider, Prateek Bhajanka, Dale Gardner

Security and risk management leaders evaluating VA products and services need to
understand the important role they play in risk-based vulnerability management. VA identifies
and assesses vulnerabilities proactively to establish the security and risk posture, not just to
meet compliance mandates.

Overview
Key Findings
■ Vulnerability assessment buyers are shifting from tools that only identify vulnerabilities, to
those that proactively assess and manage the risks posed by those weaknesses. This is
primarily being addressed by new vendors offering vendor-agnostic products, prompting
companies offering solutions to update their offerings.

■ The three VA solutions that dominate the market (Qualys, Tenable and Rapid7) are most often
shortlisted by Gartner clients.

■ Vendors in adjacent markets, such as endpoint detection and response, security information
and event management, IT systems, and configuration management, are adding VA
capabilities.

■ The assessment of standard IT assets across a network is universally supported by VA

vendors. Support for less-common technologies — such as containers, operational
technology/supervisory control and data acquisition, cloud services, and mobile — varies
widely.

Recommendations
Security and risk management leaders responsible for security operations and vulnerability
management who are selecting and operating these solutions should:

■ Evaluate VA solutions’ capabilities for aiding in the prioritization of vulnerabilities and how the
assessment phase is performed, so they can be more-efficiently managed in the organization.

■ Assess the workflow, enterprise management and third-party technology integrations VA

solutions provide with compensating controls. These include intrusion prevention systems,
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
web application firewalls, patch management solutions and ticketing systems, as well as risk
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 1/23
12/30/2019 Gartner Reprint

prioritization tools, such as vulnerability prioritization tools, to support general IT and security
operations with better insight and efficiency possibilities.

■ Select VA solutions with consideration for asset demographics and coverage of emerging
technologies and approaches that you are planning to use, such as cloud and virtualization,
DevOps and software containers. More than one provider may be required.

Market Definition
This document was revised on 25 November 2019. The document you are viewing is the
corrected version. For more information, see the  Corrections page on gartner.com.

The VA market is made up of vendors that provide capabilities that identify, categorize, prioritize
and orchestrate the remediation or mitigation of vulnerabilities. These include unsecured system
configurations or missing patches, as well as other security-related updates in the systems
connected to the enterprise network directly, remotely or in the cloud. Whether delivered on-
premises, in the cloud or in virtual environments, VA products or services have several common
capabilities:

■ Discovery, identification and reporting on device, OS and software vulnerabilities

■ The ability to report the secure configuration of IT assets

■ A baseline of conditions for systems, the applications on those systems, and databases to
identify and track changes in state over multiple periods of time (days, weeks, months, etc.)

■ Compliance reporting with content and format to support specific compliance regimes, control
frameworks and multiple roles in the organization

■ Support for pragmatic risk assessment and remediation prioritization provided by the ability to
correlate vulnerability severity, asset criticality and prevailing usage by attackers, using threat
intelligence and various flavors of analytics and machine learning (ML)

■ The ability to understand how a threat actor may pivot or move throughout an environment, and
which systems/techniques will be successful

■ Support for IT operations teams with information, prioritization guidance and

recommendations for remediation and configuring compensating controls

■ Management and administration of decentralized and distributed scanner instances and

architectures

■ The ability to deliver some level of, or plug into, other workflow management tools, such as
ticketing systems, to discover, act on and confirm the resolution of vulnerabilities

We use cookies
Gartner has to deliver the best
deliberately notpossible
called experience on our
this market website. To learn
“vulnerability more, visit ourWe
management.” Privacy Policy.
believe theBy
continuing to use this site, or closing this box, you consent to our use of cookies.
management of vulnerabilities has always involved additional people and processes, not just
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 2/23
12/30/2019 Gartner Reprint

technology. These people, processes and additional technologies are also represented by teams
outside the cybersecurity group in almost all cases, especially when it comes to critical
vulnerability management processes, such as patching.

There is also a large, existing market for professional and managed services that are looking to
help end-user organizations with VA and, sometimes, management. Outsourcers, managed
security service providers (MSSPs) and now some managed detection and response (MDR)
providers have options to deliver VA “as a service” for a long time. It remains a popular choice for
many organizations to have this capability delivered this way.

Even on the technology front, no single solution does full end-to-end management of
vulnerabilities. There is a mix of security compensating controls — intrusion detection and
prevention systems (IDPSs), web application firewalls (WAFs), network segmentation, privileged
access management/identity and access management (PAM/IAM), and security orchestration,
automation and response (SOAR). Other critical technologies include IT operations tooling for
patching, as well as items such as ticketing systems in this mix to perform the full life cycle of
modern vulnerability management. Simply put, vulnerability management is a process
underpinned by VA technology that triggers other processes, such as IT operations performing
patch management. This Market Guide focuses on the assessment and prioritization of this
function in a security program.

Market Description
VA can be delivered via an on-premises solution based on software, appliances, the cloud, hosted
solutions and/or a hybrid of these options. Moreover, it is widely available from almost all MSSPs,
consultants, outsourcers and is emerging to be delivered by some MDR providers.

VA technology typically supports security operations, network asset visibility and/or compliance
use cases. Security use cases include vulnerability and security configuration assessments
(SCAs) for enterprise risk identification, reduction and reporting against various compliance
standards.

Vulnerability prioritization technology (VPT) as a capability is a welcome evolution in an

organization’s ability to assess vulnerabilities. VPT solutions — formerly described by the term
“threat and vulnerability management” (TVM) — use the utility of VA telemetry, asset criticality
context and multiple, preintegrated threat intelligence sources, while augmenting this data via
advanced analytics. This combination enables organizations to have fundamentally different
views of their specific cyber risks. This can then save significant time, because acting on these
prioritized results will substantially reduce your organization’s attack surface with the least
amount of time with the most efficient use of staff resources. The leading disruptors are startups.
However, VPT concurrently exists as a feature in all the major vulnerability scanning vendor
offerings natively or as an add-on subscription.

Compliance use cases are still strong drivers and include meeting scanning requirements for
regulatory
We ortoother
use cookies delivercompliance regimes,
the best possible such
experience onas
ourthe Payment
website. Card
To learn Industry
more, visit our Data
PrivacySecurity
Policy. By
continuing
Standardto(PCI
use this site,or
DSS) or the
closing this box,Institute
National you consent to our use of and
of Standards cookies.
Technology (NIST). These
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 3/23
12/30/2019 Gartner Reprint

requirements can also include application assessment of the infrastructure in scope of the
compliance standard.

The VA market is characterized by small or midsize security vendors, compared with the large
network security and anti-malware vendors. Some are privately held, primarily with offerings
around VA, with vendors marketing VA as one component of a broader unified security
management portfolio of technologies or services, such as CrowdStrike, F-Secure and Microsoft,
etc. Large vendors (e.g., McAfee, IBM and Symantec) that offer VA often OEM this technology
from one of these pure-play providers. One interesting dynamic in recent years is how many new
startups have entered or are gaining mind share in this market around vulnerability visibility and
prioritization. Risk Based Security, Kenna Security, RiskSense, Skybox Security, NopSec and Balbix
are other examples of this development.

Market Direction
VA is a mature market, and VA is regularly defined as a standard component of information
security management and regulatory frameworks as a mandatory process. The adoption of
MSSPs, outsourcers and, recently, MDRs to execute VA for end-user organizations continues to be
popular and is experiencing growth.

Revenue in the VA market is concentrated among a few providers, with a large percentage going
to three vendors (Qualys, Rapid7 and Tenable). Based on Gartner inquiries, these three also
dominate vendor visibility on enterprise shortlists. However they have credible competition, and,
although they lead on overall size of client base, they do not substantially lead on feature
capabilities.

In addition to competing with other VA product and service vendors, VA vendors must compete
with consultants, MSSPs/MDR service providers, open-source scanning tools, and other security
and IT operations products that also provide scanning and configuration assessment capabilities.
Many vendors have more-basic versions of their VA products available for free, which many small
and midsize businesses (SMBs) use for often sporadic VA exercises.

Gartner does not recommend open-source vulnerability scanning tools for business use cases.
Open-source VA solutions might be a good place to start if your organization does not have a
vulnerability scanning tool. However, these versions will not be well suited for most organizations,
because the research, testing, signature and reporting capabilities are not as comprehensive as
those of enterprise VA solutions. Moreover, there is no guarantee that an open-source product will
be continuously maintained by a responsible entity.

VA against common platforms, such as Windows or Linux, is universally covered in the market,
with only minute differences between solutions in terms of scope and coverage from the leading
vendors. Differentiating solutions based on these criteria is seldom possible. Vendors can be
difficult to differentiate based on scanning accuracy and performance alone. Gartner sees
competition increasingly based on pricing, rather than features along with the addition of
We use cookies
scanning to deliver
other assetthe best possible
types, such as experience on our
the cloud, website. To
containers learn
and themore, visit our
Internet ofPrivacy
Things Policy.
(IoT).By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 4/23
12/30/2019 Gartner Reprint

Gaps in coverage — for example, for less-common technologies or third-party applications — will
persist, because they are difficult to convert into new sales and are not widely deployed by clients.
These capabilities also require the same R&D overhead as more-common technologies to
perform assessments. In addition, most organizations developing VA are not large vendors per se,
so scalability and enterprise management features are inconsistently developed and maintained.

With these new methods of delivering IT working in fundamentally different ways (such as the
cloud, DevOps and serverless computing), they pose new challenges to perform VA as these
computing models don’t necessarily allow reuse of existing approaches. As a result, support for
these new technologies is still evolving, and can rarely be covered by a single VA vendor solution.

Concurrently, some VA vendors are also expanding their portfolios into adjacent domains with
products, such as log management, MDR, security analytics, dynamic application security testing
(DAST), assessing containers and assessing cloud services. As a result, Gartner clients have
stated that some vendors have reduced investment and focus on their VA products, with fewer
updates and new features.

Market Analysis
The VA market is primarily driven by the use cases that are described in the sections that follow.

Vulnerability Assessment
The core, and still critical focus of most users, is still around general VA. This technology has
existed commercially for more than 25 years and, as such, has had a long time to develop and see
the maturity that it does today.

VA is also seen as a foundational security operational process mandated by a majority of end-

user organizations, as well as by standards, such as NIST and PCI, and many others, including
Gartner. Recent innovations are also moving VA from more of a compliance focus, to being a key
process in understanding and dealing with an organization’s attack surface. It provides unique
insights in managing it.

An ongoing evolution of VA is that a number of providers are also offering various levels of
sophistication of vulnerability prioritization. Although a generalization, the pure-play VPT solutions
offer better capabilities; however, this is also already delivered in various forms of sophistication
by many VA vendors today. Examples include Qualys Threat Protect, Tenable Predictive
Prioritization, Rapid7 Real Risk Prioritization and Balbix. Pure-play VPT vendors, however, offer
some additional capabilities over other VA solutions, in that they can take telemetry from multiple
VA/DAST solutions.

The technology is available in multiple form factors from physical or virtual appliances, software,
agent-based and other options for scanning cloud services (see “A Guide to Choosing a
Vulnerability Assessment Solution” and “Toolkit: Vulnerability Assessment RFP”).

Vendors
We in adjacent
use cookies markets,
to deliver the such experience
best possible as endpoint detection
on our and
website. To response
learn (EDR),
more, visit MDRPolicy.
our Privacy and security
By
information
continuing to useand
thisevent
site, ormanagement
closing this box, (SIEM) have
you consent to recently
our use of begun
cookies.offering VA. Arctic Wolf,
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 5/23
12/30/2019 Gartner Reprint

Crowdstrike, Kaspersky and Microsoft have recently entered the market.

Dynamic Application Security Testing

DAST tools analyze applications in their dynamic, running state during testing or operational
phases. They simulate attacks against an application (typically web-enabled applications and
services), analyze the application’s reactions and, thus, determine whether it is vulnerable. These
tools are considered to be a form of application security testing (see “Magic Quadrant for
Application Security Testing”). However, many organizations use DAST tools in conjunction with,
and occasionally as a replacement for, more traditional vulnerability assessment products. DAST
tools focus on discovering vulnerabilities or defects, such as those described by The Open Web
Application Security Project (OWASP; see Note 1) Top 10 within internally developed code. VA
tools are a superior means of discovering vulnerability issues in application infrastructures. The
combination of DAST and VA can be a highly effective means of identifying security issues in a
“full stack” (infrastructure and internal code) implementation of an application.

DAST tools are typically run in the context of the development effort, usually late in the process, in
conjunction with other types of testing. They’re also frequently found in the arsenal of tools used
by penetration testers. They’re also sometimes run in production environments; however, this
introduces a number of complications and risks. Similar to VA tools, DAST tools will typically
require the means to authenticate to an application to perform a full test, potentially exposing
credentials for production systems. DAST may also prove disruptive to application operation,
leading to degradation of performance or application instability. And, a successful test — one that
demonstrates the existence of a vulnerability — could lead to an inadvertent security incident. For
these reasons, production DAST should be approached with caution. Because they search for
different kinds of vulnerabilities, DAST should not be used as a replacement for VA.

DAST tools can be acquired from a number of sources. There is a variety of open-source options,
with popular choices, including the ZED Attack Proxy (ZAP), Nikto and Burp Suite. (Burp Suite
offers a community edition, focused on manual tests, and more robust commercial versions).
Given the common demand for DAST and VA, all of the major VA vendors also offer various
options for DAST. Finally, almost all major application security testing suites include a DAST
component. This market is covered in more depth in the “Magic Quadrant for Application Security
Testing.”

Security Configuration Assessment

SCA (see “Best Practices for Secure Policy Configuration Assessment”) has long been a feature
of VA tooling and provides the ability to remotely assess and verify not just missing vulnerabilities,
but configurations of systems in an environment. All of the VA vendors featured in this Market
Guide now offer this capability in some form, although some may require this capability to be
licensed separately, and the depth of the capability may vary. It is frequently used to fulfill
regulatory compliance, such as for PCI, or internal security policy compliance.

This
We useintegration ranges
cookies to deliver frompossible
the best basic experience
passwordonpolicy checks
our website. To to advanced,
learn more, visit application-level
our Privacy Policy. Bycontrol
analysis.toThis
continuing trend
use this site,led to thethis
or closing current VA/SCA
box, you consentdual
to ourfunctionality.
use of cookies.The use of dissolvable agents

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 6/23
12/30/2019 Gartner Reprint

deployed during an authenticated scan enabled some vendors to achieve even deeper scanning
across the tested systems. Many organizations still separate vulnerability scanning and
configuration weakness assessment.

Gartner research indicates it’s common for customers to purchase tools that perform unified
vulnerability and SCA scanning, then use them only for VA (sometimes without credentials) —
thus, missing an excellent opportunity to further mature their vulnerability management programs.
In addition, SCA can often be handled by other areas of IT in your organization.

There are mature and open standards for performing this key process in the form of guidelines,
such as Center for Internet Security ( CIS) benchmarks. They are an excellent start (strongly
recommended by Gartner) for proactively increasing your security posture (see “Secure by
Default: Using System Hardening to Prevent Threats”). An effective system hardening program can
drastically reduce the attack surface from threats, while achieving compliance objectives. A large
number of assessment tools support the CIS benchmarks (and other standards), and end users
are strongly advised to invest in this process, on top of just running vulnerability assessment
scanning. Used before asset deployment, this will reduce the chance of a breach, lower the
ongoing operational overhead of systems, while positively proving that hosts are still in a secure
state during their life in your environment.

Cloud Security Posture Assessment

Most security practitioners are now accustomed to running traditional assessment tools against
virtual machines (VMs) running in something like Amazon Web Services (AWS), Microsoft Azure
and Google Cloud. This is, of course, a good practice, in that it needs to happen and public
infrastructure as a service (IaaS) won’t save you from having to patch and maintain the workloads
running there.

However, the immutability of the cloud means that doing things such as only having to assess a
“gold image” is now not just possible, but is the most efficient way to perform VA for these types
of workloads. It reduces the overall assessment burden, because images can be rapidly recycled
in public cloud environments, versus the more static nature of traditional computing models.

Many VA solutions are still deficient in VA when it comes to the public cloud, particularly when it
involves assessing the management or control “plane” of your tenant in the public cloud. The
plethora of organizations that have been found to have things such as open-to-the-world Amazon
S3 buckets is a good example of this issue. IT security leaders need to urgently address this
coverage gap. Gartner client inquiry consistently shows that this class of issue is not being
addressed properly. The cloud management plane is accessible from anywhere on the internet
and literally controls your “virtual data center/computing in the cloud.” Hence the importance of
assessing it regularly.

APIs support the rapid and programmatic ability to assess state and instrument the cloud, making
APIs also ideal for vulnerability assessment functions. APIs can also be used for near-real-time
remediation
We ofdeliver
use cookies to issues. theThey aren’t just
best possible passive.
experience Security
on our website.leaders
To learn should pay
more, visit ourattention to how
Privacy Policy. By these
continuing
APIs cantobe
useapplied
this site,to
or security
closing this box,cases.
use you consent
Theytoneed
our use
to of cookies. them to assess how instances
leverage
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 7/23
12/30/2019 Gartner Reprint

of the cloud are configured and used for notification in near-real time of changes to posture, while
supporting remediation.

Specialist third-party offerings and solutions beyond a few mainstream service offerings are still
required in most cases. Leading CSPs (e.g., Amazon and Microsoft) are delivering some of these
features to provide users with better default security opens in their ecosystem. However, they are
generally for their own solutions, and do not cover other cloud services. Examples include
 Amazon Inspector and Microsoft Security Center for Azure. This capability is also assessed in
other Gartner research, for example, the “Magic Quadrant for Cloud Access Security Brokers” and
the “Innovation Insight for Cloud Security Posture Management,” where vendors that have these
capabilities are also discussed.

Operational Technology Assessment

Coverage for operational technology (OT) assets and technologies, such as supervisory control
and data acquisition (SCADA) or industrial control system (ICS) devices, is less mature and is not
present in all solutions. Many VA vendors claim SCADA or ICS support for their solutions. OT
requires especially careful consideration. When looked at in detail, in terms of scope (types of OT
supplier features that are assessable, etc.), and, importantly, ongoing dedicated support, end
users may be unimpressed.

Most organizations with OT environments will not actively run scans on those types of devices,
unless it is passive scanning. Nor would they get permission to do so from asset owners, for fear
of affecting the availability or critical systems/devices.

Different vendors take different approaches toward OT VA. Some vendors take an agent-based
approach, which records changes in the system and analyzes them. Other vendors passively
analyze the network packet, and some vendors take asset information correlation with a
vulnerability database.

Two key artifacts for this type of asset are their criticality (critical infrastructure like a power
station) and their fragility from using a VA tool on the point of view. The security industry is rife
with war stories of “that time I tried to do a VA scan on this subnet …” which led to serious
business-level issues of SCADA/ICS types of equipment affecting business. Things such as
stopping manufacturing production, and needing to travel to remote locations to reboot ICS
equipment, are common if extreme caution is not taken. There are a number of reasons for this.
Perhaps two stand out above the others:

■ ICS/SCADA equipment was never designed to face the blow torch of the internet and high-
performance networking

■ This equipment has a productive shelf life often measured in decades, not years — well past
support for the critical underlying IT technologies in them, such as an older base OS. As such,
they are often running older components across the OS, management, application, database
andcookies
We use other to
components.
deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 8/23
12/30/2019 Gartner Reprint

For example, Linux Kernel v.2.x (and older) and Windows XP hosts are common in these
scenarios. In short, they carry with them mountains of threat (and technical) debt, with no
traditional remedy such as patching. Both lead to the outcome that requires careful planning to be
in place when conducting VA. For example, the use of passive assessment only, virtual patching
with IDPS and WAF, and better monitoring of these devices are ways to help meet asset
identification and security monitoring needs. At the same time, these measures will significantly
reduce the risk of unplanned outages caused by VA processes going astray.

Vulnerability Prioritization Technology

Directly related to, and starting to be subsumed by the VA market is what Gartner has now defined
as VPT (previously labeled TVM). Today’s leading VPT tools come from startups and generally do
not run assessment activity themselves; instead, they agnostically leverage the (often multiple)
existing sources of telemetry that end users already have in place. Tools that create vulnerability
telemetry (e.g., traditional VA tools, dynamic web application testing and penetration testing data)
are supported by these VPT tools.

Their key benefit is what they do with this telemetry. They use primarily two other forms of data.
Threat intelligence on attacker activity and vulnerability use in malware, and internal asset
exposure and criticality to provide fundamentally better view of real risk for an organization to
understand cyber risk and prevent breaches.

This helps significantly in the prioritization work that doesn’t fall on security practitioners, but, in
most cases, on IT operations teams, which have to do the last-mile legwork patching and dealing
with the tail end of the vulnerability management process.

For security teams, the benefit is that they are presented with what today is a generally smaller list
of higher-risk issues. These can then directly map into tools that security teams often have
already deployed and have been managing for more than a decade, such as IDPSs and/or WAF
systems, to help with configuring these compensating controls.

Breach and Attack Simulation

Like the burgeoning VPT market, breach and attack simulation (BAS) vendors have also emerged
and have been evolving in recent years. These vendors have technology that is deployed at
various parts of the environment and use agents and/or VMs to actively test the environment for
issues, simulating common methods used by attackers (see “Utilizing Breach and Attack
Simulation Tools to Test and Improve Security”). These tools are being positioned more as
automated penetration testing tools or as security controls assessment and assurance tools, not
as a vulnerability assessment solution. In fact however, there is overlap with VA in some
functionality, as BAS tools assess the environment for vulnerabilities without needing to use a VA
tool or import telemetry from one. They focus, however, on a smaller subset of vulnerabilities that
can be leveraged during a breach (aka breach simulation). Importantly, they do not focus on
finding all vulnerabilities, but instead on those that can be reliably exploited.
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 9/23
12/30/2019 Gartner Reprint

In the context of this research, BAS helps end users by providing an “attackers eye view” of your
environment, from the inside out, including how your existing suite of compensating controls can
be bypassed in your environment. A key capability for any risk-based vulnerability management
(RBVM) program is in having more options to pursue in your operations that are centered on more
than patching alone.

Security leaders can also use simulation breaches as input into follow-up prioritization activities
involving “what actions to take next.” These actions range from configuring/updating
compensating controls, such as IDPS and WAF, to network segmentation and, of course, patching.
They can also highlight the configuration issues of these controls that allow dangerous activities
to occur. Operationally speaking, this attack chain modeling maps quite effectively onto the Mitre
 ATT&CK framework with a number of vendors now natively reporting using the ATT&CK
framework in their solutions.

Penetration Testing
VA is also delivered by a large number of professional services firms from consultants, product
vendors, MSSPs and now some MDRs as part of penetration testing services. The penetration-
testing industry is already well-established and is being heavily contested by large, as well as
smaller, boutique types of providers worldwide. There is no direct correlation between the size of
the organization and the quality of its work when it comes to this service. Smaller, pure-play
penetration-testing firms continue to be competitive (see “Using Penetration Testing and Red
Teams to Assess and Improve Security” and “How to Select a Penetration Testing Provider”).

Although separate from VA, penetration testing plays an important role in the prioritization and
assessment of vulnerabilities from Gartner’s RBVM methodology. These services are testing your
environment, with real-world skills and knowledge of the prevailing threat landscape. Security
leaders need to take these recommendations and apply it directly in your security programs to
address their prioritized findings. In these recommendations, Gartner consistently sees reasons
why an organization later appears in the news for the wrong reasons (i.e., because of a breach).

In addition, some VPT tools are able to process data from these reports that come from
penetration-testing engagements and use that to aid in prioritization functions.

Bug Bounties and Crowdsourced Security Testing

Bug bounties have existed for some time, and, as the name implies, they are rewards or payments
provided to a security researcher in response to the discovery and responsible disclosure of a
defect (aka vulnerability) in a system. For the most part, in the past, such bounties were often
provided on an ad hoc basis, at the discretion of the firm receiving the report. More recently,
commercial firms have introduced formal bug bounty and crowdsourced vulnerability discovery
programs, which rely on large groups (sometimes thousands) of security researchers to drive
programs. Initially viewed with some skepticism, the programs have gained growing acceptance,
as large technology firms and governmental organizations have embraced the programs.
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 10/23
12/30/2019 Gartner Reprint

Vulnerabilities discovered during testing can include application issues, as well as problems
within infrastructure components, so they span VA and application security domains. Vendors
have expanded their offerings to include vulnerability disclosure management programs,
certification testing (such as for PCI DSS), and variations on more-traditional penetration tests.
Vulnerability disclosure management programs are designed to address reports of vulnerabilities
received from independent researchers, outside the context of a formal crowdsourced program.
In this case, vendors rely on the crowd to evaluate and validate the disclosure, then coordinate the
payment of a bounty (if offered) and pass along the report to the appropriate team in the
organization for remediation or mitigation.

Bounty programs can be for a limited duration, or be ongoing. Although approaches vary, vendors
typically charge buyers some type of program management fee. It varies based on the specific
services being offered, and the volume of vulnerabilities disclosed or the size of the bounty
payout. Organizations will also be required to fund bounty payouts via the bug bounty vendor
being the broker.

When evaluating vendors, consider the type of service desired, because offerings can vary. Rules
of engagement — describing acceptable behavior on the part of researchers, the specific parts of
a system to be tested (and, which components are off limits), and types of findings for which a
vulnerability will be paid (for example, a firm may already be familiar with certain vulnerabilities in
applications, or may not be interested in common or easily discovered vulnerabilities — this must
be communicated to researchers in advance).

Bounty payments can vary considerably. Easily found problems might merit only some form of
“swag,” such as T-shirts. More complex vulnerabilities will fetch hundreds or thousands of dollars.
Finally, complex or rare vulnerabilities might merit a payout of hundreds of thousands of dollars;
however, these are — by their nature — quite rare, and the exception. In implementing a program,
organizations must consider how findings will be integrated into their existing workflows for
application and infrastructure vulnerability remediation and mitigation (this is covered in more
detail in the “Market Guide for Application Crowdtesting Services”).

VA Methods
Historically, most VA solutions have focused on network scanning, primarily due to the
complexities and overheads involved with deploying and managing large agent populations (see
“A Guidance Framework for Developing and Implementing Vulnerability Management”). However,
virtual hosts are more challenging to assess, because they may be running for only short periods
of time and can be provisioned on-demand and autoscaled up and down. For virtual systems,
agents can be included in the base image and then enabled in each machine at startup. IaaS
environments also pose challenges similar to virtual environments. However, leading IaaS
providers have what is generally regarded as excellent externally facing APIs that facilitate
excellent enumeration and policy configuration details, as well as support the ability to remediate
and make changes via these same APIs
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 11/23
12/30/2019 Gartner Reprint

Remote and mobile users cannot be reliably assessed by remote scanning. They benefit from an
agent-based approach, often delivered from the cloud, when the assessment is conducted locally
on the host, and the results are sent to the management instance.

Another significant factor contributing to increased agent usage in VA is the paradigm shift from
scheduled VA to continuous monitoring (see Table 1).

Table 1: VA Scanner Deployment Methods

Effective Scanner Deployment Model

AP
Types of On- Cloud- Bas
Agent-
Organizational Premises Delivered Passive Scan
Based
IT and Related Network Network Scanning (Deliv
Scanning
Assets Scanning Scanning Mult
Way

Assets connected ✔ ✔ ✔ ✔
to the
organization’s
network

Assets operating ✔ ✔* ✔* ✔*
remotely or while
mobile

Internet-facing ✔ ✔ ✔ ✔
assets

Cloud service
tenant
configuration: SaaS
and PaaS

Cloud computing: ✔ ✔ ✔
IaaS

Mobile devices ✔ ✔

Bring your own ✔ ✔* ✔*

device (BYOD)
assets (not owned
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
by organization)
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 12/23
12/30/2019 Gartner Reprint

Effective Scanner Deployment Model

AP
Types of On- Cloud- Bas
Agent-
Organizational Premises Delivered Passive Scan
Based
IT and Related Network Network Scanning (Deliv
Scanning
Assets Scanning Scanning Mult
Way

Virtualized assets ✔ ✔ ✔ ✔

Network assets ✔ ✔
(e.g., firewalls and
routers)

Operational ✔** ✔** ✔

technologies (e.g.,
SCADA and ICS)

* Although these devices are on the network or stay connected to the virtual private network (VPN), assessme
possible via the other standard means — for example, from network and passive scanning; however, credentia
scanning is often not an option.** Although the network scanning can be performed on OT devices, it’s seldom
recommended because of performance and availability impact that may result.*** API refers to using the API
cloud service itself. This is often delivered from the cloud, but this does not preclude scanning from on-premi
appliances or software either.

Source: Gartner (November 2019)

Analysis of Vulnerability Risk Impact and Remediation Prioritization

VA tools capture large amounts of data, regularly exceeding the ability of IT operations to
remediate the sheer volume of found issues. Many VA solutions assign a critical severity score on
the vulnerabilities based on the Common Vulnerability Scoring System ( CVSS) scores. These are
calculated based on a formula that depends on several metrics that approximate ease and impact
of exploit. However, these metrics fail at scale when the objective is, for example, to identify the
100 vulnerabilities out of the thousands of vulnerabilities your environment may have that are the
most critical to address — not just their initial severity rating.

Importantly, this is not the problem with these standards such as CVE/CVSS/CWE per se. As an
industry, we are in a far better place for having the well-conceived  CVE,  CVSS and  CWE
standards, and they perform their functions admirably. These critical underpinning standards
enable these newer analytics methods to flourish.

Not
We useall vulnerabilities
cookies are
to deliver the created
best possibleequally. Exploitability,
experience prevalence
on our website. in malware
To learn more, and exploit
visit our Privacy Policy. Bykits,
continuing to use this
asset context, and site, or closing
active this box, you
exploitation byconsent
threat to our use
actors of critical
are cookies. qualifiers in assessing cyber risk
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 13/23
12/30/2019 Gartner Reprint

(see “Implement a Risk-Based Approach to Vulnerability Management”). Many VA vendors have

added capabilities to support improved vulnerability prioritization, thus further enabling Gartner’s
RBVM methodology.

An emerging market of VPT consolidates the output of different security testing technologies,
such as VA and DAST, to permit a more pragmatic and holistic assessment model of IT assets
risk. These are designed to support organizations’ RBVM initiatives by providing formalized
workflow, as well as reporting and collaboration capabilities across multiple IT functions. They
usually do not execute VAs themselves, but consolidate and normalize output from multiple
vulnerability and application security solutions, as well as from penetration-testing engagements.

Methods are applied that analyze and prioritize vulnerabilities by using threat intelligence,
organizational asset context, and risk modeling approaches such as attack path analysis. This is
also an area in which advanced analytics methods are also being used, such as ML. This permits
more granular and intelligent remediation strategies than the more simplistic severity approaches,
especially at scale and when remediating with constrained resources.

ML is also being used by some providers to help predict the likelihood that a vulnerability will be
exploited “in the wild.” As this continues to improve it will prove to be a real boon to risk
management, as well as security operations, as it allows organizations to prioritize and focus on
higher-risk scenarios.

Apart from the stand-alone VPT market (mostly startups), the major VA vendors are developing
the capabilities of VPT products in their roadmap execution. The capabilities offered by the VA
vendors may not be as comprehensive as a stand-alone vendor at this point. However, this can be
a good starting point for clients using a homogeneous environment of your VA vendor for security
testing. Also, buying an add-on product from the same vendor helps vendor consolidation, and
sometimes cost, with less effort placed on new training and tool deployment. This is a key area of
innovation that end users are strongly advised to seek out in their procurement cycles and
prioritize in the future.

Risk ratings are provided to each vulnerability based on a proprietary, assessment data
processing engine. Some specific vendor tools specialize in remediation prioritization and
analysis that can ingest data generated by various VA tools, and use proprietary algorithms to
provide risk ratings. These tools automate some of the manual tasks in the remediation process
by delivering automated workflow capabilities via dashboards and integration with IT operations
management (ITOM) tools.

This capability is becoming a key differentiator for end users to mandatorily consider from their
VA vendors and a point of differentiation for startups delivering a point solution.

Representative Vendors
The vendors listed in this Market Guide do not imply an exhaustive list. This section is intended to
provide
We moretounderstanding
use cookies of the market
deliver the best possible and
experience onits
ourofferings
website. To(see
learnNote
more,3).
visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 14/23
12/30/2019 Gartner Reprint

The vendors listed (see Table 2 through Table 5) in this Market Guide provide mature capabilities
for VA of common network-based devices, as well as features to allow the analysis, reporting and
management of vulnerabilities and remediation.

Table 2: Breach and Attack Simulation Vendors

Vendor Product

AttackIQ  AttackIQ

Cymulate  Cymulate

Core Security  Core Impact

FireEye  Verodin Security Instrumentation Platform

Pcysys  Pcysys

Picus Security  Breach and Attack Simulation

SafeBreach  Breach and Attack Simulation

XM Cyber  HaXM

Source: Gartner (November 2019)

Table 3: OT Vulnerability Assessment Vendors

Vendor Product

Claroty  Security Posture Assessment

CyberX  Risk and Vulnerability Management

Forescout Technologies (SecurityMatters)  Forescout Operational Technology

Indegy  Industrial Cyber Security Suite

Nozomi Networks  Guardian

We use cookies to deliver the best possible experience on our website.

Radiflow To learn
 iSEC: more, visitAssessment
ICS Security our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 15/23
12/30/2019 Gartner Reprint

Source: Gartner (November 2019)

Table 4: Vulnerability Assessment Vendors

Vendor Product

Alert Logic  Network Vulnerability Management

Balbix  Risk Based Vulnerability Management

Beyond Security  Vulnerability Assessment and Management

BeyondTrust  Vulnerability Management

Digital Defense  Frontline Vulnerability Manager

F-Secure  Radar

Greenbone Networks  Vulnerability Management

Outpost24  Network Security Assessment

Positive Technologies  Vulnerability Management

Qualys  Vulnerability Management

Rapid7  InsightVM

Tenable  Tenable.io  Tenable Security Center

Tripwire  IP360

Source: Gartner (November 2019)

Table 5: Vulnerability Prioritization Technology Vendors

Vendor Product

Brinqa  Vulnerability Risk Service

We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
Conventus  NorthStar Navigator
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 16/23
12/30/2019 Gartner Reprint

Vendor Product

Kenna Security  Kenna Security Platform

NetSPI  Resolve

NopSec  Unified VRM

RiskSense  Risk-Based Vulnerability Management

Resolver  Risk Vision

Skybox  Vulnerability Control

Vulcan  Vulcan Cyber

ZeroNorth  Risk Visibility and Assurance

Source: Gartner (November 2019)

Market Recommendations
Gartner considers VM a foundational component of any cybersecurity program. It is key to
understanding cyber risks and how they intersect with digital businesses.

How VA is included in the vulnerability management process often varies considerably, based on
the size and maturity of the organization. Some organizations deploy VA in a stand-alone
capacity, providing audit or assessment capabilities to assess risks or to measure compliance.
Others use it in a more operational capacity to assist IT operations in prioritizing and verifying that
things such as patches have been successfully applied. Others integrate VA into their DevSecOps
processes, so that VA is applied to applications continuously as they are developed and deployed.
Many organizations do both. However, the buying center is often the security organization/audit
organization. IT operations participates in the configuration assessment.

Enterprises with more-mature security programs augment VA and SCAs with more-advanced
penetration and custom application testing. This is aimed at validating where they can positively
prove they are vulnerable to previously unidentified attacks.

Vulnerability Assessment as a Feature or a Product

VA capabilities are offered in the stand-alone VA market (Rapid7, Qualys, Tenable, etc.), as well as
adjacent markets in which VA capabilities are being used to supplement other threat detection
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
and response capabilities. For example, EDR and SIEM vendors have added VA as a part of their
continuing to use this site, or closing this box, you consent to our use of cookies.
broader offerings to assist with threat detection, investigation and response use cases and to
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 17/23
12/30/2019 Gartner Reprint

offer another level of visibility into the systems under the scope. Although the VA feature may
appear to be the same as the capability offered by VA vendors, organizations should exercise due
diligence to understand different use cases offered by the products.

VA as a feature in EDR products doesn’t have network scanners to perform network-based

scanning or point-in-time assessment and also doesn’t cover the systems or devices where an
agent can’t be installed. The VA features present in EDR products are focused on providing
immediate mitigation or remediation using compensatory controls to reduce the attack surface —
for example, Symantec SEP and Trend Micro Apex One — whereas VA products focus on holistic
vulnerability management and reporting along with risk-based prioritization.

Organizations looking for a fully featured VA solution for VM should consider a VA product for
broader coverage of assets, vulnerabilities and threats.

Risk-Based Vulnerability Management

Gartner has called out the critical need to assess assets for configuration issues and
vulnerabilities, and to be able to prioritize what you do with that assessment, based on the risk to
your organization. This takes into account the prevailing threat landscape (see “Implement a Risk
Based Approach to Vulnerability Management” and “It’s Time to Align Your Vulnerability
Management Priorities With the Biggest Threats”) and other elements, such as asset criticality.
The evolution of this guidance is Gartner’s RBVM methodology, highlighted above.

Figure 1 describes this iterative process, which is critical to achieving better outcomes for any
vulnerability management program. The key reasoning behind this is that vulnerabilities, and their
exploitation by attackers of all skills levels and motivations, are driving the threat landscape as we
see it. In addition, most-malicious activity is also coming from already-known vulnerabilities and
not zero-day vulnerabilities.

Figure 1. Gartner’s Risk-Based Vulnerability Management Methodology

We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 18/23
12/30/2019 Gartner Reprint

Security and risk management (SRM) leaders selecting solutions or services should:

■ Evaluate the scope of device and third-party OS and application coverage, especially for those
that are deployed and are not considered mainstream.

Providers should be able to align to your organization’s computing architecture. They should
provide wide (numbers of classes of assets, such as endpoints, servers, storage, networking,
mobile and security) and deep (supporting compliance frameworks, as well as thorough
assessment capabilities for assets under coverage) support for your IT assets. Although
obtaining 100% scanning coverage is ideal, from a practical standpoint, covering as many
technologies as possible is an acceptable pragmatic outcome. In-depth assessments of
databases and applications, such as ERP systems (e.g., SAP or Oracle), are not widely
supported in traditional VA solutions.

Some providers also address static analysis security testing (SAST) and DAST functionality
around source code and/or web application assessments. However, this is a submarket
covered elsewhere at Gartner in the “Magic Quadrant for Application Security Testing.”

■ Appraise the methods that a VA solution provides to aid in the assessment of the impact,
We use cookies to
criticality deliver
that thenthehelps
best possible experience
guide with on our website. of
the prioritization To vulnerabilities.
learn more, visit ourThis
Privacy Policy. By
is about
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 19/23
12/30/2019 Gartner Reprint

identifying and quantifying your organization’s intersection with the prevailing threat
landscape.

Practical prioritization guidance is a key outcome to make VA tools pragmatically in raising

your organization’s security posture. VA tools can produce large reports, which continue to be
difficult to use effectively. Hence, IT security managers should add additional capabilities to VA
tools that can decrease manual effort, and provide analysis and recommendations on the
vulnerabilities to focus on first. The vulnerabilities that you have that are being exploited in the
wild are a key example of these types of recommendations. If the capabilities provided by the
VA solutions are insufficient, evaluate VPT solutions or supported third-party integration tools.

■ Evaluate the assessment deployment options.

As the shift proceeds from regularly scheduled scans to continuous monitoring and more agile
and decentralized deployments, the available methods to scan for vulnerabilities will play an
increasing role. This includes the ability to use an agent on remote assets for mobile and off-
site users, and for transient, virtualized architectures and DevOps practices, as well as the
ability to assess system images at rest or in containers. There’s a steady move to deliver this
from the cloud. Although on-premises options are still available, the move in this direction is
undeniable in terms of the vendors and clients moving to having more “cloud power”
somewhere in their use.

Leading vendors are moving to prepare their cloud platforms to deliver VA. In the coming years,
some on-premises editions will start to be marked end of life or to have no new features, other
than scanning/assessment security content support delivered.

■ Assess the vendor’s current support, and future plans and roadmap, for supporting emerging
technologies.

Organizations with large or growing cloud, virtualization and DevOps usage must select a VA
solution(s) with these computing demographics in mind, and must consider a vendor’s current
and future commitment to these technologies. In many cases, gaps will be closed only by
collaboration with technology partners and third-party integrations and additional solutions,
rather than native support in the VA solutions. Integrations with platform management
systems, such as enterprise mobility management (EMM) suites, hypervisors and cloud
security platforms, are especially important, providing extended visibility and some vulnerability
assessment capabilities.

■ Evaluate available vendor portfolio synergies.

Some of the vendors in this Market Guide also offer their VA solutions as one component in a
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
broader integrated portfolio. Depending on your requirements, these combined technologies
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 20/23
12/30/2019 Gartner Reprint

can provide a sum-greater-than-the-parts security posture/solution, and also prove cost-

effective, due to bundle licensing. However, if they’re not seeking these from the outset, then
potential buyers of VA solutions should not be tempted by the implied benefits.

In addition, the following capabilities are critical, especially in larger enterprises:

■ Scope, efficacy and speed of content updates

■ Capability to centrally manage, administrate and schedule scanners and scans

■ Role-based access control (RBAC), which supports on-premises identity management, as well
as standards such as Security Assertion Markup Language (SAML), which support on-premises
and cloud-based identity and access management (IAM) solutions

■ Integrated support for managing and tracking vulnerability data, such as vulnerability
management workflow and ticket management related to vulnerability remediation

■ Support for new types of infrastructure-like cloud services and OT

■ Integration with enterprise workflow and security management solutions, such as configuration
management databases (CMDBs), enterprise directories, and identity and IAM solutions

■ Flexible architecture options, such as virtualized deployment and cloud-based scanning

■ The ability to automate scanning and alerting by supporting scheduling and workflow-based
capabilities.

■ Exceptions management for multiple phases, scanning, creating tickets, reporting, etc.

■ Support for presenting APIs from the VA tool, so that other tools — e.g., security information
and event management (SIEM); IPS; WAF; and security operations, automation and reporting
(SOAR) tools — can instrument and take information from them for integration into security
operations

The Problem of Intersectionality

Gartner continues to field inquiries on vulnerability management that highlight the
“intersectionality” problem. This refers to dealing with the intersection of multiple items that
overlap and connect with each other in different ways in terms of people, process and technology.
For example, does an IT SRM leader focus on vulnerabilities related to:

■ Asset criticality to the business

■ Compliance mandates driven by security or other parts of the organization

■ The cultural importance of a specific device (e.g., the CEO’s laptop) versus others
We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this
■ Vulnerability site, or closing
severity scorethis box, you consent to our use of cookies.
https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 21/23
12/30/2019 Gartner Reprint

■ Class of vulnerability — for example, OWASP Top Ten or ones that are remotely exploitable

■ No credentials required for exploitation

■ News/media cycle driving awareness of an issue at a particular point in time

■ IT operations processes that support different types/classes of devices — for example, focus
mainly on the DevOps and web application versus servers that power these applications

■ Attackers’ use of vulnerabilities

■ General security operations configuration changes of compensating controls for things such as
IPS and WAF to help with virtual patching

■ Security doing the scanning, but almost never owning the asset — unlikely to have the control to
apply patching to systems

These factors are, in their own right, valid considerations and perhaps compelling priorities for
your security operations. However, vulnerability management, from a security perspective is
primarily about stopping bad things from happening (e.g., ransomware, breaches, data loss and
reputational damage) and compliance.

Evidence
This research is based on a combination of briefings from the vendors mentioned in the text, as
well as client inquiries and  quantitative research on the strong correlation between malware and
vulnerabilities.

Note 1
OWASP Top Ten
Over the years, the OWASP Top Ten has evolved, and, with that, its definition has changed on what
types of vulnerabilities do and do not classify as the Top Ten. This research uses this  definition.

Note 2
Representative Vendor Selection
The vendors listed are representative of the VA market, as well as other directly related products.
Unlike a Magic Quadrant, a Market Guide methodology does not use metrics such as revenue,
market share and visibility for inclusion. Instead, we hope to provide a credible list of vendors that
would meet the needs of the IT security leaders that this research is targeting as readers of this
research.

We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 22/23
12/30/2019 Gartner Reprint

© 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc.
and its affiliates. This publication may not be reproduced or distributed in any form without Gartner's prior
written permission. It consists of the opinions of Gartner's research organization, which should not be
construed as statements of fact. While the information contained in this publication has been obtained from
sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy
of such information. Although Gartner research may address legal and financial issues, Gartner does not
provide legal or investment advice and its research should not be construed or used as such. Your access and
use of this publication are governed by Gartner’s Usage Policy. Gartner prides itself on its reputation for
independence and objectivity. Its research is produced independently by its research organization without input
or influence from any third party. For further information, see "Guiding Principles on Independence and
Objectivity."

About Careers Newsroom Policies Site Index IT Glossary Gartner Blog Network Contact Send
Feedback

© 2018 Gartner, Inc. and/or its Affiliates. All Rights Reserved.

We use cookies to deliver the best possible experience on our website. To learn more, visit our Privacy Policy. By
continuing to use this site, or closing this box, you consent to our use of cookies.

https://www.gartner.com/doc/reprints?id=1-1XWO60AS&ct=191211&st=sb 23/23