Review Criteria for API-powered Digital Business Platforms

Updated 04-28-2020

Overview
A Vendor Experience
B Architecture
C API Gateway
D API Analytics and Monitoring
E API Security
F Developer Portal
G Microservices
H Governance and SDLC
I Training and Support

Section A - Vendor Experience

Requirement Details Response
A1 Please describe your company's strategy APIs are a critical part of our company's ability
A2 around
When didAPIyour
management.
API management product first to prosper
We're in an increasingly
interested digital society.
in the track record of your It is
A3 become generally available (GA)? company in API management.
Is your API Management platform a leader in We're interested in understanding how 3rd-party
A4 the
Are Gartner Magic Global
there Fortune Quadrant?
500 companies using industry
In additionanalyst
to theexperts
productmeasure
features,your platform
we would like
A5 your API management product?
Provide examples of companies who are to
Companies that rely on your platform to you
understand the real world experience
A6 running
Can youmission
providecritical APIsofon
examples your
your API
thought significantly
APIs, social,influence
and mobiletheir
arecustomer
fast moving topics.
A7 leadership in the API space? We would like to
What kind of experience do you have running a While many vendors are nowwork with a vendor
offeringwho leads
cloud-
A8 managed
Do you provide use cases and testimonials for We would like to know more about your real that
cloud solution at scale for your cloud based versions of their products, it is critical
A9 your
How existing customers?
do you onboard and partner with world experience.
A10 customers for success?
Can you provide some statistics for your
largest customers in terms of volume and
Section B - Architecture
Requirement Details Response
B1 Does your product support SaaS, customer- Depending on present and future project
B2 managed, and hybrid deployments?
Can your customer-managed offering requirements, we may need different
B3 (sometimes called architecture
Does the platform "Private Cloud") operate
support multi- The ability to run a multi-tenant environment can
B4 tenancy?
Can multiple teams work independently with be
An important
enterprisewhenSDLCdealing withdevelopment
(software multiple lineslife
of
B5 runtime
How does isolation?
the platform support a multi-region, cycle) can be a complicated process
Geographical redundancy is important both forwith many
B6 multi-data
Explain how center
your deployment to ensure
solution supports the
flexible high availability
Unexpected andinalso
bursts APIfor latency
Traffic areand
bound to
B7 scaling and describe what is needed
Does your solution provide a centralized to happen. We need to know that our
Ease of management is one of the day-to-daycapacity can
B8 interface
Does the for managing
solution multiple
support data center
zero downtime considerations in choosing
For critical applications andaaplatform such as
geographically
B9 patching and updates?
Does the solution have the ability to do dispersed
For latency sensitive applications, intelligentbe
user base, how can the platform
B10 intelligent traffic routing
Does the solution supportto agive users
hybrid the
deployment routing to the
For system to nearest
system point of presence
calls within a singlecan be
data
B11 model? This is one in which traffic
Does the solution provide the ability to start out center, it can be useful to eliminate
Requirements and philosophies will change the latency
as a SaaS (Public Cloud) version and later during the lifetime of an API or for evaluation
Section C - API Gateway
Requirement Details Response
C1 Does the product support OpenAPI (formerly
C2 known
Does the as product
Swagger) to design
facilitate rapid APIs and
prototyping of
C3 mock APIs?
Does the product help create uniform,
C4 consistent,
Is it possiblewell-formed
for a company APIs,toevenenforceif the In some cases, we have security requirements
C5 behavior for all APIs exposed
How are existing SOAP services added? by the system? How that must be verified.
is complex How does the product
data transformation handled?
C6 Can deployments of assets be automated for How hard is it to incorporate into existing
C7 the
Candevelopment
your platformlifecycle?
reference existing assets development standard tools? What development
C8 such as encryption libraries,
How does your product support threat schema validation
C9 detection
Please describeby detecting fraudulent
your product's data to protect
ability
C10 from traffic spikes.
Please describe the product's ability to manage
C11 API
Can consumption through quotas.
quotas be synchronized across Can quotas
multi-
C12 region deployments?
Does the platform support publishing existing
C13 services
Does theinproductvarioussupport
formatsAPI - for example and For example getCustomerInfo API would require
virtualization
C14 mashups?
Please describe your ability to enhance API multiple
Many times, back-end calls to be
configuration canmade
become to multiple
C15 functionality through both configuration
Please describe any out of the box functions and prohibitively complex to accomplish the same
C16 for traffic throttling, caching, quotas, payload
Are standard transformations included? (XML In order to reuse existing systems or to talk with
C17 to JSON,
Does JSON support
the proxy to XML,compression?
SOAP to REST, legacy
Can messagessystems,be it is important
both sent and that the platform
received by the
C18 Does the proxy support HTTP & HTTPS? proxy
How can we configure the platform towill
in a compressed format? This save
secure
C19 Are streaming connections supported? the
For communications
long running transactionsinto the system,
or largeand out of
payloads,
C20 can the proxy
Please describe the debugging tools built into Distributed systems are more complex than stream traffic?
C21 the
Canplatform.
the debugging tool show a "before" and client server systems.
This functionality can be What toolsduring
crucial does forensics
the
C22 "after" of each policy
How is versioning supported? during replay? Also can or during pre-production testing
To minimize impact to developers and users, of a policy.
C23 Are all policies and system configurations versioning
A standardneeds formattolike beXML
flexible. Versioning
allows for easy
C24 stored using standards
Does the product support caching? based formats? Can transformation
Caching at the API gateway levelinminimizes
and manipulation a variety hits
of
C25 against the back end
In addition to an expiration, can the cache be While it is important to be able to set a cache to systems.
C26 manipulated
Do you support programmatically?
a multi-level cache model ? For expire In-memory at a certain
cache is point
veryinfast,
time,butit is
hasalso
limitations
C27 example, is the in-memory
Does the product support caching based on cache able to spill of
To optimize caching, the platform shouldcaching
size. The ability to perform multi-level be
C28 payload
Does theinformation
proxy haveand rateHTTP
limiting,headers?
quotas,Isand this able
Access to cache
to databased
and load on many types ofsystems
on back-end
C29 spike
Can API arrests?
mediation behavior change must
In thebe configurable
dynamic world of and controllable.
APIs and mobile The
C30 dynamically based upon factors
Does the proxy support dynamic routing such as user applications
In the dynamic world of APIs and mobileplatform
it is often necessary for the
C31 (orchestration—or intelligent
How effectively and to what extent can the routing to a applications
In the interest it is
of often necessary
minimizing for the platform
professional
C32 core functionality of the platform
Does the platform support extensions using be customized services
If customers want to build extensions to can
and increasing time to market, the
C33 common languages like Java,
Can the platform host and run unmodified Python, or platform
With the capabilities, is it possible
increasing popularity using it
of Node.js,
C34 Node.js
Does theapplications
platform have in order
wizards to implement
to generate would
In order befor
useful to havetothis
API teams capability
be agile, and built into
rapidly
C35 APIs from OpenAPI (formerly Swagger),
Does your product provide flexibility to extend SOAP configure/build and deploy APIs, it's important to
C36 the functionality and implement
How does the product support API Lifecycle attribute
C37 governance?
Can your product publish APIs for external and
C38 internal
How do consumers?
you manage How are these
API visibility and managed
restrict
C39 access
Does the platform support the ability for an in
to consumers? Is this configuration API
C40 to callyour
Does another managed
product support API a endpoint
common out errorof
handling pattern?
Section D - API Analytics and Monitoring
 Requirement Details Response
D1 Please describe the out-of-the-box analytics The reports in this list should require no
D2 reports
Does the provided
UI allowbyforthe tool.
drill down on each of the configuration. Normally
Drill down analytics allowsthesefor will
quickinclude
triagebasic
of the
D3 charts? health of an API
Does the product provide easy-to-use custom No vendor can provide every report we program and assists in need
rapid out
D4 reporting
Are there capabilities over multiple
maps for detailing dimensions
geo-location of of the decisions
Many box. The platform
in an APIshouldprogramhave area based
wizard
D5 API calls? upon the location
Are the analytics collected asynchronously (so The single greatest factor in the user of users. The platform should
D6 as not to impede runtime traffic)? satisfaction of an app is
Do the analytics data, once collected, provide We are not interested in creating a data silo. its response time. Are
D7 an API for easy access and export? The collected analytics
Can the solution be used to provide business Beyond operational level and developer level data must be accessible
D8 level
Whatvisibility?
level of operational visibility can the metrics, how does
Beyond simple the platform
graphs of traffic,provide visibility
what visibility
D9 solution provide based on API traffic
What tools are available out of the box to do flowing would an ops team gain from using
The tool needs to both provide visibility into the
D10 various
Does the kinds of trend
product allowanalysis
customers andto inspection
create trends (to prepare
Do reports need tofor becapacity
configured bursts
beforeor product
D11 reports on-demand? launching the system?
What metrics and dimensions are supported by The tool must support a variety of analytics use Can reports be
D12 the tool?provide service performance
Do you cases without requiring additional programming
D13 monitoring, reporting, and analysis?
Is payload data captured? Can this data be For example, imagine an API call allows the
D14 used
What forarereporting?
the exception management reporting user to search for a list of products by
D15 capabilities?
Does your product provide end-to-end visibility A transaction tracing identifier is passed
D16 by
Doessupporting
your productthe creation
provide or injection of
application a
usage between systems to correlate individual system
D17 visibility and trending performance
Does your solution support billing based on a statistics?
D18 period
Does the of time and/or
solution aggregate
provide transactions
performance
D19 management data with counters
What level of reporting is available perto the API
D20 Consumer? (call latency, SLA compliance,
Does your product provide the ability to easily
D21 integrate
Are all of analytic
your billingdata with
and other systems,
developer for
usage data
D22 available via an API to allow an easy
Does your product include the ability to detect
anomalous behavior in API traffic, and to alert
Section E - API Security
Requirement Details Response
E1 How is single-sign on supported for
E2 Administrators
How is single-signandonOperators
supported of for
your product?
visitors to
E3 the developer portal?
How is single-sign on supported for Users of
E4 the
WhatAPIs
aremanaged by your
the standard product?
industry security
E5 certifications available for your
What are the product data security product?
controls for
E6 customer data? I.E.
Does the product data processing,
support open standards data such
E7 as OpenID Connect to delegate authentication
Explain the mechanisms you use to support
E8 API security
Please (e.g.the
describe tokens,
supportencryption, policyfor
in the product OAuth is one of the most widely used forms of
E9 OAuth.
Does the product support connecting to Active authentication
Okta, Ping, andforActive
consumer or partner
Directory are thefacing
most
E10 Directory
Does the to verify support
product credentials
bothand retrieve
secure common forms of authentication
Different types of APIs and differentin use today.
types of
E11 channels and secure payloads?
Does the product or platform provide support data
CORS require differentresource
(Cross-origin types of security.
sharing) is a
E12 for
DoesCORS?
the platform protect against XML or standard
As part ofmechanism
a defense inthat allows
depth JavaScript
strategy, does the
E13 JSON
OAuth attacks?
2.0 doesn't include a mechanism for platform help in protecting against modern
E14 verifying the integrity
Can the product of payloads;
be extended Does the
to support
E15 custom/proprietary security mechanisms?
Can APIs be secured at the operation level?
E16 (Ex:
Can can
yourdo GET, enforce
product but not POST or PUT)
time-relative
authorization? For example, can your product
E17 Can your product expose APIs that bridge
E18 security
Does theprotocols? For example,
product include a secure,accept an
encrypted
E19 store? Can the product connect to a secure
Does your product have a way to report on the
E20 security
How does stance of all APIs
the solution managed
product within, to
mitigate
E21 sophisticated bot and malicious client
Can the product solution include third-party attacks?
E22 client verification, such as through Recaptcha
Is your public cloud offering PCI DSS certified? Many APIs require (or eventually require)
E23 If so, your
Does whatpublic
versions areoffering
cloud certified?
support the payment processing as part of the monetization
E24 delivery of HIPAA compliant services?
Is your public cloud offering HITRUST The HITRUST CSF is an industry-agnostic
certified? certifiable framework for regulatory compliance
Section F - Developer Portal
Requirement Details Response
F1 Please describe how the tool facilitates on- Developer and partner productivity depends on
F2 boarding. Can the provide
Does the solution portal be deployed as part of an
interactive efficient
While onboardingis experience.
documentation How does
important, experience
F3 documentation to allow API consumers
Is the registration form customizable? to shows
Corporate policies may dictate that we is
that a developer's time to value greatly
collect
F4 Can the customer customize, skin, and modify certain pieces of information when onboarding a
F5 the portal without vendor involvement?
Does the portal leverage standard CMS As a follow up to the previous question, if we are
F6 technologies
Does the toolto ensurethe
provide easy to find
ability skill sets
to revoke or to
In be
theable
event toofperform this work
an expired on our
contract withown,
a the
F7 suspend developer keys? developer or when an abnormal
Does the solution support a delegation model Large partners require the ability to maintain the situation
F8 which
Does theallows enterprises
developer portaltosupport
let theirintegration
partners existing relationships
Internal guidelines mightwithrequire
their own
the developers.
support of
F9 with existing Identity Providers? single sign-on with existing
What mechanisms for filtering which APIs are We want to make it easy for developers identity solutions.
to find
F10 visible
Pleaseto which API
describe the users
abilityasforthey browse or
the platform to the
Some appropriate
of the APIs API Product
will need toandbe also control
monetized.
F11 support monetization.
Are the pricing modelsWhat are the various
configurable without Given
Can the that there are
financial multiple
models ways tothrough
be created monetize
F12 coding?
Does the platform integrate with third-party configuration only or do they require
Once the metering has been performed, it will custom
payment systems? be necessary to pass the transaction to a
Section G - Microservices
Requirement Details Response
G1 Can the solution's capabilities be used to
G2 manage the consumption
Can the API managementof a microservice?
solution manage Microservices architectures are often polyglot
G3 multiple microservices, each
Can the API management solution built inact
a different
as a environments
Microservices consisting
architecturesof services built in
often contain many
G4 facade or lightweight composition
Can the API management solution manage layer, independent microservices, each
Many companies are transitioning to providing their
G5 multiple
Can APImicroservices
proxies be built alongside legacy
and deployed microservices architectures over
One of the benefits of a microservice time. During
G6 independently of other API proxies?
Does the API management solution support a architecture
Adding a callisout
thetoability to deploy
the internet them each of
to proxy
G7 hybrid
Can themodel for all of the inter-process
API management solution be used in the internal calls within a microservice mesh can
G8 combination with an existing
Can the API management microservices
layer be scaled at One of the benefits of a microservices
G9 the same rate as the underlying
Does the API management solution microservices
provide architecture is the ability to scale a microservice
G10 security policies for microservices?
Does the API management solution provide In a microservices architecture, gaining visibility
analytics capability for microservices? into the complex web of interdependencies can
Section H - Governance and SDLC
Requirement Details Response
H1 Does your API management product support We have, or may want to adopt, devops
H2 continuous
How integration
are APIs promoted and continuous
from delivery
development all the practices
APIs for automation,
will need workflows,
to be developed and moveprocesses,
thru
way tothe
production
platform and howdecentralized
does the system help? several different steps along the SDLC
H3 Does support Our organization has multiple business units
governance of independent API teams within an and/or divisions that operate independently of
H4 Is it possible for a company to enforce behavior for In some cases, we have security and
all APIs exposed by the system? governance
H5 How does the solution handle role based Auditing andrequirements that our APIs
compliance processes must
dictate that
H6 access
Does thecontrols
producttoinclude
ensureadifferent members
secure audit of RBAC (Role Based Access Control) is
trail to
H7 record what changes were made, by
Does the product include message logging whom,
H8 capabilities?
Does the product provide the ability to
H9 generate
Does the reports
producton security
allow policies thattoit
an administrator
view summaries of activity by other
Section I - Training and Support
Requirement Details Response
I1 Do you have 24X7 support?
I2 Please describe the maintenance schedule for
I3 your
Do youproduct
provide free training for architecture,
I4 development,
Do you provideand operations
online tutorialson
toyour
help website?
us learn
I5 your product?
Is there a user community that can be
leveraged to gain first-hand insights,