In association with By focusing on data, value

propositions, and talent,

enterprises can get ready to
take the next step beyond
experimentation.

Successfully
deploying machine
learning
2  MIT Technology Review Insights

Preface
“Successfully deploying machine learning” is an MIT Technology Review Insights
report developed in collaboration with JPMorgan Chase. The report is based on
interviews with analysts, scientists, and senior executives of companies using
machine learning and AI. The interviews were conducted in the autumn of 2022 to
evaluate the integration of AI and ML into disparate parts of the enterprise, while
scaling effectively and responsibly. Adam Green was the writer, Michelle Brosnahan
was the editor, and Nicola Crepaldi was the publisher. The research is editorially
independent, and the views expressed are those of MIT Technology Review Insights.

We would like to thank the following experts for providing their time and insights:

Eric Boyd, Corporate Vice President of AI Platform, Microsoft

David Castillo, Firmwide Head of AI/ML Technology and General Manager of AI/ML
Product Line, JPMorgan Chase

Samik Chandarana, Chief Data and Analytics Officer for Corporate and Investment
Banking, JPMorgan Chase

Drew Cukor, Firmwide Head of AI/ML Transformation and Engagement, JPMorgan

Chase

Iya Khalil, former Global Head of AI Innovation Lab, Novartis

Aleksander Mądry, Director of MIT Center for Deployable Machine Learning and
Cadence Design Systems Professor of Computing, MIT

Lidia Mangu, Managing Director, Head of Machine Learning Center of Excellence,

JPMorgan Chase
 MIT Technology Review Insights 3

Foreword

This is a pivotal moment for AI and machine learning. As a data scientist, I’m always thinking
about this. But the rise in popularity of AI and machine learning in people’s everyday lives,
and the daily prominence in news reports today, highlights the notable innovations in AI and
what they mean for individual consumers and some of the most impactful institutions in the
world. Collaborating on this report with MIT Technology Review Insights was an impactful
experience, as we had the opportunity to hear survey respondents and interviewees’
perspectives on AI in business.

At JPMorgan Chase, we excel at solving practical problems using cutting-edge techniques.

Our Machine Learning Center of Excellence (MLCOE) partners with business groups and
their dedicated data science teams, combining their business knowledge with our deep
AI and machine learning expertise to build and deploy high-impact solutions. To date, our
MLCOE has focused on natural language processing (NLP) and speech recognition, as well
as time series analysis, reinforcement learning, representation learning, recommendation
systems, and most recently, knowledge graphs. All of these MLCOE skills support the many
use cases at JPMorgan Chase that help us to create better, faster, smarter experiences for
our clients and customers.

Of course, none of this happens without talent. I’m very proud to work at JPMorgan Chase,
recognized as a financial services leader in AI and supporting more than 900 data scientists
across our company.

I hope you find this report inspiring as your teams dive deeper into the capabilities available
now, along with the new techniques that will emerge in this innovative field.

Dr. Lidia Mangu

Managing Director, Head of Machine Learning Center of Excellence, JPMorgan Chase
 4  MIT Technology Review Insights

0
CONTENTS

01 Executive summary.................................................................................5

Methodology................................................................................................6

02 Machine learning is serious business................................................7

Human-like characteristics of AI.............................................................8

From proving to scaling.............................................................................9

03 Building out machine learning............................................................12

Beyond the borders.................................................................................14

04 Mobilizing talent......................................................................................15

Sought-after hybrid talent......................................................................16

Improving access to ML.......................................................................... 17

05 ESG for the AI era....................................................................................19

06 Conclusion................................................................................................ 22
 MIT Technology Review Insights 5

01 Executive
summary

A
fter decades of research and development, core data scientists. Firms need hybrid and translator
mostly confined to academia and projects in talent to guide AI/ML design, testing, and governance,
large organizations, artificial intelligence (AI) and a workforce strategy to ensure all users play a role
and machine learning (ML) are advancing in technology development. Competitive companies
into every corner of the modern enterprise, should offer clear opportunities, progression, and
from chatbots to tractors, and financial markets to impacts for workers that set them apart. For the
medical research. But companies are struggling to move broader workforce, upskilling and engagement are key
from individual use cases to organization-wide adoption to support AI/ML innovations.
for several reasons, including inadequate or inappropriate
data, talent gaps, unclear value propositions, and • Centers of excellence (CoE) provide a foundation
concerns about risk and responsibility. for broad deployment, balancing technology-sharing
with tailored solutions. Companies with mature
This MIT Technology Review Insights report, capabilities, usually larger companies, tend to develop
commissioned by and produced in association with systems in-house. A CoE provides a hub-and-spoke
with JPMorgan Chase, draws from a survey of 300 model, with core ML consulting across divisions to
executives and interviews with seven experts from develop widely deployable solutions alongside bespoke
finance, health care, academia, and technology to tools. ML teams should be incentivized to stay abreast
chart elements that are enablers and barriers on the of rapidly evolving AI/ML data science developments.
journey to AI/ML deployment. The following are the
report’s key findings: • AI/ML governance requires robust model
operations, including data transparency and
• Businesses buy into AI/ML, but struggle to scale provenance, regulatory foresight, and responsible
across the organization. The vast majority (93%) AI. The intersection of multiple automated systems
of respondents have several experimental or in- can bring increased risk, such as cybersecurity issues,
use AI/ML projects, with larger companies likely to unlawful discrimination, and macro volatility, to advanced
have greater deployment. A majority (82%) say ML data science tools. Regulators and civil society groups
investment will increase during the next 18 months, and are scrutinizing AI that affects citizens and governments,
closely tie AI and ML to revenue goals. Yet scaling is with special attention to systemically important sectors.
a major challenge, as is hiring skilled workers, finding Companies need a responsible AI strategy based on
appropriate use cases, and showing value. full data provenance, risk assessment, and checks and
controls. This requires technical interventions, such as
• Deployment success requires a talent and skills automated flagging for AI/ML model faults or risks, as
strategy. The challenge goes further than attracting well as social, cultural, and other business reforms.
6  MIT Technology Review Insights

0
Methodology

The survey that forms the basis of this report was conducted by MIT Technology Review Insights in September
and October of 2022. Following are the key demographic details of the 300 executives who took part in it. The
respondents hold senior technology roles in their organizations and are C-level executives: chief information
officers; chief technology officers; chief data/analytics officers; and chief AI officers; as well as senior vice
presidents or vice presidents of AI, of data platforms, or of engineering; and heads of AI and machine learning.
These executives work in organizations that have between $1 billion and $500+ billion in revenue. The
executives surveyed are located in four regions: Asia-Pacific, Europe and North Africa, Latin America, and
North America. The industries represented in the survey include consumer goods and retail, financial services,
hospitality and leisure, IT and telecommunications, manufacturing, media and marketing, pharma and health
care, professional services, and transport and logistics.

“The journey is evolutionary,

but the technology is
revolutionary.”
Samik Chandarana,
Chief Data and Analytics Officer for
Corporate and Investment
Banking, JPMorgan Chase
 MIT Technology Review Insights 7

02 Machine learning is
serious business

A
I and ML are advancing in every corner of or infectious disease research, and in everyday
the modern enterprise. The achievements of contexts like anticipating customer behavior. Identifying
AI/ML, like the protein-folding predictive anomalies in large datasets can detect fraud, cancer,
powers of DeepMind’s AlphaFold or the and manufacturing defects.
rapid, human-like responses of OpenAI’s
ChatGPT, garner media attention, but may also signal the A 2022 MIT Technology Review Insights survey,
rise of “boring” AI tools—supply chain optimization or polling 300 respondents across four continents and
automated form completion—that offer some of the nine industries, found that 71% of respondents use
clearest, if less exciting, indications of the technology’s AI/ML across several projects, or heavily across the
fitness for real-world use. organization, with larger companies more likely to be
active adopters.
How AI will affect business depends on how these
capabilities apply to vastly different domains. Prediction, Financial firms are not new to data science, but many
for instance, is one of AI’s greatest assets, according powerful AI/ML techniques that were not computationally
to Eric Boyd, corporate vice president of AI Platform at feasible before are making waves, says Samik
Microsoft. AI and ML can be useful for environmental Chandarana, chief data and analytics officer, corporate
and public health contexts like weather forecasting and investment banking at JPMorgan Chase.

Figure 1: AI/ML adoption

Seventy-one percent of all respondents surveyed say AI/ML is either heavily in use or in use across some
projects.

27 44 22 7
MATURE/HEAVILY
IN USE
% %
IN USE FOR A FEW
PROJECTS
EXPERIMENTAL
%
NASCENT
%
Source: MIT Technology Review Insights survey, 2022
8  MIT Technology Review Insights

Figure 2: AI/ML adoption among companies of different sizes

Larger companies, which tend to have more mature AL/ML capabilities and do more development in-house,
are more likely to be active adopters of AI/ML

$50+ billion
27% 64% 9%

$101 - $500 billion

60% 30% 10%

$51 - $100 billion

43% 43% 14%

$11 - $50 billion

26% 52% 16% 6%

$1 - $10 billion
14% 40% 35% 11%

Source: MIT Technology Review Insights survey, 2022 Mature/heavily in use In use for a few projects Experimental Nascent

“Every trader, banker, and payments person has navigate sanctions on Russian entities due to the war in
always used data, so the journey is evolutionary, but the Ukraine—names of sanctioned entities can be difficult
technology is revolutionary.” In finance, AI/ML can be to weed out from ordinary addresses without advanced
used to optimize portfolios, underwrite loans, provide techniques like AI. “We can reduce false positives,”
automated customer support, and detect fraud, as well as explains Chandarana. “You can use ML to really power
numerous other emerging use cases.1,2 that ahead and increase people’s ability to get through
that workload significantly faster.”
As an example of some of the promising developments,
JPMorgan Chase uses AI tools to flag companies Human-like characteristics of AI
likely to see activist shareholder pressure when filings The future of AI/ML depends on how its capabilities
drift from performance norms or expectations, and to can be adapted for different domains.

“Monoliths—the idea that you can build a platform and

it makes everyone happy—don’t work.”
Drew Cukor, Firmwide Head of AI/ML Transformation and Engagement, JPMorgan Chase
 MIT Technology Review Insights 9

From proving to scaling a few pockets of data scientists and small teams doing
Companies often start AI/ML by identifying use cases ML on the side, but not as part of their formal role.
and building models to surpass the status quo. Once There was no infrastructure for ML yet.”
proven, they move to other departments or functions,
fine-tuning the tools, and securing engagement from A turning point, Mangu says, came when the MLCoE
the business. convinced the credit business to provide data for
predicting bond prices and used a deep learning-
JPMorgan Chase began exploring data science in the powered predictive model for the problem. “We showed
early 2010s, says Chandarana, but started working them the results and they were very happy and said
strategically with business units around 2017 through JPMorgan Chase already had the best bond pricing
its Machine Learning Center of Excellence (MLCoE). system in the market [at the time], so this baseline was
Lidia Mangu, head of the MLCoE recalls, “There were really tough to beat!” Then the ML team identified other

Figure 3: Investment in AI use cases

Financial firms are not new to data science, but the capabilities at their disposal are novel thanks to new
techniques and increased data availability.

AI’S HUMAN-LIKE CAPABILITIES EXAMPLES

Personalized, interactive technology for customer

Generate new, engaging, realistic content and
service, generating and reviewing code, and drafting
data based on existing ideas.
and sourcing complex documents.

Awareness and recognition of surroundings. Cameras in self-driving cars.

Interpreting and ascribing meaning to Converting hand-drawn sketches and notes to web
information. pages.

Knowing whether someone searching online for an

Handling ambiguity. object wants to buy the object or be connected with
fellow users.

Assessing if an unusual insurance claim is likely to

Applying judgment.
be fraudulent.

Weighing whether to follow an inconclusive health

Making decisions, even with incomplete
screening with a more invasive test, another
information.
screening, or an all-clear.

Source: Compiled by MIT Technology Review Insights based on data from Ana Landeta Echeberria, Artificial Intelligence for Business: Innovation, Tools and Practices, 20233
10 MIT Technology Review Insights

AI/ML applications in health care

Advancements in AI technologies like federated learning and swarm learning are paving the way for novel
AI applications in heavily regulated—and privacy forward—industries like health care.

One of the most promising early use cases of AI/ML in health care is in medical diagnostics. A 2022 study
conducted by the U.S. Government Accountability Office revealed three emerging approaches for AI/ML
diagnostic technologies: autonomous, adaptive, and consumer-oriented. From detecting cancer to predicting
Alzheimer’s risks, such advances help medical professionals identify and interpret the complex patterns of
diagnostic data faster and more accurately, to improve patient treatment. The study also uncovered potential
risks and limitations related to the efficacy of these advancements.

AI/ML technologies also are transforming medical research, such as helping to shorten drug development
timelines from years into weeks or days.4 Other advancements include:

• Alphabet’s DeepMind and Meta recently deployed AI-powered protein folding prediction models5, a task that
eluded scientists for decades, with profoundly positive implications for drug development and life sciences.

• Generative AI tools like OpenAI’s text-to-image DALL-E 2 are used by biotech researchers to design
proteins never before seen in nature.6

DESCRIPTION POTENTIAL BENEFIT POTENTIAL LIMITATIONS

• Technologies that • Fast, consistent information at the point • Developers may not be
independently interpret of care. able to create and medical
images or other patient professionals may not
data to render a • Improved clinician capacity and patient adopt algorithms that
Autonomous diagnosis. access. diagnosis certain diseases
technologies • Earlier and more accurate detection. autonomously.

• Technologies that • May provide more accurate diagnoses or • Changes in the algorithm
update their algorithms information by incorporating additional data may lead to adverse
by incorporating new population or individual data. outcomes such as
patient data. inconsistent or poorer
• Food and Drug Administration my be algorithmic performance.
Adaptive able to streamline its regulatory review
algorithms of adaptive algorithms by reviewing
potential changes to an algorithm
during the initial review phase, rather
than reviewing individual updates to
algorithms. This could allow for rapid
improvement of algorithms.
• Could expand or improve features for
users

• Technologies such as • Can give medical professional more • Need further research to
wearables and at- information about patients to improve understand whether some
home devices that are diagnosis and treatment. devices improve patient
marketed to consumers outcomes.
and may assist medical • May increase access to care for
Consumer- professionals in consumers, particularly in underserved • Effectiveness may depend
orientated monitoring a patient’s areas, such as rural settings, that lack on patient’s ability to
technologies medical conditions. specialists. understand or willingness
to accept the health
information presented.

Source: Compiled by MIT Technology Review Insights based on data from the U.S. Government Accountability Office, 2023.
 MIT Technology Review Insights 11

73 %
potential use cases, using natural language processing
models to extract information from support chats,
news, and emails—a laborious process for humans, but
easy for AI. “We did demo after demo and this led to an
avalanche of new projects—up to 70 in the past year
alone,” says Mangu. OF RESPONDENTS
FROM COMPANIES
According to the survey, 28% of executives see $500+ BILLION IN SIZE
scaling as their primary AI/ML deployment challenge. SAY FINDING USE CASES
Other polls agree: McKinsey found only 15% of IS THE BIGGEST OBSTACLE
respondents successfully scaled automation across
multiple parts of the business,7 and Gartner data the company to gather the data that is relevant,” he
shows just over half (53%) of projects make it from AI explains. Data can be noisy or of poor quality, or fail to
prototype to production.8 capture the trends needed for that solution.

Other key obstacles include hiring skilled workers, Models drift over time, he adds. “There are cases in
which survey respondents said was the number one the real world that you did not prepare your model
challenge. Finding use cases, proving economic value, for.” He cites credit card fraud protection algorithms
and outdated IT infrastructure were also primary issues being disrupted during the pandemic, when consumer
flagged by respondents. behaviors suddenly changed.

Aleksander Mądry, director of the MIT Center for Mądry warns media hype has led to unrealistic
Deployable Machine Learning, says data is a common expectations and uncritical enthusiasm from company
scaling and deployment problem. “Sometimes leaders. “They don’t actually stop and ask: is the problem
companies cannot run a particular application because I’m trying to solve an ML problem to begin with, do I have
they do not have sufficient data, or it is difficult for the right data for it, and is there even an objective?”

Figure 4: Key deployment challenges with AI/ML

Deployment challenges: two-thirds (66%) of respondents say hiring skilled workers is difficult. Question: What
are the primary challenges your company has had with deploying machine learning? (Top three choices.)

66% 47% 39% 33% 29%

Hiring skilled Finding use cases Showing value Lack of modern Getting executive
workers infrastructure buy in

28% 28% 15% 14%

Lack of usable Scaling Lack of investment Lack of compute

data/data strategy in ML/AI power
Source: MIT Technology Review Insights survey, 2022
12 MIT Technology Review Insights

03
A
Building out machine
learning

lthough off-the-shelf AI/ML tools are

available, companies that want to deploy at
scale tend to develop internally. The survey
found larger firms with mature AI/ML are
more likely to build in-house.
Figure 5: Characteristics of mature AI use
Of respondents from companies with mature
AI programs, most (73%) say AI usage is tied to
revenue, most (70%) say AI usage is federated
across the company, and nearly half (45%) say
CoEs are proving a popular structural approach to their AI tools are largely built in-house.
scaling AI and ML, and are used everywhere from Pfizer

73%
to Procter & Gamble.9 “There will be commonalities
across different use cases in the way that you want Tied to revenue goals
to brigade. You want this core expertise,” says Mądry.
“The center should have an internal consulting unit in

70%
which they are brought on to different use cases in the
Federated across the
company and work closely with people to deploy and organization
prioritize prototype solutions.”

55%
Boyd adds that CoEs can calibrate ML into a hub-and-
Providing a return on
spoke model. “We recommend companies start with a
investment
central team, but each business unit needs AI experts
because they understand the products, the customer,

48%
the concerns, and the things they’re going to need to do
Centralized into one
effectively.” department

“They bring the deep Being scaled/plans to scale

46%
business knowledge, we
bring the heavy-duty ML Largely being built in-house
45%
knowledge.”
Lidia Mangu, Managing Director, Head of
Allowing you to partner with
outside firms 32%
Machine Learning Center of Excellence, Source: MIT Technology Review Insights survey, 2022

JPMorgan Chase
 MIT Technology Review Insights 13

“Companies have to get their culture right so that they

can embrace a data-driven and experimentation-
driven mindset.”
Eric Boyd, Corporate Vice President of AI Platform, Microsoft

Drew Cukor, firmwide head of AI/ML transformation across functions, from wholesale banking to retail.
and engagement, at JPMorgan Chase, says it’s “There are certain models that we build for one asset
important to find the sweet spot between costly class in markets for credit, but with customization, we
bespoke products that cannot be reused versus single can also use it for equities, rates, or other businesses,”
platforms that cannot meet diverse needs. “Monoliths— says Mangu. The CoE capabilities Mangu’s team has
the idea that you can build a platform and it makes built out to support the firm include natural language
everyone happy—don’t work,” says Cukor. processing (NLP), speech analytics, recommendation
systems, anomaly detection, time series analysis,
Cukor says the bank has three criteria for assessing reinforcement learning, and graph modeling.
whether an ML project is worthwhile. The first: “Reuse.
In other words, when we [release it], can others grab There are parallels with the journey of Novartis,
it and reuse it so that similar use cases can quickly the Swiss pharmaceutical giant that is among the
benefit from the investment.” The others are cost and industry’s most active AI/ML enterprises. Its AI
speed—120 days is an optimal timeline from project innovation lab researches and commercializes novel
conception to production—a rough benchmark for therapies, helping “answer the questions that matter to
a single project. This ensures these tools are useful us and enrich the research around AI so that we can

Figure 6: Selected AI organizational models

STRUCTURE MODEL FUNCTIONS

• A center of excellence that works with all domains and departments.

Star • Anyone can approach the team to get their AI model or application built.
• Suitable for smaller companies, or companies just starting.

• Dedicated to individual business problems.

• Subset teams work directly with people in a domain.
Matrix • Team members stay in their horizontal, share ideas, learn techniques, and
compare domain-specific solutions.
• Suitable for larger or AI-mature companies.

• Every department has its own data scientists and engineers who build and
deploy models, without reporting to a central team.
Fully embedded • Enables focus and cross-functional communication, and eases incorporation
of AI solutions.
• Used by the largest and most AI-mature companies.

Source: Compiled by MIT Technology Review Insights based on data from Transforming Data with Intelligence (TWDI), 2021 10
14 MIT Technology Review Insights

0
enhance our drug discovery,” says Iya Khalil, who until Microsoft provides the engine for OpenAI’s
recently was the global head of the AI Innovation Lab breakthrough ChatGPT and GPT family of models.
at Novartis. “This is about scale; produce the algorithm And Microsoft’s Azure platform offers an ML stack,
once and now many different scientists can use it.” explains Boyd—a base layer of developer tools for the
The company has more than 200 data scientists ML lifecycle, a suite of cognitive services, and applied
working with biologists and clinical scientists. AI, which analyzes customers behavior. It offers not
just computing power, but model-testing and scrutiny
In the finance sector, Mangu advocates partnership to ensure responsible development, says Boyd.
between business units and ML experts, but the latter
must look outwardly and track field advances while In the case of Novartis, a global pharmaceutical
also remaining focused on internal business needs. company, it worked with the tech industry to divide
“They bring the deep business knowledge, we bring labor in ways that could play to its advantage.
the heavy-duty ML knowledge, while we always try to Novartis looks for fundamental AI research that tech
keep up with the state of the art, and be aware of what’s companies have developed that it can extend in the
happening in the ML world,” she explains. “If ML teams health-care space. “That’s one axis of the type of
were only embedded in the business, they might have partnership that we want to have; an entity that is
less time to keep current with ML advances.” doing pure AI research, and we can use those models
to drive our science,” says Khalil.
Beyond the borders
Companies often lack the capacity to build AI/ML Startups are also building technology and services to
internally, and as a result regularly collaborate with aid business adoption. Venture capital funds invested
vendors to meet their AI/ML needs. For example, $67 billion in AI specialist firms in 2022.11

“When you deploy machine learning, it’s not just about

getting technical trust in the system as an engineer,
it’s also about the people who use them actually
trusting it as well.”
Aleksander Mądry, Director of MIT Center for Deployable Machine Learning and Cadence
Design Systems Professor of Computing, MIT
 MIT Technology Review Insights 15

04 Mobilizing talent

T
he battle for AI/ML talent is a test across the Firms compete for a limited pool of ML talent, much
board; in a recent Deloitte study, one in four of which is targeted by Big Tech firms. Mangu notes,
AI-experienced companies had a moderate firms are more likely to succeed in recruitment if they
skills gap, with a further 23% calling it major or confidently articulate the unique opportunities they
significant.12 The MIT Technology Review offer. “We have vast amounts of data, a very diverse
survey shows that hiring skilled workers is the single range of use cases and projects, and the opportunity
greatest challenge for respondents in all but the very to make a large impact, to see internal and external
largest firms. The talent problem has three dimensions: clients using what you build,” says Mangu. She adds
finding individuals with core AI/ML skills, cultivating that AI/ML teams at the bank are smaller than in a
blended skills that combine data science with domain tech firm, meaning greater visibility and opportunities
knowledge, and producing a wider data-literate workforce. to progress.

Figure 7: Greatest challenges encountered by respondents while deploying AI/ML

REVENUE OF
GREATEST CHALLENGE PERCENTAGE
RESPONDENT COMPANY

$1-10 billion HIRING SKILLED WORKERS

69%
$11-50 billion HIRING SKILLED WORKERS
65%
$51-100 billion HIRING SKILLED WORKERS
65%
$101-500 billion HIRING SKILLED WORKERS
67%
$500+ billion FINDING USE CASES
73%
Source: MIT Technology Review Insights survey, 2022
16 MIT Technology Review Insights

Figure 8: Cultural changes introduced by AI/ML adoption

Most respondents (73%) expect ML adoption will promote agility, learning, and innovation.

58% Create 50%

Create a more fewer silos as
Promote agility,
competitive, departments work
learning, and
results-focused from the same
innovation
culture data

73%

Reduce
Increase job 44% Increase
friction between 42% 33%
satisfaction as education and
departments and
people do better training in data,
individuals that
work vs. rote machine learning,
use ML/AI
tasks and AI
differently

Source: MIT Technology Review Insights survey, 2022

Sought-after hybrid talent small and because encouraging expansion of lending

Companies need AI/ML skills blended with domain requires relying on “missing” data, domain expertise is
knowledge. Novartis identifies three types of talent: AI critical to determine the right modelling approaches
specialists, hybrid talents, and scientists. “You want all and evaluations of fitness for purpose.15
three of those in your organization, and then you have
to actively be able to connect the dots between these Chandarana warns against technology advocates
to drive progress,” says Khalil. promising AI/ML as a universal solution. With so much
input from outside financial services, context can be
Domain knowledge is vital to ensure AI/ML models lost, he argues, which can lead to overcomplicated
appropriately account for the input data, respond to solutions and data misinterpretation. Domain
a clearly definable problem, and anticipate regulatory knowledge makes sure experts can question or
or governance risks. The AI for Science research overrule a prediction or decision in vital areas like
community identifies domain knowledge as a “grand medicine, says Mądry.
challenge” in developing AI.13,14 This is seen in issues
like applying AI to credit delinquency: ML models could An emerging talent cluster is AI “translators,” people
improve predictive accuracy, reduce nonperforming who can bridge the gap between business and
loans, and even increase lending to creditworthy technical teams. This role translates business needs
customers who lack credit history (which may reflect into requirements and interprets AI results, guides
socioeconomic disadvantage rather than true risk). change management and AI integration, and provides
Because the number of credit delinquencies is so expertise in user experience design.16
 MIT Technology Review Insights 17

Company case studies

Upskill to bridge the AI/ML skill gap

Our survey showed that hiring skilled talent is one of the biggest challenges for companies implementing
AI technologies. Here are two companies taking an upskilling approach to help bridge that skill gap.

Levi Strauss & Co.

Levi Strauss & Co. introduced a companywide digital upskilling initiative with AI literacy courses for
merchandise planners, coding education, and coursework for employee skills in ML, design thinking, and
product management. In 2021, it introduced Machine Learning Bootcamp17, immersive training in coding and
ML. Senior management believes with staff engagement in AI/ML, they are ready for digital transformation.

McKinsey
McKinsey upskilled18 more than 500 employees in AI/ML techniques and applications. As a company-wide
initiative, McKinsey realized one-size-fits-all training wouldn’t be effective, as talent skillsets varied widely across
the organization. To make training accessible and successful, they developed a three-tier training program:

• AI aware: a beginner-level AI training program appropriate for anyone in the company to learn about AI
applications for business functions and the importance of data for AI applications for everyday work.
• AI ready: a series of custom AI bootcamps for those with enough domain knowledge to spot AI
opportunities. Internal experts run the bootcamps to demonstrate newer, broader AI technologies, and
then provide online tools for continuous learning.
• AI capable: an intensive six-month course to upskill technical talent for such roles as AI UX expert, AI
researcher, and machine-learning architect or engineer.

While recruiting and hiring skilled talent likely will be part of any successful AI/ML implementation, upskilling
can not only make AI technologies more accessible across an organization, but can make implementation and
deployment more successful. McKinsey notes, “no one understands how to get things done—and make them
stick—in your organization better than your own people.”

Improving access to ML
At scale, AI/ML impacts workflows as well as staff
“Companies have to get
roles and responsibilities. At best, it removes tedious,
repetitive, and dangerous tasks, freeing people for
their culture right so
advanced, value-added, and rewarding work. But some that they can embrace a
perceive digital transformation as a threat.
data-driven and
In a case study at the Wharton Business School, for
example, IT team members were perplexed by the shift experimentation-driven
to cloud computing, since managing the physical data
center was part of their role. Wharton used a pro- mindset.”
change mindset to reorient members who, liberated
from physical infrastructure, could now learn about and Eric Boyd, Corporate Vice President of
engage with the broader business.19 AI Platform at Microsoft
18 MIT Technology Review Insights

0
Boyd says culture is the biggest impediment to AI/
ML progress. “Companies have to get their culture “We want to make sure
right so that they can embrace a data-driven and
experimentation-driven mindset.” Successful firms
that every piece of data
achieve symmetrical AI culture, with harmony
and balance assimilating AI versus asymmetrical
has lineage, and we can
organizations that fail to synchronize effort.20 A Deloitte trace it back to the origin
study agrees: companies with leading AI practices
have cultural characteristics like cross-organizational and transformation of
collaboration and workforce optimism.21
that data.”
However, most companies are not taking concrete
steps to support the workforce for AI/ML. David Castillo, Firmwide Head of AI/ML
“Organizations are not paying enough attention to Technology and General Manager of AI/
training,” says Mądry. “It’s not just the training of ML ML Product Line, JPMorgan Chase
engineers, but the personnel on the ground.” Training
must be continually refreshed because of workforce
churn. It is a big investment, he says, “but this is what
it takes to deploy machine learning in a reliable and
valuable way.”

Professionals who are not experts in AI, like customer

service representatives, are no less important. These
customer-facing employees must understand what the
ML tells them, and what it does not, Mądry says. In the
MIT Technology Review survey, personalized customer
experience is the number one benefit ML offers.

Engaging the workforce is not one-directional; staff

should be part of technology development. “When you
deploy machine learning, it’s not just about getting
technical trust in the system as an engineer, it’s also
about the people who use them actually trusting it
as well,” says Mądry. “This involves guidance and
empowerment so that they can confirm the predictions,
or intervene when they think something is wrong.”
Research shows people at all levels of an organization
need to trust algorithms, and be able to make decisions
to override them, rather than needing to consult a
higher-up authority.22
 MIT Technology Review Insights 19

05 ESG for the AI era

M
any AI/ML stories focus on risks and responsibly for ML every step of the way,” says
pitfalls, such as biased decisions JPMorgan Chase’s David Castillo, firmwide head of
reflecting flaws in training data, AI/ML technology and general manager of the AI/ML
unpredictable behaviors, and even flash product line.
crashes caused by algorithm-based
automated trading. Moving forward with environmental, OmniAI, JPMorgan Chase’s in-house platform, helps
social and governance (ESG) issues shines light on data scientists and engineers (machine learning and
responsible AI, the catch-all term for the principles, data) find data, transform the data, create features,
policies, tools, and processes to ensure AI systems are and train models in a responsible manner. OmniAI
developed and operated in the service of good.23 enables AI to deploy at scale, standardizing processes
and providing security and controls for working with
The calls for explainability and accountability of AI/ confidential information.29 “By addressing controls
ML are governance challenges, especially in domains at that level, we’re creating a very interesting and
like credit or criminal justice, where parties have a legal uniquely streamlined opportunity for performing
right to dispute certain decisions. Reforms like the U.S. machine learning across the entire machine learning
Algorithmic Accountability Act of 202224 and the EU’s development lifecycle [MDLC],” says Castillo. “This
proposed AI legal framework25 show growing attention means we are being responsible across the entire
from governments to ESG risks—and a move to spectrum of the MDLC.”
complement or replace industry self-regulation.26,27
Once AI/ML is designed, tested, and built, it requires
A 2022 PwC poll suggests that responsible AI is a continuous attention. Contemporary data science
priority for firms, with 98% of respondents saying they tools continually learn and adjust in response to new
had plans to advance responsible AI in 2022, focusing data and trends. However, this can lead to unexpected
on compliance with regulations and protecting AI errors and the phenomenon of model drift.
systems from cyber threats.28 To anticipate risks, companies adopt algorithmic impact
assessments, based on systematic and structured
Responsible AI cannot be bolted on; it requires an risk assessments from the ESG sphere, to evaluate
end-to-end governance model and resourcing. Robust pathways and outcomes linked to new algorithms.30
data provenance and visibility is fundamental; without
it, companies struggle with decisions. “We want to Pressure to improve accuracy during the training phase
make sure that every piece of data has lineage, and we of model development can lead to overfitting to a
can trace it back to the origin and transformation of particular data set, which leaves the model vulnerable
that data. We are ensuring that data is being handled to errors after the model training process. “We want to
20 MIT Technology Review Insights

Increasing government attention to ESG

ESG in 2023: A selection of key ESG regulations and accountability around sustainable goods and
services, workers, stakeholders, human rights,
Australia environmental protection and restoration, political
Beginning in 2024, it plans to phase-in influence, inclusivity and equitable development,
required company-level data of carbon footprint, and consumer protection.
greenhouse gas (GHG) emissions, and climate
risk from large businesses as part of annual Singapore
financial reporting. The Green Finance Action Plan, based on
its UN 2030 net zero targets, creates ERM rules
Canada for banks, asset managers and insurers, a Green
From 2024, some banks, insurance Taxonomy (targeting the energy, transport, and
companies, and financial institutions must disclose real estate industries) and green loan framework,
how they assess and manage climate-related risks and disclosure requirements for ESG funds. It is
and opportunities, and identify how this impacts developing a blockchain-based system for ESG
business, strategy, and financial planning. They also data, using a common standard.
must detail ESG metrics and targets, including for
Scope 1, 2, and 3 GHG emissions. United States
The U.S. Securities and Exchange Commission
European Union (SEC) plans to enshrine rules that standardize
The EU is enacting ESG business regulations corporate ESG reporting with its upcoming ESG
across nearly all economic sectors, most of which and Climate Disclosure Rules. Federal and state
take effect between now and 2026. These include: governments are creating various systems of ESG
• Enhanced corporate financial disclosures regulations, such as the Inflation Reduction Act (IRA),
for sustainability spending, a taxonomy for which is the largest investment in climate change
sustainability activities, and supply chain due mitigation in U.S. history. The broad legislation
diligence rules. governs investment products with ESG features,
• Rules for specific products to boost circular GHG emissions disclosures for the largest federal
ecosystems and sustainability. contractors, and NASDAQ board diversity reporting.
• New laws for product labels, with penalties for
greenwashing. United Kingdom
• “Women on Boards” requires women hold at The Streamlined Energy and Carbon
least 40% of nonexecutive director positions at Reporting (SECR) is one of several new rules that
large companies by June 2026, or 33% applied require entities like large companies and investors
across executive and non-executive positions. and traders to complete yearly sustainability and
climate reporting, including documenting energy
India use, calculating carbon footprints, and accounting
The top 1,000 listed companies must produce for GHG emissions. The Sustainability Disclosure
a Business Responsibility Report (BRR) as a part of Requirements (SDR) targets greenwashing with
annual corporate reporting, detailing ESG metrics. investment label regulations, new disclosure
Metrics align with UN Sustainable Development requirements, and rules to limit marketing using
Goals (SDGs), and cover ethics, transparency “green” language.

Source: Compiled by MIT Technology Review Insights with data from Brightest, 202331
 MIT Technology Review Insights 21

maximize performance on holdout data, and that’s our ML tools are emerging to help companies which
proxy metric for initial model performance,” explains lack oceans of data, or which are unable to use the
Mądry. Confronting this pressure, he says, can be like data which they do hold. Transfer learning, reusing a
teaching a student who schemes to pass the exam pretrained model in a new ML model and leveraging
based on memorizing the specific answers at the small data sets to deploy large models in a reliable way
expense of learning the material at a deeper level. will impact the unfolding of ML, according to Mądry.
Generalized knowledge can be used when two models
A constant feedback and assessment loop should perform similar tasks, to reduce resources and labeled
“evaluate whether the system is doing what we data required to train new models. This is a significant
intended it to do, or if it happens to optimize internally to the evolution of ML and is increasingly used in
as a result of the process.” Mądry’s team is working development.35
on automated toolkits that recognize when model
performance is decreasing, to synthesize inputs that In health care, approaches are emerging to allow AI/
can help improve modeling and correct for the problem ML innovations without running afoul of regulatory
of drift. constraints on data privacy. “Everything around patient
records has to be HIPAA-compliant [a U.S. federal law
Gartner defines model operations (ModelOps or AI protecting patient health information], but there are
model operationalization) as “focused primarily on things that could help us take it to the next level, like
the governance and lifecycle management of a wide federated learning and swarm learning,” says Khalil.
range of operationalized AI and decision models, Swarm learning is decentralized ML combining edge
including machine learning, knowledge graphs, rules, computing, blockchain-based peer-to-peer networking,
optimization, linguistic, and agent-based models.”32 and coordination which can maintain confidentiality
without the need for a central coordinator.36 Also,
ModelOps can ensure end-to-end governance and policymakers are proposing regulatory sandboxes, like
lifecycle management of analytics, AI/ML, and decision the European Commission’s 2021 Artificial Intelligence
models. This helps validate models in production Act37 and associated incentives to encourage the safe
with effects spanning IT, risk, compliance, business testing of novel AI systems.
requirements, and operational efficiency.33,34

For Castillo, ModelOps, based on a similar lifecycle

management approach in software, is distinct. “The
controls that we have on our technology and the
controls that we have on our data aren’t necessarily the
same,” he explains. When JPMorgan Chase reviewed
the controls on the ML development lifecycle, 70% of
controls traditionally applied to software development
were unnecessarily applied to ML, and there were gaps
for ML issues not pertinent to software.

Cybersecurity is important since AI/ML systems require

large data sets, greater connectivity, and greater
digitization, which expand the risk surface. Castillo’s
team includes more than a dozen security architects
looking at threat vectors for each ML product area.
22 MIT Technology Review Insights

06 Conclusion

A
fter waves of hope and disappointment,
AI and ML are proving their utility in the
business world, but scaling across the
organization is a recurring challenge. Getting
and wrangling the right data, identifying clear
value propositions, and sourcing talent and skills are
recurring challenges.

But these can be overcome by putting in place the right

organizational structures, smart talent strategies that
emphasize the unique opportunities of ML outside of the
tech industry, and by harnessing the growing number of
tech tools that can guide responsible AI and ModelOps.
With the right strategy and sequencing, firms of all sizes
can move forward AI/ML initiatives from projects and
pilots to an organization-wide capability.

This report is for informational purposes only and it is not intended as legal,
tax, financial, investment, accounting, or regulatory advice. Opinions expressed
herein are the personal views of the individual(s) and do not represent the views
of JPMorgan Chase & Co. The accuracy of any statements, linked resources,
reported findings or quotations are not the responsibility of JPMorgan Chase & Co.
 MIT Technology Review Insights 23

References
1. “Anti-Fraud Technology Benchmarking Report,” Association of Certified Fraud Examiners and SAS, 2022, https://www.sas.com/content/dam/SAS/documents/briefs/
third-party/en/acfe-anti-fraud-technology-112576.pdf.
2. Ana Landeta Echeberria, Artificial Intelligence for Business: Innovation, Tools and Practices (Springer Nature, 2022).
3. ibid.
4. Bill Briggs, “Novartis empowers scientists with AI to speed the discovery and development of breakthrough medicines,” Microsoft, Nov 18, 2021, https://news.microsoft.
com/transform/novartis-empowers-scientists-ai-speed-discovery-development-breakthrough-medicines/.
5. Ewen Callaway, “AlphaFold’s new rival? Meta AI predicts shape of 600 million proteins,” Nature, Nov 1, 2022, https://www.nature.com/articles/d41586-022-03539-1.
6. Will Douglas Heaven, “Biotech labs are using AI inspired by DALL-E to invent new drugs,” MIT Technology Review, Dec 1, 2022, https://www.technologyreview.
com/2022/12/01/1064023/biotech-labs-are-using-ai-inspired-by-dall-e-to-invent-new-drugs/.
7. Rohit Panikkar, Tamim Saleh, Maxime Szybowski, and Rob Whiteman, “Operationalizing machine learning in processes,” McKinsey & Company, Sept 27, 2021, https://
www.mckinsey.com/capabilities/operations/our-insights/operationalizing-machine-learning-in-processes.
8. “Gartner Identifies the Top Strategic Technology Trends for 2021” (press release), Gartner, Oct 19, 2020, https://www.gartner.com/en/newsroom/press-releases/2020-
10-19-gartner-identifies-the-top-strategic-technology-trends-for-2021.
9. Thomas H. Davenport and Shivaji Dasgupta, “How to Set Up an AI Center of Excellence,” Harvard Business Review, Jan 16, 2019, https://hbr.org/2019/01/how-to-set-up-
an-ai-center-of-excellence.
10. Wilson Pang and Alyssa Rochwerger, “Choosing an Organizational Structure for Your AI Team,” TWDI, May 3, 2021, https://tdwi.org/articles/2021/05/03/ppm-all-
choosing-an-organizational-structure-for-your-ai-team.aspx.
11. “Artificial intelligence is permeating business at last: The age of “boring AI” will be anything but,” The Economist, Dec 6, 2022, https://www.economist.com/
business/2022/12/06/artificial-intelligence-is-permeating-business-at-last.
12. David Jarvis, “The AI talent shortage isn’t over yet,” Deloitte, Sept 30, 2020, https://www2.deloitte.com/us/en/insights/industry/technology/ai-talent-challenges-shortage.
html.
13. “AI for Science: Mind the Gaps,” NeurIPS AI for Science workshop, 2021, https://ai4sciencecommunity.github.io/neurips21.html.
14. Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan, “A review of some techniques for inclusion of domain-knowledge into deep neural networks,”
Nature, Jan 20, 2022, https://www.nature.com/articles/s41598-021-04590-0.
15. “Domain Knowledge (Data Science),” CFI, Nov 2, 2022. https://corporatefinanceinstitute.com/resources/data-science/domain-knowledge-data-science/.
16. “Talent and workforce effects in the age of AI,” Deloitte, 2020, https://www2.deloitte.com/content/dam/insights/us/articles/6546_talent-and-workforce-effects-in-the-
age-of-ai/DI_Talent-and-workforce-effects-in-the-age-of-AI.pdf.
17. “Training Our Employees for a Digital Future,” Levi Strauss & Co., May 17, 2021, https://www.levistrauss.com/2021/05/17/machine-learning-bootcamp/.
18. “Our own digital journey: Upskilling hundreds of McKinsey technologists in AI,” McKinsey & Company, December 20, 2021, https://www.mckinsey.com/about-us/new-at-
mckinsey-blog/ai-upskilling-for-over-500-firm-technologists.
19. “How Organizational Culture Shapes Digital Transformation,” Knowledge at Wharton, Aug 9, 2019, https://knowledge.wharton.upenn.edu/article/fedex-digital-
transformation/.
20. J. Mark Munoz and Oliver Degnan, “The Geometry of Organizational AI Culture,” California Management Review, Jun 24, 2021, https://cmr.berkeley.edu/2021/06/the-
geometry-of-organizational-ai-culture/.
21. “Fueling the AI transformation: Four key actions powering widespread value from AI, right now,” Deloitte, October 2022, https://www2.deloitte.com/content/dam/Deloitte/
us/Documents/deloitte-analytics/us-ai-institute-state-of-ai-fifth-edition.pdf.
22. Brian McCarthy and Tamim Saleh, “Building the AI-Powered Organization,” Harvard Business Review, July 2019, https://hbr.org/2019/07/building-the-ai-powered-
organization.
23. Elizabeth M. Renieris, David Kiron, and Steven Mills, “To Be a Responsible AI Leader, Focus on Being Responsible,” MIT Sloan Management Review, Sept 19, 2022, https://
sloanreview.mit.edu/projects/to-be-a-responsible-ai-leader-focus-on-being-responsible/.
24. “H.R.6580: Algorithmic Accountability Act of 2022,” Congress.gov, February 4, 2022, https://www.congress.gov/bill/117th-congress/house-bill/6580/text.
25. “Proposal for a Regulation laying down harmonised rules on artificial intelligence,” European Commission, April 21, 2021, https://digital-strategy.ec.europa.eu/en/library/
proposal-regulation-laying-down-harmonised-rules-artificial-intelligence.
26. Luciano Floridi, “The End of an Era: from Self-Regulation to Hard Law for the Digital Industry,” Springer Philosophy & Technology, Nov 10, 2021, https://link.springer.com/
article/10.1007/s13347-021-00493-0.
27. Jakob Mökander, Prathm Juneja, David S. Watson, and Luciano Floridi, “The US Algorithmic Accountability Act of 2022 vs. The EU Artificial Intelligence Act: what can they
learn from each other?” Springer Minds and Machines, Aug 18, 2022, https://link.springer.com/article/10.1007/s11023-022-09612-y.
28. “PwC 2022 AI Business Survey,” PwC, 2022, https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-business-survey.html#protect-yourself-make-your-ai-responsible.
29. “Omni means “all”,” JPMorgan Chase,” 2023, https://www.jpmorgan.com/insights/technology/omni-ai.
30. “Examining the Black Box: Tools for assessing algorithmic systems,” Ada Lovelace Institute and DataKind UK, 2020, https://www.adalovelaceinstitute.org/wp-content/
uploads/2020/04/Ada-Lovelace-Institute-DataKind-UK-Examining-the-Black-Box-Report-2020.pdf.
31. “Key ESG Regulatory Policies, Laws, Rules, and Reporting Requirements in 2023,” Brightest, January 27, 2023, https://www.brightest.io/esg-regulations.
32. “Information Technology Gartner Glossary: ModelOps,” Gartner, 2023, https://www.gartner.com/en/information-technology/glossary/modelops.
33. “ModelOps vs. MLOps,” ModelOp, 2022, https://www.modelop.com/modelops-and-mlops/.
34. “Top 10 Questions About MLOps and ModelOps,” ModelOp, 2022, https://www.modelop.com/wp-content/uploads/2021/10/Top_10_Questions.pdf.
35. “Transfer Learning for Machine Learning,” Seldon, Jun 29, 2021, https://www.seldon.io/transfer-learning.
36. Stefanie Warnat-Herresthal, et al., “Swarm learning for decentralized and confidential clinical machine learning,” Nature, May 26, 2021, https://www.nature.com/articles/
s41586-021-03583-3.
37. Tambiama Madiega and Anne Louise Van De Pol, “Artificial intelligence act and regulatory sandboxes,” European Parliamentary Research Service, June 2022, https://
www.europarl.europa.eu/RegData/etudes/BRIE/2022/733544/EPRS_BRI(2022)733544_EN.pdf
24 MIT Technology Review Insights

About MIT Technology Review Insights

MIT Technology Review Insights is the custom publishing division of MIT Technology Review, the world’s
longest-running technology magazine, backed by the world’s foremost technology institution—producing
live events and research on the leading technology and business challenges of the day. Insights conducts
qualitative and quantitative research and analysis in the U.S. and abroad and publishes a wide variety of content,
including articles, reports, infographics, videos, and podcasts. And through its growing MIT Technology Review
Global Insights Panel, Insights has unparalleled access to senior-level executives, innovators, and
entrepreneurs worldwide for surveys and in-depth interviews.

About JPMorgan Chase

JPMorgan Chase & Co. (NYSE: JPM) is a leading financial services firm based in the United States of America
(“U.S.”), with operations worldwide. JPMorgan Chase had $3.7 trillion in assets and $303 billion in stockholders’
equity as of March 31, 2023. With over 55,000 technologists globally and an annual tech spend of $14 billion,
JPMorgan Chase is dedicated to improving the design, analytics, development, coding, testing and application
programming that goes into creating high quality software and new products. Under the J.P. Morgan and Chase
brands, the Firm serves millions of customers in the U.S., and many of the world’s most prominent corporate,
institutional and government clients globally. Visit http://www.jpmorganchase.com/tech for more information.

Illustrations
Illustrations assembled by Tim Huxford with elements from Shutterstock.

While every effort has been taken to verify the accuracy of this information, MIT Technology Review Insights cannot accept any responsibility or liability for reliance by any person
in this report or any of the information, opinions, or conclusions set out in this report.

© Copyright MIT Technology Review Insights, 2023. All rights reserved.

MIT Technology Review Insights

www.technologyreview.com
[email protected]