CHECKLIST 2023

Real-Time Data Processing:

Five Use Cases to Know
By Fern Halper, Ph.D.
 Real-Time Data Processing:
Five Use Cases to Know
By Fern Halper, Ph.D.

O
rganizations are leveraging higher volumes
of diverse data in order to remain Five use cases for real-time data processing:
competitive and successful in a complex
and ever-evolving environment. This includes real- 1 Situational intelligence
time data and analytics. TDWI research indicates
real-time and streaming data are becoming more 2 Hyper-personalization
mainstream for numerous use cases. This data may
include machine-generated data, IoT data, or other 3 Operational intelligence using IoT data
streaming data types.
4 Smart products and services
Real-time data provides businesses with current
information, allowing them to make informed 5 Industrial control
(often automated) decisions quickly to become
more competitive. It can be used to drive top-line
benefits by better understanding customer behavior recommendation engines or promotional activities.
and improving the customer experience. It can also They are also finding applications for IoT analytics,
help improve operational processes by monitoring such as predicting if a machine part will fail.
systems and quickly identifying bottlenecks,
problems, and other inefficiencies. The term “real time”, however, has become broad
in its meaning. For some organizations, real-time
Successful companies are using machine learning means minutes or seconds (or even hours). In past
models for real-time data in use cases such as TDWI research for instance, respondents have

1   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: REAL-TIME DATA PROCESSING: FIVE USE CASES TO KNOW

defined real time as current enough for an application

or at the time it is needed. Yet, modern analytics
and applications often require very fast data. For 1 Situational intelligence
instance, use cases in some organizations require
millisecond response times where decisions and Situational intelligence refers to the ability of
actions need to be made immediately. In other words, organizations to analyze and act on real-time data
there are use cases where extreme real-time data to make informed decisions in situations important
response is required. for the business. Situational intelligence applications
have been in production for years, especially in areas
The challenge organizations have in dealing with such as disaster response or military operations.
extreme real-time data is that their architecture However, this kind of application is also important in
simply cannot scale to support high-volume/ business, especially when real-time action needs to
high-velocity data-intensive workloads. Extreme occur. Although not all situational intelligence use
real-time data has highlighted the limitations of cases involve extreme real-time data, it is necessary
traditional relational database management systems for some use cases involving critical infrastructure
(RDBMS) in terms of speed and scalability. Having such as network analysis or cybersecurity as well
to query a traditional database and then return the as financial use cases such as fraud detection and
answer adds latency to an application. Traditional prevention.
platforms cannot process millions of concurrent
transactions per second while maintaining low Use cases include:
latency and high throughput with no data loss.
• Fraud prevention. Machine learning is often
What is needed for an increasing number of use used in financial services to identify patterns
cases is a modern real-time platform that can handle associated with fraudulent transactions versus
massive amounts of data with millisecond response non-fraudulent transactions. By leveraging
times. It is not a matter of throwing hardware and real-time data and responding in milliseconds,
software stacks at the problem. Modern real-time organizations can improve their situational
platforms must be optimized for real-time data awareness, detect anomalous behavior, and
processing. This means the platform is able to handle prevent fraudulent activities before they occur,
high-velocity data streams and perform complex prior to the transaction being completed—leading
analytics on that data in real time, with low latency— to better outcomes and reduced financial losses.
which often means in-memory.
• Network analysis. Managing the quality of a
This TDWI Checklist Report examines five categories network service with millions of elements spread
of business use cases for extreme real-time data that across the country relies on real-time streaming
can benefit a wide range of organizations. It also analytics. The quality of that network needs
discusses how a modern real-time data platform can to be monitored and managed for customer
support these use cases. satisfaction and updated every second. For
instance, in a cellular network, the quality of
the network is continually assessed by real-time

2   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: REAL-TIME DATA PROCESSING: FIVE USE CASES TO KNOW

monitoring to identify overloaded cells and

areas of weak transmission. Identifying these
weaknesses allows the provider to route calls 2 Hyper-personalization
through less-trafficked parts of the network. All
of this may require millisecond response time. Personalization is critical in the digital age—
consumers expect it. Hyper-personalization is the
• Cybersecurity. Cyber threats to data continue practice of using customer data to create highly
to be a major issue for companies, and TDWI customized experiences tailored to an individual
research indicates that security is a major user’s preferences and interests. Real-time data
priority for organizations. Cybersecurity focuses provides information about customers’ behaviors
on specific technical implementations needed and preferences in order to deliver more targeted
to protect systems and networks. Modernizing marketing offers. Real-time data also allows
cybersecurity involves enabling better and marketers to adapt their strategies based on what
faster ways of identifying and responding to they’re seeing in current activity. At TDWI, we see
threats, which often involve real-time data and a lot of activity around the use of recommendation
analysis. For example, it can take companies engines on websites and the adoption of
days to respond to malware attacks (if they programmatic ad networks.
detect them at all). However, some organizations
are implementing new technologies that use Some examples of real-time hyper-personalization
machine learning and deep learning algorithms include:
to detect and ward off threats in real time.
• Recommendation engines. Recommendation
All these examples make use of extreme real-time engines utilize customer history as well as the
data. For instance, in the case of fraud detection, it history of others with similar profiles to identify
is necessary to process huge volumes of card swipes, products to be promoted as a user browses a
credit card taps, and online payments per second, website in real time. By leveraging tools such as
with only milliseconds for the actual fraud detection machine learning, companies can automatically
algorithm. In the case of network analysis, it is tailor offers and promotions to each customer’s
necessary to process vast amounts of packets and needs in real time (such as at checkout). This
determine how they should be routed for optimal type of customization helps businesses build
network efficiency. A modern real-time platform loyalty while increasing engagement with their
can support these kinds of use cases by providing target audience.
real-time ingestion, performing complex analytics
(such as machine learning) in memory, and providing • Programmatic advertising. Programmatic
accurate and up-to-date aggregate numbers for advertising is a form of automated media buying
broader decision-making, such as number of devices that utilizes real-time data to purchase ad
in a Zip code right now. impressions. This streamlined process allows
for highly sophisticated targeting and helps
marketers to reach their audience quickly and
efficiently. Machine learning and other advanced
analytics are often used in programmatic

3   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: REAL-TIME DATA PROCESSING: FIVE USE CASES TO KNOW

advertising. With real-time data, programmatic sensors and other devices is being used to determine
advertisers can immediately respond to consumer when a part failure might occur. Sensors may collect
behavior changes and trends to optimize vibration, rotor position, temperature, pressure flow,
campaigns in real time. and other data for analysis, either in aggregate or
in the stream. Common failure patterns are used to
• Gameplay adjustments. Gamers need to be identify imminent failures and proactively either
challenged, but not comprehensively defeated, shut down the machine or notify individuals to take
when they play. This means that in addition to the proper action to replace the indicated parts.
the immediate state, a gaming platform needs
to track the performance of a player and tweak Data for IoT use cases may involve massive amounts
the difficulty as the game progresses, without of data from multiple sensors, some of which are
the player even realizing that their playing on the edge of the network. That means that the
experience is being closely managed. data needs to first move through the network and
then be ingested, aggregated, and analyzed quickly.
Real-time data platforms are an important part of Additionally, a key aspect of edge computing is that
hyper-personalization. By leveraging a real-time often it is involved in use cases that also include
platform that can aggregate historical data as well “digital twin” technology—this utilizes virtual
as real-time data and analyze it quickly, companies representations of physical assets or processes for
can create customized offers for individuals at any simulation. For instance, data may be sent in real
given moment based on their particular situation or time to a digital twin to simulate certain conditions
context—the offer is happening during the event in and make adjustments in real time. This can require
milliseconds, not after the fact. high-speed processing and orchestration of multiple
device data streams in the same facility. Sending the
data to a hyperscaler data center is not practical.

A modern data platform can support these kinds of

Operational intelligence
3 using IoT data
use cases by taking multiple streams of data, analyzing
them, and then alerting those who need to know.

TDWI sees a lot of interest in the Internet of Things

(IoT). For instance, in a recent TDWI survey, over
20% of respondents were already collecting machine-
Smart products and
generated data. IoT devices —including sensors,
RFID tags, and more—are already in your home in
4 services
smart appliances; you’re wearing them for health
and wellness. They are on factory floors, in office More organizations and government agencies
buildings, on farms, and much more. are looking to offer smart products and services.
Examples include smart wearable devices, smart
For example, predictive and preemptive maintenance thermostats, connected appliances, smart meters,
is a very popular use case for IoT. Here, data from patient monitors, and more. Some of these examples

4   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: REAL-TIME DATA PROCESSING: FIVE USE CASES TO KNOW

don’t necessarily require a millisecond response

time, but many do.
5 Industrial control
One example application is traffic congestion where
connected cars use real-time data to monitor and Industry 4.0, also known as the fourth industrial
optimize performance, while also providing real-time revolution, is a term used to describe the integration of
navigation, traffic updates, and safety alerts. Another advanced technologies, such as artificial intelligence,
example is smart cities that might include smart the Internet of Things (IoT), and real-time data, into
energy grids to automatically optimize the use of manufacturing and other industries.
resources. Both of these examples require real-time
data as well as contextual data such as location data. In Industry 4.0, real-time data plays a critical role
Another example is a smart medical device that might in enabling the continuous flow of information
aggregate data including respiratory rate in addition between machines, systems, and people to create
to other vital signs to determine the probability of a a highly connected and responsive production
medical outcome in real time. environment. Real-time data is used to monitor and
control production processes, allowing for real-time
Smart products and services process huge amounts decision-making and rapid response to changing
of transactions and need to analyze this data in real conditions. This leads to increased efficiency,
time. For instance, a smart city application to estimate reduced downtime, and improved product quality.
traffic congestion and reroute city drivers (similar The use of real-time data in Industry 4.0 is an
to the connected cars application) might collect important factor in driving innovation and competi-
sensor data every second from different intersections tiveness in the manufacturing industry.
that needs to be analyzed in real time. It might
also use contextual data such as the location of the Some examples include:
intersection as part of the model. Model results can be
sent to a service that notifies drivers of a new route if • Connected machines. Linked to the smart
the model predicts congestion. product concept discussed earlier, connected
machines exchange information. For example,
In all these examples, a real-time data platform can in agriculture, connected farm equipment
provide fast ingestion of multiple high-velocity data can coordinate and optimize farming systems
streams into the platform, combining and orchestrating using geolocation data. In the connected
the workflows, to process this data and perform factory, machines can exchange data with other
complex operations including data aggregation and machines, processes, and people to optimize
analysis with millisecond response time. production processes, improve product quality,
and reduce downtime. In utilities, such as
energy, connected devices and systems are used
to monitor energy usage and optimize energy
production and distribution.

5   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: REAL-TIME DATA PROCESSING: FIVE USE CASES TO KNOW

• Robotics. Robotics technology is being used

in manufacturing to automate repetitive and
dangerous tasks. Robots are being integrated Concluding thoughts
into the production line to perform tasks such
as assembly and inspection. Some robots have In all the use cases described in this report (and
sensors and cameras to provide real-time data those yet to come), a modern, real-time data
to optimize their performance. A connected platform will be needed. The platform provides
robot may be able to receive information the ability to make decisions based on machine
about a change in the production process and learning and other advanced analytics applied to
immediately adjust its performance accordingly. a large event stream. This platform will handle
real-time data ingestion (including multiple streams
• Shop floor orchestration. Orchestration from different channels), process transactions, and
enables large numbers of devices to operate as enable advanced analytics with millisecond response
a team instead of as a collection of individual time—while maintaining ACID compliance. In order
agents. Orchestration is a key for increasing to accomplish this, the modern real-time platform
productivity and ROI of shop floor–level robotics. needs to support in-memory processing to maintain
Additionally, the more intelligence that is put state and make decisions in the moment and at scale.
into the orchestration layer, the less may be
needed for the individual devices.

In these use cases, data will be coming from

multiple sources and needs to be analyzed at
About our sponsor
once. Applications built on modern real-time data
platforms require in-memory capabilities to analyze
the data from multiple devices, in real time, to make
a decision and take action.
The Volt Active Data Platform enables companies to
unlock the full value of their data and applications
by making it possible to have scale without
compromising on speed, accuracy, or consistency.
Based on a simplified stack and an ingest-to-
action layer that can perform sub 10-millisecond
decisioning, Volt’s unique, no-compromises
foundation gives enterprises the ability to
maximize the ROI of their 5G, IoT, AI/ML, and other
investments, ensure “five 9’s” uptime, prevent fraud
and intrusion, deliver hyper-personalized customer
engagement, and save on operational costs.

Visit www.voltactivedata.com to learn more.

6   TDWI RESEARCH tdwi.org

 TDWI CHECKLIST REPORT: REAL-TIME DATA PROCESSING: FIVE USE CASES TO KNOW

About the author About TDWI Research

Fern Halper, Ph.D., is vice president TDWI Research provides industry-leading research
and senior director of TDWI Research and advice for data and analytics professionals
for advanced analytics. She is well worldwide. TDWI Research focuses on modern data
known in the analytics community, management, analytics, and data science approaches
having been published hundreds of and teams up with industry thought leaders and
times on data mining and information practitioners to deliver both broad and deep
technology over the past 20 years. Halper is also understanding of business and technical challenges
coauthor of several Dummies books on cloud surrounding the deployment and use of data and
computing and big data. She focuses on advanced analytics. TDWI Research offers in-depth research
analytics, including predictive analytics, machine reports, commentary, assessments, inquiry services,
and topical conferences as well as strategic planning
learning, AI, cognitive computing, and big data
services to user and vendor organizations.
analytics approaches. She has been a partner at
industry analyst firm Hurwitz & Associates and a
lead data analyst for Bell Labs. She has taught at both
Colgate University and Bentley University. Her Ph.D.
is from Texas A&M University. You can reach her by About TDWI Checklist Reports
email ([email protected]), on Twitter (twitter.com/
fhalper), and on LinkedIn (linkedin.com/in/fbhalper). TDWI Checklist Reports provide an overview of
success factors for a specific project in business
intelligence, data warehousing, analytics, or a related
data management discipline. Companies may use
this overview to get organized before beginning a
project or to identify goals and areas of improvement
for current projects.

A Division of 1105 Media © 2023 by TDWI, a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or part
6300 Canoga Avenue, Suite 1150 are prohibited except by written permission.

Woodland Hills, CA 91367 Email requests or feedback to [email protected].

Product and company names mentioned herein may be trademarks and/or registered trademarks
E [email protected] of their respective companies. Inclusion of a vendor, product, or service in TDWI research does not
constitute an endorsement by TDWI or its management. Sponsorship of a publication should not be
tdwi.org construed as an endorsement of the sponsor organization or validation of its claims.

7   TDWI RESEARCH tdwi.org