CHAPTER 1.

UNDERSTANDING
BIG DATA

PROFESSOR: LEE YOUNG-CHAN

BIG DATA FUNDAMENTALS: CONCEPTS, DRIVERS & TECHNIQUES – PRENTICE HALL

1
OUTLINE

 Concepts and Terminology

 Big Data Characteristics
 Different Types of Data
 Case Study Background
2
BIG DATA OVERVIEW

 What is Big Data?

 When is Big Data used?
 - -> Big Data is a field dedicated to the analysis, processing, and
storage of large collections of data that frequently originate from
disparate sources.
 - -> Big Data is used when traditional data analysis, processing
and storage technologies and techniques are insufficient. 3
INSIGHTS AND BENEFITS OF BIG DATA
 Operational optimization
 Actionable intelligence
 Identification of new markets
 Accurate predictions
 Fault and fraud detection
 More detailed records
 Improved decision-making
4

 Scientific discoveries
CONCEPTS AND TERMINOLOGY

5
DATASETS

 Datasets are collections or groups of related data.

 Each group or dataset member (datum) shares the same set of
attributes or properties as others in the same dataset.
 Examples:
 tweets stored in a flat file
 a collection of image files in a directory
 an extract of rows from a database table stored in a CSV formatted file
 historical weather observations that are stored as XML files 6
DATASETS

 Figure 1.1: three datasets based on three different data formats.

7
DATA ANALYSIS

 Data Analysis is the process of examining data to find facts,

relationships, patterns, insights and/or trends.
 Overall goal of data analysis is to support better decision
making.
 Example:
 the analysis of ice cream sales data in order to determine how the number of ice
cream cones sold is related to the daily temperature.
 The results support decisions related to how much ice cream a store should order in
relation to weather forecast information. 8
DATA ANALYSIS

 Figure 1.2: The symbol used to represent data analysis.

9
DATA ANALYTICS

 Data Analytics is a discipline that includes

 (a) the management of the complete data lifecycle (which
encompasses collecting, cleansing, organizing, storing,
analyzing and governing data),
 (b) the development of analysis methods, scientific
techniques and automated tools.
 Is a broader term that encompasses data analysis. 10
DATA ANALYTICS

 Figure 1.3: The symbol used to represent data analytics.

11
DATA ANALYTICS

 Different kinds of organizations use data analytics tools and techniques in

different ways.
 Example:
 In business-oriented environments, data analytics results can lower operational
costs and facilitate strategic decision-making.
 In the scientific domain, data analytics can help identify the cause of a
phenomenon to improve the accuracy of predictions.
 In service-based environments like public sector organizations, data analytics can
help strengthen the focus on delivering high-quality services by driving down costs.
12
DATA ANALYTICS

 Four general categories of analytics that are distinguished by the

results they produce.

descriptive diagnostic predictive prescriptive

analytics analytics analytics analytics

13
DATA ANALYTICS
 Figure 1.4: Value and complexity increase from descriptive to prescriptive
analytics.

14
DESCRIPTIVE ANALYTICS
 To answer questions about events that have already occurred. It contextualizes
data to generate information.
 Sample questions:
 What was the sales volume over the past 12 months?
 What is the number of support calls received as categorized by severity and geographic location?
 What is the monthly commission earned by each sales agent?
 The result reports are generally static, display historical data in the form of data grids or charts.

 Queries are executed on operational data stores from within an enterprise, for
example a Customer Relationship Management system (CRM) or Enterprise
Resource Planning (ERP) system via ad-hoc reporting or dashboards. (see Figure 15

1.5)
DESCRIPTIVE ANALYTICS
 Figure 1.5: The operational systems, pictured left, are queried via descriptive
analytics tools to generate reports or dashboards, pictured right.

16
DIAGNOSTIC ANALYTICS
 To determine the cause of a phenomenon that occurred in the past using
questions that focus on the reason behind the event.
 Sample questions:
 Why were Q2 sales less than Q1 sales?
 Why have there been more support calls originating from the Eastern region than
from the Western region?
 Why was there an increase in patient re-admission rates over the past three
months?
 The executed queries are performed on multidimensional data held in
analytic processing systems performing drill-down and roll-up analysis. 17

(see Figure 1.6)

DIAGNOSTIC ANALYTICS

 Figure 1.6: Diagnostic

analytics can result in
data that is suitable for
performing drilldown and
roll-up analysis.

18
PREDICTIVE ANALYTICS

 To determine the outcome of an future event and generate future

predictions based on a built model.
 Sample questions (what-if rationale):
 What are the chances that a customer will default on a loan if they have missed a
monthly payment?
 What will be the patient survival rate if Drug B is administered instead of Drug A?
 If a customer has purchased Products A and B, what are the chances that they will
also purchase Product C?
19
PREDICTIVE ANALYTICS

 The strength and magnitude of the associations based upon past

events will form the basis of models (which include patterns, trends
and exceptions found in historical and current data).
 The models have implicit dependencies on the conditions under
which the past events occurred. If these underlying conditions
change, then the models that make predictions need to be updated.

20
PREDICTIVE ANALYTICS
 Figure 1.7: Predictive analytics tools can provide user-friendly front-end
interfaces.

21
PRESCRIPTIVE ANALYTICS
 Prescriptive analytics build upon the results of predictive analytics by
prescribing actions that should be taken and explaining the reason
“why” (because they embed elements of situational understanding).
 Sample questions:
 Among three drugs, which one provides the best results?
 When is the best time to trade a particular stock?
 Various outcomes are calculated, and the best course of action for
each outcome is suggested. This approach shifts from explanatory
to advisory and can include the simulation of various scenarios. 22
PRESCRIPTIVE ANALYTICS

 Figure 1.8: Prescriptive analytics

involves the use of business rules and
internal and/or external data to
perform an in-depth analysis.

23
BUSINESS INTELLIGENCE (BI)

 BI applies analytics to large amounts of enterprise’s data generated

by its business processes and information systems (which has
typically been consolidated into an enterprise data warehouse) to
gain insight into the performance of an enterprise.
 The output of BI can be surfaced to a dashboard that allows
managers to access and analyze the results and potentially refine
the analytic queries to further explore the data. (see Figure 1.9)
24
BUSINESS INTELLIGENCE (BI)
 Figure 1.9: BI can be used to improve business applications, consolidate data in data
warehouses and analyze queries via a dashboard.

25
KEY PERFORMANCE INDICATORS (KPI)
 A KPI is a metric
 to gauge success within a particular business context.
 to identify business performance problems and demonstrate regulatory
compliance.
 is the quantifiable reference points for measuring a specific aspect of a
business’ overall performance.
 KPIs are linked with an enterprise’s overall strategic goals and
objectives.
 KPIs are often displayed via a KPI dashboard, and compare the 26

actual measurements with threshold values of KPI. (see Figure 1.10)

KEY PERFORMANCE INDICATORS (KPI)
 Figure 1.10: A KPI dashboard acts as a central reference point for gauging business
performance.

27
BIG DATA CHARACTERISTICS

28
FIVE BIG DATA TRAITS

 Are commonly referred to as the Five Vs

 To differentiate “Big” data from other forms of data.

29
VOLUME
 The anticipated volume of data is high, substantial and ever-growing.
 Figure 1.12 provides a visual representation of the large volume of data being
created daily by organizations and users world-wide.

Figure 1.12: Organizations and users

world-wide create over 2.5 EBs of
data a day. As a point of comparison,
the Library of Congress currently
holds more than 300 TBs of data.

30
VOLUME

 Typical data sources for generating high data volumes:

 online transactions, such as point-of-sale and banking
 scientific and research experiments, such as the Large Hadron Collider
and Atacama Large Millimeter/Submillimeter Array telescope
 sensors, such as GPS sensors, RFIDs, smart meters and telematics
 social media, such as Facebook and Twitter

31
VELOCITY

 Data can arrive at fast speeds, and enormous datasets can

accumulate within very short periods of time.
 Depending on the data source, velocity may not always be
high. For example, MRI scan images are not generated as
frequently as log entries from a high-traffic webserver.

32
VELOCITY
 Figure 1.13: Examples of
high-velocity Big Data can
easily be generated in a
given minute
 350,000 tweets,
 300 hours of video footage
uploaded to YouTube,
 171 million emails,
 330 GBs of sensor data 33

from a jet engine.

VARIETY
 Variety is the multiple formats and types of data.
 Data variety brings challenges for enterprises in terms of data
integration, transformation, processing, and storage.
 Figure 1.14 gives an example of data variety, which includes
structured data (i.e. financial transactions), semi-structured data
(i.e. emails) and unstructured data (i.e. images).

34
VERACITY
 Veracity refers to the quality or fidelity of data.
 Noise is data that cannot be converted into information and thus has no value.
 Signals are date that have value and lead to meaningful information.

 The signal-to-noise ratio

 Data with a high signal-to-noise ratio has more veracity than data with a lower
ratio.
 The signal-to-noise ratio is dependent upon the source data and data type. For
example: Data that is acquired in a controlled manner (via online customer
registrations), usually contains less noise than data acquired via uncontrolled
35

sources, such as blog postings.

VALUE

 Value is the usefulness of data for an enterprise.

 Value is intuitively impacted by
 (a) the veracity
 (b) the timeliness of generated analytic results (how long
data processing takes)
 (c) the lifecycle-related concerns
36
VALUE

 (a) the veracity

 the higher the data fidelity, the more value it holds
for the business.

37
VALUE
 (b) the timeliness
 value and time are inversely
related. The longer it takes to
turn data into meaningful
information, the less value it
has for a business.
 because analytics results
have a shelf-life; for
example, a 20 minute
delayed stock quote has little
to no value for making a
trade compared to a quote 38
that is 20 milliseconds old
VALUE

 (c) the lifecycle-related concerns

 How well has the data been stored?
 Were valuable attributes of the data removed during data cleansing?
 Are the right types of questions being asked during data analysis?
 Are the results of the analysis being accurately communicated to the
appropriate decision-makers?

39
DIFFERENT TYPES OF DATA

40
THE DATA PROCESSED BY BIG DATA SOLUTIONS CAN BE

Human-generated data Machine-generated data

 The result of human interaction with  Is generated by software programs
systems, such as online services and and hardware devices in response to
digital devices. real-world events.
 For example, a log file captures an
authorization decision made by a
security service, and a point-of-sale
system generates a transaction
against inventory to reflect items
purchased by a customer.
41
 Human-generated data Machine-generated data

 Figure 1.16: Examples of human-generated data  Figure 1.17: Examples of machine-generated

include social media, blog posts, emails, photo data include web logs, sensor data,
sharing and messaging. telemetry data, smart meter data and appliance
usage data.

42
PRIMARY TYPES OF DATA

 Both human-generated and machine-generated

data can be in various formats or types:
 structured data
 unstructured data
 semi-structured data
 metadata (another type)
43
STRUCTURED DATA
 Is a data model or schema and is often stored in tabular form.
 To capture relationships between different entities and is stored in a relational
database.
 Structured data is frequently generated by enterprise applications and
information systems like ERP and CRM systems.
 Example: banking transactions, invoices, and customer records.
 Figure 1.18: The symbol used to represent structured data stored in a tabular
form.

44
UNSTRUCTURED DATA
 Data that does not conform to a data model or data schema, and
accounts for 80% of the data within any given enterprise.
 Unstructured data has a faster growth rate than structured data.
 Form of unstructured data: textual or binary, and often conveyed via
files that are self-contained and non-relational.
 A text file may contain the contents of various tweets or blog postings.
 Binary files are often media files that contain image, audio or video data.

45
UNSTRUCTURED DATA

 Special purpose logic is usually required to process and store unstructured

data.
 For example: to play a video file, it is essential that the correct codec (coder-
decoder) is available.
 Unstructured data cannot be directly processed or queried using SQL.
 If it is required to be stored within a relational database, it is stored in a table as a
Binary Large Object (BLOB).
 Alternatively, a Not-only SQL (NoSQL) database is a non-relational database that
can be used to store unstructured data alongside structured data.
46
SEMI-STRUCTURED DATA
 Semi-structured data has a defined level of structure and consistency, but is not
relational in nature.
 It is hierarchical or graph-based, and commonly stored in files that contain
text.
 It is more easily processed than unstructured data.
 Figure 1.20: XML, JSON and sensor data are common forms of semi-
structured.

47
SEMI-STRUCTURED DATA

 Examples of common sources of semi-structured data: electronic

data interchange (EDI) files, spreadsheets, RSS feeds and sensor
data.
 Semi-structured data often has special pre-processing and storage
requirements.
 An example of pre-processing is the validation of an XML file to
ensure that it conformed to its schema definition.
48
METADATA
 Metadata provides information about a dataset’s characteristics
and structure.
 It is mostly machine-generated and can be appended to data.
 The tracking of metadata is crucial to Big Data processing, storage
and analysis because it provides information about the pedigree of
the data and its provenance during processing.
 Examples:
 XML tags providing the author and creation date of a document
49

 attributes providing the file size and resolution of a digital photograph

METADATA
 Big Data solutions rely on metadata, particularly when processing
semi-structured and unstructured data.
 Figure 1.21: The symbol used to represent metadata.

50
CASE STUDY BACKGROUND

51
CASE STUDY BACKGROUND

 Company introduction
 Company history and Company structure
 IT environment – Technical Infrastructure and Automation Environment
 Business Goals and Obstacles to adopt a data-driven IT solution
 Big Data adoption - Case Study Example

52
COMPANY INTRODUCTION

 Ensure to Insure (ETI) is a leading insurance company that provides

a range of insurance plans in the health, building, marine and
aviation sectors.
 25 million globally dispersed customer base.
 5,000 employees.
 more than 350,000,000 USD annual revenue.

53
 COMPANY HISTORY
 50 years ago: started as an exclusive health insurance provider.
 Later, ETI has extended its services to property and casualty insurance plans in
the building, marine and aviation sectors.
 Each of four sectors has a core team of specialized and experienced agents,
actuaries, underwriters and claim adjusters.

ETI’s key
departments

Customer Human
Underwriting Claims Settlement Legal Marketing Accounts IT
care resource
department department department department department department department
department department 54
Agents
• generating the company’s revenue by selling policies
Actuaries
• managing risk assessment
• designing new insurance plans and revising existing plans
• performing what-if analyses and making use of dashboards and scorecards for scenario
evaluation
Underwriters
• evaluating new insurance applications and deciding on the premium amount
Claim adjusters
• dealing with investigating claims made against a policy
• arriving at a settlement amount for the policyholder

55
COMPANY HISTORY
 Communication channels between Customer care department and prospective
and existing customers:
 telephone
 email
 social media
 Core competence:
 providing competitive policies and premium customer service that does not end once
a policy has been sold.
 helping to achieve increased levels of customer acquisition and retention.
 relying heavily on its actuaries to create insurance plans that reflect the needs of its 56
customers.
 policy
quotation
customer
IT ENVIRONMENT – relationship policy
TECHNICAL management
(CRM)
administration

INFRASTRUCTURE AND
AUTOMATION A set of client-
ENVIRONMENT enterprise server,
claims
resource mainframe management
planning (ERP) platforms and
systems

risk
billing
assessment
document
management
57
IT ENVIRONMENT – FUNCTIONS OF EACH SYSTEM
 Policy quotation system
 To create new insurance plans
 To provide quotes to prospective customers
 Is integrated with the website and customer care portal to provide website visitors and customer care
agents the ability to obtain insurance quotes
 Policy administration system
 To handle policy lifecycle management, including issuance, update, renewal and cancellation of policies

 Claims management system

 To deal with claim processing activities
 A claim is registered when a policyholder makes a report, then assigned to a claim adjuster who analyzes
the claim in light of the available information that was submitted when the claim was made, as well other
background information obtained from different internal and external sources. Based on the analyzed58
information, the claim is settled following a certain set of business rules.
IT ENVIRONMENT – FUNCTIONS OF EACH SYSTEM
 Risk assessment system
 Is used by the actuaries to assess any potential risk, such as a storm or a flood that
could result in policyholders making claims
 To run probability-based risk evaluation that involves executing various
mathematical and statistical models.
 Document management system
 A central repository for all kinds of documents, including policies, claims, scanned
documents and customer correspondence.
 Billing system
 To keep track of premium collection from customers
59
 To generate various reminders for customers who have missed their payment via
email and postal mail.
IT ENVIRONMENT – FUNCTIONS OF EACH SYSTEM
 ERP system
 Day-to-day running of ETI, including human resource management and accounts
 CRM system
 To record all aspects of customer communication via phone, email and postal mail
 To serve as a portal for call center agents for dealing with customer enquiries.
 To allow the marketing team to create, run and manage marketing campaigns.
 = => Data from these above operational systems is exported to an Enterprise
Data Warehouse (EDW)
 To generate reports for financial and performance analysis.
 To generate reports for different regulatory authorities to ensure continuous 60

regulatory compliance.
BUSINESS GOALS AND OBSTACLES
 Over the past few decades, ETI is suffering the falling share price and decrease in market share.
 A committee comprised of senior managers was formed to investigate and make recommendations.

Main reason Consequence

The existing regulations change and new regulations are

introduced very fast and frequently. But the company is slow to
Had to pay heavy fines
respond and has not been able to ensure full and continuous
compliance.

The insurance plans are created and policies are underwritten

without a thorough risk assessment Reducing the profit made on
- -> incorrect premiums being set and more payouts being investments
made than anticipated

The insurance plans are generally based on the actuaries’ Customers whose
experience and analysis of the population as a whole circumstances deviate from the 61
average set are not interested
- -> only apply to an average set of customers in such insurance plans.
Main reason Consequence

Direct monetary loss + indirect

The increased number of complex and hard-to-detect
loss (due to the costs related to
fraudulent claims and the associated payments being
the processing of fraudulent
made against them
claims)

A significant increase of natural disasters such as floods,

storms and epidemics Loss in revenue
- -> increasing the number of high-end genuine claims

Customer defection due to slow claims processing and

Loss in the number of customer
insurance products that no longer match the needs of
+ declines in revenue
customers

The emergence of tech-savvy competitors that employ Loss in the number of customer
the use of telematics to provide personalized policies + declines in revenue
62
STRATEGIC GOALS TO IMPROVE PROFITABILITY
1. Decrease losses by:
 (a) improving risk evaluation and maximizing risk mitigation, which applies to both creation of
insurance plans and when new applications are screened at the time of issuing a policy,
 (b) implementing a proactive catastrophe management system that decreases the number of potential
claims resulting from a calamity, and
 (c) detecting fraudulent claims.

2. Decrease customer defection and improve customer retention with:

 (a) speedy settlement of claims and
 (b) personalized and competitive policies based on individual circumstances rather than demographic
generalization alone.

3. Achieve and maintain full regulatory compliance at all times by employing enhanced risk 63

management techniques that can better predict risks

OBSTACLES TO ADOPT A DATA-DRIVEN IT SOLUTION
 Acquiring, storing and processing unstructured data from internal and external
data sources – Currently, only structured data is stored and processed
 Processing large amounts of data in a timely manner – The amount of data
processed cannot be classified as large, and the reports take a long time to
generate.
 Processing multiple types of data and combining structured data with
unstructured data – Unstructured data such as documents and call center logs
that cannot currently be processed, while structured data is used in isolation for
all types of analyses.

64
= = > a recommendation that ETI should adopt Big Data
BIG DATA ADOPTION - CASE STUDY EXAMPLE
1. IT team and skills for Big Data implementation
 Problems
 No in-house Big Data skills
 Have to choose between hiring a Big Data consultant or sending its IT team on a Big Data training course.
 Solutions
 Sending only the senior IT team members to the Big Data training course.
 For long-term plan, this trained team members will become a permanent in-house Big Data resource and can also
train junior team members to further increase the in-house Big Data skillset.
2. During Big Data training course
 Problems
 No common vocabulary of terms
 Lack of business exposure and understanding BI and the establishment of appropriate KPIs
 Solutions
 Building a terms glossary for datasets including claims, policies, quotes, customer profile data and census data. 65

 Explaining BI by using the monthly report generation process for evaluating the previous month’s performance as
an example
 BIG DATA ADOPTION - CASE STUDY EXAMPLE
3. Data Analytics
 Deciding to use of both descriptive and diagnostic analytics
 Descriptive analytics is for:
 querying the policy administration system to determine the number of polices sold each day
 querying the claims management system to find out how many claims are submitted daily
 querying the billing system to find out how many customers are behind on their premium payments.
 Diagnostic analytics is for
 various BI activities, such as performing queries to answer questions such as why last month’s sales target was not met.
 performing drill-down operations to breakdown sales by type and location so that it can be determined which locations
underperformed for specific types of policies.
 In the future, utilizing predictive and prescriptive analytics in a gradual manner by first implementing predictive analytics
and then slowly building up their capabilities to implement prescriptive analytics.
 predictive analytics will enable detection of fraudulent claims by predicting which claim is a fraudulent one and in case
of customer defection by predicting which customers are likely to defect. 66

 later, via prescriptive analytics, prescribing the correct premium amount considering all risk factors or prescribing the
best course of action to take for mitigating claims when faced with catastrophes, such as floods or storms.
BIG DATA ADOPTION - CASE STUDY EXAMPLE
4. Identifying Data Characteristics
 Volume
 A large amount of transactional data is generated as a result of processing claims, selling
new policies and changes to existing policies.
 A large volumes of unstructured data, both inside and outside the company, including
health records, documents submitted by the customers at the time of submitting an
insurance application, property schedules, fleet data, social media data and weather data.
 Velocity
 For in-flow data, some is low velocity (such as the claims submission data and the new
policies issued data), some is high (such as webserver logs and insurance quotes).
 For out-flow data, social media data and the weather data may arrive at a fast pace.
 For catastrophe management and fraudulent claim detection, data needs to be processed 67

quickly to minimize losses.

BIG DATA ADOPTION - CASE STUDY EXAMPLE
4. Identifying Data Characteristics
 Variety
 Have to incorporate a range of datasets that include health records, policy data, claim data,
quote data, social media data, call center agent notes, claim adjuster notes, incident
photographs, weather reports, census data, webserver logs and emails.
 Veracity
 Inside ETI’s boundary, data has high veracity thanks to data validation performed at multiple
stages such as data entry, function-level input validation, data persistence.
 Outside ETI’s boundary, data has low veracity (such as social media data and weather data)
that requires an increased level of data validation and cleansing
 Value
68

 Have to draw maximum value out of the available datasets by ensuring the datasets are
stored in their original form and that they are subjected to the right type of analytics.
BIG DATA ADOPTION - CASE STUDY EXAMPLE
 5. Identifying Types of Data
 Structured data: policy data, claim data, customer profile data and quote data.
 Unstructured data: social media data, insurance application documents, call center agent
notes, claim adjuster notes and incident photographs.
 Semi-structured data: health records, customer profile data, weather reports, census data,
webserver logs and emails.
 Metadata is a new concept as ETI’s current data management procedures do not create nor
append any metadata.
 Why? - -> Because all data in ETI is stored and processed is structured in nature and originates
from within the company. Hence, the origins and the characteristics of data are implicitly known.
 Solution - -> for the structured data, the data dictionary and the existence of last updated
timestamp and last updated user-id columns within the different relational database tables can be 69
used as metadata.
THANK YOU

70