Introduction to
Data Engineering
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
A data engineer builds data pipelines to enable
data-driven decisions
Get the data Get the data
Add new value
to where it can into a usable
to the data
be useful condition
So… how do we
get the raw data Manage the Productionize
from multiple
systems and data data processes
where can be
store it durably?
A data lake brings together data from across the
enterprise into a single location
Replicate
Raw Data
Data Lake
Spread
RDMBS sheets
Other
Offline systems
files and apps
Key considerations when building a Data Lake
1. Can your data lake handle all the
types of data you have?
Replicate
2. Can it scale to meet the
demand? Data Lake
3. Does it support high-throughput
ingestion?
We need an elastic data
4. Is there fine-grained access container that is flexible
control to objects? and durable to stage all
our data …
5. Can other tools connect easily?
Cloud Storage is designed for 99.999999999% annual durability
Replace/decommission Content storage and
Backup Analytics and ML
infrastructure delivery
Quickly create buckets with cloud shell
gsutil mb gs://your-project-name
What if your data is not usable in its original form?
SOME ETL Data Processing
ASSEMBLY
REQUIRED
Cloud Dataproc Cloud Dataflow
Extract, Transform,
and Load
What if your data arrives continuously and endlessly?
THIS DATA
Streaming Data
DOES NOT
WAIT
Processing
Cloud Cloud
Dataflow BigQuery
Pub/Sub
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
Common challenges encountered by data engineers
Access to data Data accuracy Availability of Query
and quality computational performance
resources
Challenge: Consolidating disparate datasets, data
formats, and manage access at scale
Access to data Data accuracy Availability of Query
and quality computational performance
resources
Getting insights across multiple datasets is difficult
without a data lake
Data is scattered across
No common tool exists to
Google Analytics 360,
analyze data and share
CRM, and Campaign
results with the rest of
Manager products,
the organization.
among other sources.
Customer and sales
Some data is not in a
data is stored in
queryable format.
a CRM system.
Data is often siloed in many upstream source systems
Example Query:
Give me all the
in-store promotions
for recent orders and
their inventory levels
Stored in a separate system
and restricted access
Challenge: Cleaning, formatting, and getting the data
ready for useful business insights in a data warehouse
Access to data Data accuracy Availability of Query
and quality computational performance
resources
Assume that any raw data from source systems needs to be
cleaned and transformed and stored in a data warehouse
Query: Give me the best
performing in-store
Missing data and all promotions in France
timestamps
stored as UTC
Promotion list stored
as .csv files or manual
spreadsheets
Challenge: Ensuring you have the compute capacity
to meet peak-demand for your team
Access to data Data accuracy Availability of Query
and quality computational performance
resources
Challenge: Data Engineers need to manage server and cluster
capacity if using on-premise
Under-provisioned
Resources (Wasting time) On-Premises
Compute Capacity
Under-utilized
Consumption (Wasting $$$)
Capacity
Time
Challenge: Queries need to be optimized for
performance (caching, parallel execution)
Access to data Data accuracy Availability of Query
and quality computational performance
resources
Challenge: Managing query performance
on-premise comes with added overhead
● Choosing a Query Engine
● Continually patching and Is there a better way to
updating query engine software manage server
overhead so we can
● Managing clusters and when to focus on insights?
re-cluster
● Optimize for concurrent queries
and quota / demand between
teams
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
BigQuery is Google’s data warehouse solution
Data Tables and
Data mart Data lake Grants
warehouse views
BigQuery replaces BigQuery BigQuery defines Function the same Cloud IAM grants
a typical data organizes data schemas and way as in a permission to
warehouse tables into units issues queries traditional data perform specific
hardware setup called datasets directly on warehouse actions
external data
sources
Cloud allows data engineers to spend less time managing
hardware and enabling scale; Let Google do that for you
Typical Big Data Processing With Google
Monitoring Insights Insights
Performance Resource
tuning provisioning
Utilization Handling
improvements growing scale
Deployment &
Reliability
configuration
You don't need to provision resources before using BigQuery
Resources On-demand storage
and compute
Consumption
Allocation
Time
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
A data engineer gets data into a useable condition
Get the data Get the data
Add new value
to where it can into a usable
to the data
be useful condition
Manage the Productionize
data data processes
A data warehouse stores transformed data in a
usable condition for business insights
Replicate Extract, Transform, Load
Pipeline
Raw Data Data
Data Lake
Warehouse
What are the key
considerations when
deciding between data
warehouse options?
Considerations when choosing a data warehouse
include:
● Can it serve as a sink for both batch
and streaming data pipelines?
Extract, Transform, Load
● Can the data warehouse scale to meet Pipeline
Data
my needs? Warehouse
● How is the data organized, cataloged,
and access controlled?
● Is the warehouse designed for
performance?
● What level of maintenance is required
by our engineering team?
BigQuery is a modern data warehouse that changes
the conventional mode of data warehousing
Complex Restricted Optimized Optimized Needs continuous Needs an army of
Traditional ETL to only a for legacy for batch patching DBAs for operational
DW few users BI data and updates tasks
Automate Make Build the Tee up Fully Simplify
data insights foundation real-time managed data
BigQuery delivery accessible for AI insights operations
You can simplify Data Warehouse ETL pipelines with external
connections to Cloud Storage and Cloud SQL
● Postgres Federate
d Query
Cloud ● MySQL
SQL ● SQL Server
Cloud
Storage
Demo Federated Queries with
BigQuery
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data
Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
Cloud SQL is fully managed SQL Server, Postgres, or MySQL
for your Relational Database (transactional RDBMS)
● Automatic encryption
Cloud
SQL ● 30TB storage capacity
● 60,000 IOPS
(read/write per second)
● Auto-scale and auto
backup
Why not simply use Cloud
SQL for reporting workflows?
RDBMS are optimized for data from a single source and
high-throughput writes vs high-read data warehouses
You will likely need and
encounter both a database and
data warehouse in your final
architecture
Cloud BigQuery
SQL
● Scales to GB and TB ● Scales to PB
● Ideal for back-end ● Easily connect to
database applications external data sources
● Record based storage for ingestion
● Column based storage
Relational database management systems (RDBMS)
are critical for managing new transactions
RDBMS are optimized for
high throughput WRITES
to RECORDS
The complete picture: Source data comes into the data lake, is
processed into the data warehouse and made available for insights
line ML Model
pip e
re
Featu
Data
Data Lake Other Team Data
Warehouse Eng Pipeline
Warehouse
BI p
ipe
line
Reporting
Dashboards
Who leads these other
teams that we will have to
partner with?
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
A data engineer builds data pipelines to enable
data-driven decisions What teams rely on
these pipelines?
Get the data Get the data
Add new value
to where it can into a usable
to the data
be useful condition
Manage the Productionize
data data processes
Many teams rely on partnerships with data
engineering to get value out of their data
Machine Learning Data Analyst Data Engineer
Engineer
How might each of these teams rely on data engineering?
Machine learning teams need data engineers to help
them capture new features in a stable pipeline
fe at ures ?”
h e s e t i m e
a l l oft u c t ion
“Are t p ro d
bl e a
availa
eline ML Model
re pi p
Raw Data Data Featu
Data Lake
Warehouse
“Can you help us get more
features (columns) of data for
our machine learning model?”
Add value: Machine learning directly in BigQuery
FROM
ML.EVALUATE(MODEL
`bqml_tutorial.sample_model`,
TABLE eval_table)
1 Dataset 2 Create/train 3 Evaluate 4 Predict/classify
CREATE MODEL `bqml_tutorial.sample_model` FROM
OPTIONS(model_type='logistic_reg') AS ML.PREDICT(MODEL
SELECT `bqml_tutorial.sample_model`,
table game_to_predict) )
AS predict
Data analysis and business intelligence teams rely on
data engineering to showcase the latest insights
in yo ur
va i la ble ? ”
i s a ce s s
hat data us to ac
“W us e for
o
wareh
Reporting
p eline Dashboards
B I pi
Raw Data Data
Data Lake
Warehouse
“Our dashboards are slow, can you
help us re-engineer our BI tables
for better performance?”
Add value: BI Engine for dashboard performance
Data
BigQuery BI
Studio
Engine
Batch or
Streaming Sheets
BigQuery
Partner
● No need to manage OLAP cubes
BI tools or separate BI servers for
dashboard performance
● Natively integrates with
BigQuery streaming for real-time
data refresh
● Column oriented in-memory BI
execution engine
Other data engineering teams may rely on your
pipelines being timely and error free
“How can we both ensure
dataset uptime and
performance?”
Data Pipeline Other Team Data
Raw Data Data Lake
Warehouse Warehouse
“We’re noticing high demand for your
datasets -- be sure your warehouse
can scale for many users”
Add value: Cloud Monitoring for performance
● View in-flight and completed queries.
● Track spending on BigQuery resources.
● Use Cloud Audit Logs to view actual job
information (who executed, what query
was ran).
● Create alerts and send notifications.
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
A data engineer manages data access and governance
Get the data Get the data
Add new value
to where it can into a usable
to the data
be useful condition
Manage the Productionize
data data processes
Data engineering must set and communicate a
responsible data governance model
line ML Model
pip e
re
Featu
Raw Data Data
Data Lake Other Team Data
Warehouse Eng Pipeline
Warehouse
BI p
ipe
line
● Who should have access? Reporting
● How is PII handled? Dashboards
● How can we educate end-users on
our data catalog?
Cloud Data Catalog is a managed data discovery +
Data Loss Prevention API for guarding PII
Data Catalog
Simplify data discovery at any scale:
Fully managed metadata management service with
no infrastructure to set up or manage
Unified view of all datasets:
Central and secure data catalog across Google
Cloud with metadata capture and tagging
Data governance foundation:
Security compliance with access level controls along
with Cloud Data Loss Prevention integration for
handling sensitive data
Demo Finding PII in your dataset with
DLP API
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
A data engineer builds production data pipelines to
enable data-driven decisions
Get the data Get the data
Add new value
to where it can into a usable
to the data
be useful condition
Manage the Productionize
data data processes
Data engineering owns the health and future of their
production data pipelines
line ML Model
pip e
re
Featu
Raw Data Data
Data Lake Other Team Data
Warehouse Eng Pipeline
Warehouse
BI p
ipe
line
● How can we ensure pipeline health and data Reporting
cleanliness? Dashboards
● How do we productionalize these pipelines to
minimize maintenance and maximize uptime?
● How do we respond and adapt to changing
schemas and business needs?
● Are we using the latest data engineering tools and
best practices?
Cloud Composer (managed Apache Airflow) is used
to orchestrate production workflows
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
Ocado’s customer service department is
bombarded with messages
Can we use ML to prioritize these messages?
“My order is missing 12
“I love Ocado! You’re the
steaks. I need them for a
best.”
party I’m hosting in 6 hours.”
“I scheduled my delivery for 3
“My delivery is five hours late. PM today. Something came
Can I get a full refund?” up and I won’t be home at
that time. Can I reschedule?”
Ocado’s GCP solution helps them respond to urgent customer
emails 4x faster with ML
Increased contact center efficiency enables representatives to spend extra time on high-priority tasks
AI Platform
http://www.multichannel-blog.co.uk/2017/05/03/google-the-future-of-cloud-conference-in-london-3-4th-may/
Twitter democratized data analysis using BigQuery
“We believe that users with a wide range of technical skills should be able to discover
data and have access to SQL-based analysis and visualization tools that perform well”
-- Twitter
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/democratizing-data-analysis-with-google-bigquery.html
Recap
● Data sources
● Data lakes
● Data warehouses
● Google Cloud solutions for
Data Engineering
Concept Review:
Data sources feed
into a Data Lake and AI
are processed into
Platform
Notebooks
your Data Warehouse AI
Platform
for analysis
Data stores
AI
Platform
Notebooks
Here’s a useful guide
for “GCP products in
4 words or less”
https://github.com/gr
egsramblings/google-
cloud-4-words
Updated continually By Greg Wilson -
Google DevRel
Agenda
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Transactional Databases vs Data Warehouses
Partner effectively with other data teams
● Manage data access and governance
● Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
Lab
Using BigQuery to do
Analysis
Objectives
● Execute interactive queries in the BigQuery console
● Combine and run analytics on multiple datasets
