Putting Apache Kafka
Building a Real-time Data Platform for Event Streams!
to Use!
JAY KREPS, CONFLUENT!
A Couple of Themes!
Theme 1: Rise of Events!
Theme 2: Immutability Everywhere!
Level! Example! Immutable Alternative!
Mutable local state! Counter in a for loop! Functional Programming!
Mutable process-wide state! ConcurrentHashMap! Functional Programming!
!
Mutable on disk structures! B-Tree! LSM!
Distributed systems! Dynamo-like key-value store! State machine replication!
Mutability in databases! RDBMS! Event Sourcing!
Company-wide data flow! Double write! Kafka!
Theme 3: Datacenter-Level Thinking!
Experience at LinkedIn!
2009: We want all our data in Hadoop!!
What is all our data?!
Initial approach: “gut it out”!
Problems!
• Data coverage!
• Many source systems!
• Relational DBs!
• Log files!
• Metrics!
• Messaging systems!
• Many data formats!
• Constant change!
• New schemas!
• New data sources!
Needed: organizational scalability!
Θ(N) => Θ(1)!
How does everything else work?!
?!
Relational database changes!
Apps and Services
OLTP Queries
Relational
Databases
Data Guard CSV Dump
Cache
ODS Hadoop
Poll For Changes
App App App
Relational Transforms
Data
Caches & Warehouse
Derived Stores
Transforms
NoSQL!
App App App
Key-value
Store
ETL Load
Hadoop
User events!
Apps and Apps and Apps and
Services Services Services
HTTP
Log Aggregation
NFS
rsync
NFS
Load Transform & Load
Relational
Hadoop Data
Warehouse
Transform
Application Logs!
Apps and Apps and Apps and
Services Services Services
Splunk
Messaging!
App App App App App
Broker Broker
Processor Processor Processor Processor
App App App
Broker
Processor Processor Processor Processor
Metrics and operational data!
App App App
Monitoring
This is a giant mess!
Apps and Services Apps and Services Apps and Services
OLTP Queries
HTTP
ActiveMQ HTTP
Monitoring
Relational Apps Apps Log Aggregation
Databases
Splunk
Key-value
Store
Data Guard NFS
CSV Dump
ActiveMQ Cache
rsync
Poll For Changes ODS Hadoop Load NFS
Apps Apps
App App App
Relational Transforms
Data
Transform & Load
Caches & Warehouse
Derived Stores
Transforms
Impossible ideas!
• Publish data from Hadoop to a search index!
• Run a SQL query to find the biggest latency
bottleneck!
• Run a SQL query to find common error patterns!
• Low latency monitoring of database changes or user
activity!
• Incorporate popularity in real-time display and
relevance algorithms!
• Products that incorporate user activity!
An infrastructure solution?!
Idea: Stream Data Platform!
Search Impala
Apps Hive
Monitoring
Stream
Data HADOOP:
DWH
RDBMS Platform: Offline
? Data
Stream Map-
NoSQL Processing
Reduce
Real-time
Analytics Spark
Synchronous
Req/Response Near real time
Offline batch
0 - 100s ms > 100s ms > 1 hour
First Attempt: Messaging systems!!
Problems!
• Throughput!
• Batch systems!
• Persistence!
• Stream Processing!
• Ordering
guarantees!
• Partitioning!
Second Attempt: Build Kafka!!
What does it do?!
Producer Producer Producer Producer Producer
Kafka Cluster
Consumer Consumer Consumer Consumer Consumer
Commit Log Abstraction!
Reader 1 Reader 2
1 1 1 Writes
0 1 2 3 4 5 6 7 8 9
0 1 2
Old New
Logs & Publish-Subscribe Messaging!
Source
System
writes
1 1 1
Log 0 1 2 3 4 5 6 7 8 9 0 1 2
reads reads
Destination Destination
System A System B
A Kafka Topic!
Partition 1 1 1
0 0 1 2 3 4 5 6 7 8 9
0 1 2
Partition Writes
0 1 2 3 4 5 6 7 8 9
1
Partition 1 1 1
0 1 2 3 4 5 6 7 8 9
2 0 1 2
Old New
Replication!
Server 1 Server 2 Server 3
A:0 A:0 A:0
A:1 A:1 A:1
B:0 B:0 Controller
Scaling Consumers!
Kafka Cluster
Server 1 Server 2
P0 P3 P1 P2
C1 C2 C3 C4 C5 C6
Consumer Group A Consumer Group B
Kafka: A Modern Distributed System for Streams!
Scalability of a filesystem!
◦ Hundreds of MB/sec/server throughput!
◦ Many TB per server!
Guarantees of a database!
◦ Messages strictly ordered!
◦ All data persistent!
Distributed by default!
◦ Replication!
◦ Partitioning model!
Producers, Consumers, and Brokers all fault tolerant and horizontally
scalable!
Stream Data Platform!
Search Impala
Apps Hive
Monitoring
KAFKA:
Stream HADOOP:
DWH
RDBMS Data Offline
Platform Data
Stream Map-
NoSQL Processing
Reduce
Real-time
Analytics Spark
Synchronous
Req/Response Near real time
Offline batch
0 - 100s ms > 100s ms > 1 hour
Batch Data => Batch Processing!
Stream processing is a!
generalization!
of batch processing !
and request/response processing!
Request/Response processing: !
One input => One output!
Batch processing: !
All inputs => All outputs!
Stream Processing: !
Some inputs => some outputs!
(you choose how much “some” is)!
Stream Processing a la carte!
Input Kafka Topic
Transform Transform Transform
Intermediate Your code
Kafka Topic
cat input | grep “foo” | wc -l
Transform Transform Transform
Output Kafka
Topic
Hadoop Live
Data Store
Stream Processing with Frameworks!
+! =! Stream
Processing!
Unix Pipes, Modernized!
cat /usr/share/dict/words | wc -l
On Schemas!
Bad Schemas < No Schemas < Good Schemas!
Put it all together!
Apps Apps Apps Apps
Social Key-Value
Search Oracle Newsfeed OLAP
Graph Storage
Apps
Log
Search Apps
Monitoring
Kafka
Security &
Fraud Samza
Real-time
Analytics
Hadoop Teradata
At LinkedIn!
• Everything in the company is a real-time stream!
• > 800 billion messages written per day!
• > 2.9 trillion messages read per day!
• ~ 1 PB of stream data!
• Tens of thousands of producer processes!
• Backbone for data stores!
• Search!
• Social Graph!
• Newsfeed!
• Primary storage (in progress)!
• Basis for stream processing!
Elsewhere!
Why this is the future!
1. System diversity is increasing!
2. Data diversity and volume is
increasing!
3. The world is getting faster!
4. The technology exists!
• Mission: Make this a practical reality
everywhere!
• Product!
• Apache Kafka!
• Schemas and metadata management!
• Connectors for common systems!
• Monitor data flow end-to-end!
• Stream processing integration!
Questions?!
• Confluent!
• @confluentinc!
• http://confluent.io !
• http://blog.confluent.io/2015/02/25/
stream-data-platform-1 !
• Apache Kafka!
• @apachekafka!
• http://kafka.apache.org!
• http://linkd.in/199iMwY !
• Me!
• @jaykreps!
