Home Jobs Tools Coding Ground Current Affairs UPSC Notes Online Tutors Whiteboard Net Meeting Tutorix

ard Net Meeting Tutorix Login Premium

Categories Library Videos Q/A eBooks

Apache Kafka - Introduction

Advertisements

Previous Page Next Page

Apache Kafka Tutorial In Big Data, an enormous volume of data is used. Regarding data, we have two main challenges.The
first challenge is how to collect large volume of data and the second challenge is to analyze the
Apache Kafka - Home collected data. To overcome those challenges, you must need a messaging system.
Apache Kafka - Introduction Kafka is designed for distributed high throughput systems. Kafka tends to work very well as a
Apache Kafka - Fundamentals replacement for a more traditional message broker. In comparison to other messaging systems, Kafka
has better throughput, built-in partitioning, replication and inherent fault-tolerance, which makes it a
Apache Kafka - Cluster Architecture good fit for large-scale message processing applications.
Apache Kafka - Work Flow

Apache Kafka - Installation Steps What is a Messaging System?

Apache Kafka - Basic Operations A Messaging System is responsible for transferring data from one application to another, so the
applications can focus on data, but not worry about how to share it. Distributed messaging is based on
Simple Producer Example
the concept of reliable message queuing. Messages are queued asynchronously between client
Consumer Group Example applications and messaging system. Two types of messaging patterns are available − one is point to
Integration With Storm point and the other is publish-subscribe (pub-sub) messaging system. Most of the messaging patterns
follow pub-sub.
Integration With Spark

Real Time Application(Twitter) Point to Point Messaging System

Apache Kafka - Tools In a point-to-point system, messages are persisted in a queue. One or more consumers can consume
Apache Kafka - Applications the messages in the queue, but a particular message can be consumed by a maximum of one
consumer only. Once a consumer reads a message in the queue, it disappears from that queue. The
Apache Kafka Useful Resources typical example of this system is an Order Processing System, where each order will be processed by
one Order Processor, but Multiple Order Processors can work as well at the same time. The following
Apache Kafka - Quick Guide diagram depicts the structure.

Apache Kafka - Useful Resources

Apache Kafka - Discussion

Selected Reading

UPSC IAS Exams Notes

Developer's Best Practices

Questions and Answers

Effective Resume Writing

HR Interview Questions

Computer Glossary Publish-Subscribe Messaging System

Who is Who In the publish-subscribe system, messages are persisted in a topic. Unlike point-to-point system,
consumers can subscribe to one or more topic and consume all the messages in that topic. In the
Publish-Subscribe system, message producers are called publishers and message consumers are
called subscribers. A real-life example is Dish TV, which publishes different channels like sports,
movies, music, etc., and anyone can subscribe to their own set of channels and get them whenever their
subscribed channels are available.

What is Kafka?
Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle
a high volume of data and enables you to pass messages from one end-point to another. Kafka is
suitable for both offline and online message consumption. Kafka messages are persisted on the disk
and replicated within the cluster to prevent data loss. Kafka is built on top of the ZooKeeper
synchronization service. It integrates very well with Apache Storm and Spark for real-time streaming
data analysis.

Benefits
Following are a few benefits of Kafka −
Reliability − Kafka is distributed, partitioned, replicated and fault tolerance.

Scalability − Kafka messaging system scales easily without down time..

Durability − Kafka uses "Distributed commit log" which means messages persists on disk as
fast as possible, hence it is durable..
Performance − Kafka has high throughput for both publishing and subscribing messages. It
maintains stable performance even many TB of messages are stored.

Kafka is very fast and guarantees zero downtime and zero data loss.

Use Cases
Kafka can be used in many Use Cases. Some of them are listed below −
Metrics − Kafka is often used for operational monitoring data. This involves aggregating
statistics from distributed applications to produce centralized feeds of operational data.
Log Aggregation Solution − Kafka can be used across an organization to collect logs from
multiple services and make them available in a standard format to multiple con-sumers.
Stream Processing − Popular frameworks such as Storm and Spark Streaming read data
from a topic, processes it, and write processed data to a new topic where it becomes
 available for users and applications. Kafka’s strong durability is also very useful in the context
of stream processing.

Need for Kafka

Kafka is a unified platform for handling all the real-time data feeds. Kafka supports low latency
message delivery and gives guarantee for fault tolerance in the presence of machine failures. It has the
ability to handle a large number of diverse consumers. Kafka is very fast, performs 2 million writes/sec.
Kafka persists all data to the disk, which essentially means that all the writes go to the page cache of
the OS (RAM). This makes it very efficient to transfer data from page cache to a network socket.

Previous Page Print Page Next Page

Advertisements

About us Terms of use Privacy Policy FAQ's Teach with Us Contact

© Copyright 2020. All Rights Reserved.