Thanks to visit codestin.com
Credit goes to www.scribd.com

Scribd
Upload
5 views1 page

Flafka Consumer

The document outlines the configuration for an Apache Flume agent named 'flume1', detailing its sources, channels, and sinks. It specifies a Kafka source connected to a topic, a memory channel, and an HDFS sink with various properties for data handling. Additionally, it includes settings for channel capacity and transaction limits.

Uploaded by

santymusic59
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
5 views1 page

Flafka Consumer

The document outlines the configuration for an Apache Flume agent named 'flume1', detailing its sources, channels, and sinks. It specifies a Kafka source connected to a topic, a memory channel, and an HDFS sink with various properties for data handling. Additionally, it includes settings for channel capacity and transaction limits.

Uploaded by

santymusic59
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 1

# Sources, channels, and sinks are defined per

# agent name, in this case flume1.


flume1.sources = kafka-source-1
flume1.channels = hdfs-channel-1
flume1.sinks = hdfs-sink-1

# For each source, channel, and sink, set


# standard properties.
flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource
flume1.sources.kafka-source-1.zookeeperConnect = localhost:5181
flume1.sources.kafka-source-1.topic = flafkatopic
flume1.sources.kafka-source-1.batchSize = 100
flume1.sources.kafka-source-1.channels = hdfs-channel-1

flume1.channels.hdfs-channel-1.type = memory
flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = test-events
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = /tmp/kafka/%{topic}/%y-%m-%d
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0

# Other properties are specific to each type of


# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.
flume1.channels.hdfs-channel-1.capacity = 10000
flume1.channels.hdfs-channel-1.transactionCapacity = 1000

You might also like