added KafkaV2 sink using new kafka producer in 0.8.2-SNAPSHOT #127
Conversation
|
suro-pull-requests #95 FAILURE |
|
licenseTest was timed out. This PR doesn't have any unit testing and it should be fine. |
There was a problem hiding this comment.
There is ProducerRecord constructor with key argument. SuroKeyedMessage is for semantic partitioning and if you discard the key information, semantic partitioning logic will be broken.
There was a problem hiding this comment.
In the new producer, partitioning logic with the key is murmur32 hashing but the old one's hash partitioning logic is key.hashCode(). So, we need to create ProducerRecord with the partitioning key as keyedMessage.getKey().hashCode() % (number of partition) for the backward compatibility. But how can we get a number of partition in advance?
There was a problem hiding this comment.
OK, there's a method in the producer
@Override
public List<PartitionInfo> partitionsFor(String topic) {
waitOnMetadata(topic, this.metadataFetchTimeoutMs);
return this.metadata.fetch().partitionsForTopic(topic);
}
|
@metacret , we also noticed that the stats published at http://localhost:7103/surosinkstat are broken. I'll send another pull request when these two issues are fixed. |
There was a problem hiding this comment.
Regarding this one, when I see from kafka.produer.BaseProducer the following code snippet:
override def send(topic: String, key: Array[Byte], value: Array[Byte]) {
val record = new ProducerRecord(topic, key, value)
if(sync) {
this.producer.send(record).get()
} else {
this.producer.send(record,
new ErrorLoggingCallback(topic, key, value, false))
}
}
According to ProducerPerformance testing tool, I don't see much difference between sync mode and async mode of new producer, but could you check the performance again?
Also, ThreadPoolQueuedSink is for the slow entry point client such as ElasticSearch or Kafka old producer but this new shiny producer looks awesome, which means we might not need a ThreadPoolQueuedSink but QueuedSink. What do you think?
There was a problem hiding this comment.
The new producer does not have a synchronous mode (see the Javadoc for KafkaProducer#send: https://apache.googlesource.com/kafka/+/trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java), so I don't know what else I can test.
Regarding the ThreadPoolQueuedSink, it may be a good idea to keep the thread pool since any one thread may get blocked while waiting for the KafkaProducer.send() request to finish. We'd want the ability to start sending another request simultaneously because the next message may be going to a different kafka broker, right? But actually the Kafka producer has its own thread pool and message buffer, so that may make all the requests fast on the Sink side. I don't have a confident answer to the ThreadPoolQueueSink question.
|
Even though KafkaSinkV2 implementation is trivial, could you write the unit test? |
|
suro-pull-requests #96 FAILURE |
|
suro-pull-requests #97 FAILURE |
|
@starzia Thanks a lot for your new commits. We're almost done! I left a few comments and could you fix them? Also, PR is failing due to licenseTest timeout, I asked Netflix tools team and it will be fixed soon. Do not care about PR failure too much for a while. |
|
@starzia Netflix tools team told me the timeout seems happening on the test. Could you check on your local repository whether all tests are fine? |
|
I'll be able to check on Monday. Thanks.
|
|
@starzia May I ask you one more fix? Even though I didn't see much difference between sync mode and async mode of new producer, we'd better to have both of them as optional. In your PR, you're calling get() method for every Future returned by send() method. This will block the operation in the Runnable. So that's why we would need to remove ThreadPool and let KafkaProducer do the non-blocking IO. What do you think? |
|
@metacret, the tests are working fine here. My latest commit fixes the partitioning backward compatibility and key storage issue. I forgot to look at the async mode. I will look at that now. |
|
suro-pull-requests #100 FAILURE |
|
@metacret I think it's better to force the Kafka producer to use synchronous mode because then the user does not have to worry about another queue (the internal Kafka queue). This would be a fourth queue (we already have the thrift input buffer, the Message Set Processor queue, and the sink queue). Do you have a use case in mind for the async mode? One thing we may want to add is an option to disable requeueing of failed messages in KafkaSinkV2. This can cause an infinite buildup of messages if Kafka is down (which is exactly what I want in my use case). Let me know what you think. Also, about the tests, they took 13 minutes on my machine so maybe that is why your tools team is seeing a timeout? |
|
suro-pull-requests #101 FAILURE |
|
OK, regarding asynchronous one, I agree with you. I will take a look at why this PR keeps failing and merge it. |
|
suro-pull-requests #102 FAILURE |
|
suro-pull-requests #103 FAILURE |
|
@starzia Could you merge trunk and update this PR? I mistakenly merged one PR which contains the test which keeps failing. |
|
@metacret done. |
|
suro-pull-requests #105 SUCCESS |
added KafkaV2 sink using new kafka producer in 0.8.2-SNAPSHOT
@metacret here is the pull request that we discussed. Please see suro-server/conf/sink_using_kafkaV2.json for the configuration syntax.