I'm running this pubsub connector from master as of opening this ticket, within a docker container based on confluentinc/cp-kafka-connect:3.3.0-1, which features kafka 0.11.0
Eventually, while piping messages from kafka to pubsub, the task will run into an HTTP/2 error and never be able to recover, repeating the error message below. This results in an increasing backlog, and a constantly repeated batch of messages.
The only thing that comes to mind with this is some sort of network address cache, causing the task to try and connect to a pubsub node which has been shut down. I found that the java included in the confluent container does not define a value for java's networkaddress.cache.ttl setting, so it defaults to forever. I then changed the value to 30 seconds (by forking the container to change the file itself) and restarted my workers, but the problem came back after a few hours.
# java -version
openjdk version "1.8.0_144"
OpenJDK Runtime Environment (Zulu 8.23.0.3-linux64) (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (Zulu 8.23.0.3-linux64) (build 25.144-b01, mixed mode)
[2017-09-13 05:17:57,552] WARN Commit of WorkerSinkTask{id=athena-staging-pubsub-1} offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask)
[2017-09-13 05:17:57,839] ERROR Commit of WorkerSinkTask{id=athena-staging-pubsub-1} offsets threw an unexpected exception: (org.apache.kafka.connect.runtime.WorkerSinkTask)
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing offsets. The underlying error was: The request timed out.
[2017-09-13 06:47:52,831] ERROR WorkerSinkTask{id=athena-staging-pubsub-1} Offset commit failed, rewinding to last committed offsets (org.apache.kafka.connect.runtime.WorkerSinkTask)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
session_timed_out
at com.google.pubsub.kafka.sink.CloudPubSubSinkTask.flush(CloudPubSubSinkTask.java:293)
at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:117)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:305)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:164)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:148)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
session_timed_out
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:459)
at com.google.pubsub.kafka.sink.CloudPubSubSinkTask.flush(CloudPubSubSinkTask.java:290)
... 11 more
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
session_timed_out
at io.grpc.Status.asRuntimeException(Status.java:545)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:417)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:458)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$500(ClientCallImpl.java:385)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$3.runInContext(ClientCallImpl.java:486)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:154)
... 3 more
I'm running this pubsub connector from
masteras of opening this ticket, within a docker container based onconfluentinc/cp-kafka-connect:3.3.0-1, which features kafka 0.11.0Eventually, while piping messages from kafka to pubsub, the task will run into an HTTP/2 error and never be able to recover, repeating the error message below. This results in an increasing backlog, and a constantly repeated batch of messages.
The only thing that comes to mind with this is some sort of network address cache, causing the task to try and connect to a pubsub node which has been shut down. I found that the java included in the confluent container does not define a value for java's
networkaddress.cache.ttlsetting, so it defaults to forever. I then changed the value to 30 seconds (by forking the container to change the file itself) and restarted my workers, but the problem came back after a few hours.