Performance tests in pub/sub show that the publisher is using 50-100x more CPU than subscriber #3149
Description
I've been running performance tests with Pub/Sub trying to find the right combination of threads, memory, buffer sizes, etc.. I'm seeing that the Publisher taking ~500% CPU versus 5-8% for the Subscriber running on the same server, simultaneously handling the same message throughput. Does that surprise anyone? In looking at the running threads I see that a number of them have the following stack trace in the RSA/SSHA256 JWT signing code. I suspect that this is to a large degree the real limitation of the Publisher's performance.
Anyone else seeing this? Any recommendations on how to work around this? Is there something I can do to the transport layer turn off these or reduce their usage? Maybe GPRC configuration tweaks?
Thanks. This is using v0.38 on a Linux box running Java 7 outside of google's cloud. Would running on Google's computer engine instances?
BigInteger.oddModPow(BigInteger.java:2716)
java.math.BigInteger.modPow(BigInteger.java:2459)
sun.security.rsa.RSACore.crtCrypt(RSACore.java:183)
sun.security.rsa.RSACore.rsa(RSACore.java:122)
sun.security.rsa.RSASignature.engineSign(RSASignature.java:175)
java.security.Signature$Delegate.engineSign(Signature.java:1207)
java.security.Signature.sign(Signature.java:579)
com.google.api.client.util.SecurityUtils.sign(SecurityUtils.java:147)
com.google.api.client.json.webtoken.JsonWebSignature.signUsingRsaSha256(JsonWebSignature.java:637)
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials.getJwtAccess(ServiceAccountJwtAccessCredentials.java:300)
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials.getRequestMetadata(ServiceAccountJwtAccessCredentials.java:267)
com.google.auth.Credentials.blockingGetToCallback(Credentials.java:103)
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials.getRequestMetadata(ServiceAccountJwtAccessCredentials.java:251)
io.grpc.auth.GoogleAuthLibraryCallCredentials.applyRequestMetadata(GoogleAuthLibraryCallCredentials.java:90)
io.grpc.internal.CallCredentialsApplyingTransportFactory$CallCredentialsApplyingTransport.newStream(CallCredentialsApplyingTransportFactory.java:91)
io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:242)
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:387)
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:679)
io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32)
com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:95)
io.grpc.stub.ClientCalls.startCall(ClientCalls.java:293)
io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:268)
io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:177)
com.google.pubsub.v1.PublisherGrpc$PublisherFutureStub.publish(PublisherGrpc.java:538)
com.google.cloud.pubsub.v1.Publisher.publishOutstandingBatch(Publisher.java:333)
com.google.cloud.pubsub.v1.Publisher.access$000(Publisher.java:90)
com.google.cloud.pubsub.v1.Publisher$1.run(Publisher.java:255)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)