Environment details

• OS: Ubuntu
• Java version: 8
• google-cloud-java version(s): 1.62.0

Steps to reproduce

1. Run a publisher publishing with > 130,000 qps of 1kb sized messages
2. Run a subscriber doing minimal work in the callback other than incrementing an atomic counter

Code snippet

Integer BYTES_PER_WORKER = 100000000;
Integer workerCount = 5* Runtime.getRuntime().availableProcessors();
Subscriber.newBuilder(subscription, MinimalMessageReceiver)
                            .setParallelPullCount(workerCount)
                            .setFlowControlSettings(FlowControlSettings.newBuilder()
                                    .setMaxOutstandingElementCount(Long.MAX_VALUE)
                                    .setMaxOutstandingRequestBytes(BYTES_PER_WORKER * workerCount)
                                    .build())
                            .build();

Any additional information below

I've been working to improve the Cloud Pub/Sub loadtest framework. I've been unable to achieve the same throughput in java as I have in golang under the same conditions. I'm unable to increase subscriber throughput on a 16 core machine beyond 130MB/s with 1KB messages, but have been seeing 300+MB/s rates with golang under the same conditions. I've also tried setting the channel provider and executors to different values to no avail. Since the cpu load is stuck at about 35%, perhaps this is a contention or artificial limitation somewhere else in the client library?

In progress branch for updates here https://github.com/dpcollins-google/pubsub/tree/loadtest_updates