Nginx + headless service

There have been no deployments around the spikes. The connections are stable 10-15 minutes after deploying.

Thanks!

I should have been more precise: how do you distribute load across RPC services? Also NGINX (with grpc module) or something else? What about health checks (if it's Nginx)?

With your configuration (concurrency settings), the ResourceExhausted errors can only happen if RPC calls are not distributed uniformly (so, some instances receive mode calls than they can handle). The overall concurrency capacity at the anycable-go side is 600, so, if for some reason at some moment in time only 2/8 RPC instances were live, you can make them exhausted.

You need to tune concurrency at both ends. Right now, your RPC server has a default pool size (I guess) of 30 threads, but you try to make ~1k concurrent requests from the anycable-go side. AnyCable-Go concurrency is not a number of concurrent connections, but a number of concurrent RPC requests. The best way to pick the right values is to monitor the actual usage, see https://docs.anycable.io/anycable-go/instrumentation?id=metrics-and-what-can-we-learn-from-them.

if I want support 10k websocket client request, every response can in 6 second to finish.

It depends on what these clients are doing; do you have any monitoring set up?

Yes, we monitor anycable metrics set up on grafana , and daily average
Go routines number

RPC calls

adjust the conditional data below, speed will be improve ?
--read_buffer_size=4096 --write_buffer_size=4096 --max_message_size=512000 --rpc_concurrency=100

Could you please provide a legend for the RPC calls chart? Are these different hosts or different metrics?

different host collection on one chart , like this on pic show

the headless service not working or rpc pods getting down, is that it?

Yeah, looks so.

We are using client-side load balancing via ANYCABLE_RPC_HOST

One suggestion is to try our edge version (release is coming later this week; hopefully, tomorrow) which adds the grpc_active_conn_num metrics, showing how many gRPC connections are currently active (should be one per RPC endpoint in case of client-side load-balancing). This way, you can see if your WS servers connected to all RPC servers (and, thus, distribute calls across all of them) or not.