Constant stream of internal RPC request failures #21623

mjs · 2025-10-06T00:46:53Z

mjs
Oct 6, 2025

I'm looking at a MinIO cluster with 4 nodes. When I use mc admin trace --call internal --errors I see a constant stream of failing requests originating from all the nodes in the cluster. There seems to be two types of failures that are constantly happening.

The first are 500 errors for storageR.ReadXL, for example (hostnames etc anonymised):

2025-10-05T19:27:54.286 [500 Internal Server Error] storageR.ReadXL minio2:9000/nvme5/minio?did=9d79f517-a048-4dd8-a48e-671ed5e4ca7e&fp=some%2Fpath minio4:9000                           631µs       ⇣  631.043µs  ↑ 150 B ↓ 0 B

The second are 500 errors for peer.ClearUploadID, for example:

2025-10-05T19:42:17.290 [500 Internal Server Error] peer.ClearUploadID minio4:9000?up-id=NWI3MTcwN2ItYjA4Ni00NTg4LWE3MGQtOTJjN2ZkMTQ1OWJhLmU0M2ExYzA3LTAxOTYtNDNiMS05NzQ4LTBlODEzY2ExMjBlYngxNzU5NzExMzM3MTYzNTE1Njg0 minio1:9000                           3µs         ⇣  3.102µs   ↑ 133 B ↓ 0 B

The cluster appears to be functioning normally otherwise. Are these failing requests expected?

Answered by klauspost

Oct 24, 2025

Yes. It is normal.

View full answer

doctorwisecrypt2222 · 2025-10-24T06:34:08Z

doctorwisecrypt2222
Oct 24, 2025

the same things

1 reply

klauspost Oct 24, 2025
Collaborator

Yes. It is normal.

Answer selected by klauspost

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Constant stream of internal RPC request failures #21623

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Constant stream of internal RPC request failures #21623

Uh oh!

mjs Oct 6, 2025

Replies: 1 comment · 1 reply

Uh oh!

doctorwisecrypt2222 Oct 24, 2025

Uh oh!

klauspost Oct 24, 2025 Collaborator

mjs
Oct 6, 2025

Replies: 1 comment 1 reply

doctorwisecrypt2222
Oct 24, 2025

klauspost Oct 24, 2025
Collaborator