s3/fshttp: --timeout only enforces an idle deadline, so a slow trickle (e.g. wedged multipart upload) can hang indefinitely with no error and no retry #9841

What is the problem you are having with rclone?

--timeout does not bound total request/transfer duration -- it only enforces
an idle deadline on the raw connection, reset on every successful Read/Write
of any nonzero byte count (timeoutConn.nudgeDeadline, fs/fshttp/dialer.go).

A peer that drips data slower than a real transfer, but with gaps shorter than
--timeout, can hold a connection open indefinitely. No Read/Write ever
returns an error, so neither --retries nor --low-level-retries ever engage
either -- there is nothing for them to retry.

We hit this in production against a DigitalOcean Spaces bucket: a single
rclone copy multipart part upload hung for ~16 hours. strace//proc
showed an established TCP connection, ongoing (if minimal) syscall activity,
and slowly-increasing CPU ticks across samples -- i.e. not deadlocked, just
never finishing and never erroring. --stats=5m never printed a single
Transferred: line the entire run. The job used --retries=3 --retries-sleep=60s (defaults otherwise) and none of that ever fired.

I've confirmed the mechanism in isolation with a minimal repro rather than
relying on the live incident (which isn't reproducible on demand against a
third-party bucket): see below.

Repro

Branch: https://github.com/halindrome/rclone/tree/repro/timeout-does-not-bound-slow-trickle
(fs/fshttp/stall_repro_test.go, TestTimeoutDoesNotBoundSlowTrickle)

A local net.Listener accepts a request, then writes 20 response bytes one at
a time with a 150ms sleep between each, against a client configured with
--timeout 300ms. Every individual gap (150ms) is under the configured
timeout, so nudgeDeadline resets it each time. The request completes in
~3.08s -- 10.3x the configured timeout -- with no error, no retry, no
--low-level-retries engagement:

=== RUN   TestTimeoutDoesNotBoundSlowTrickle
    stall_repro_test.go:99: configured --timeout=300ms; request actually took
    3.082205503s (10.3x over) and SUCCEEDED with no error, no retry, and no
    --low-level-retries engagement -- --timeout does not bound a slow trickle
--- PASS: TestTimeoutDoesNotBoundSlowTrickle (3.08s)

(Test is deliberately a require-based assertion of the current, buggy
behavior succeeding, i.e. it currently passes -- happy to invert it to fail
against current behavior if that's more useful for triage.)

What is your rclone version (output from rclone version)

rclone v1.74.4
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-137-generic (x86_64)
- os/type: linux
- os/arch: amd64

(Confirmed the same mechanism is present on current master, see repro branch above.)

Which cloud storage system are you using?

Amazon S3 (provider: DigitalOcean, Spaces)

The command you were trying to run

rclone copy /backup/backup/db/ do-sfo:corvex-backups-sfo3-cold/prod/ \
  --transfers=8 --checkers=8 --retries=3 --retries-sleep=60s \
  --stats-one-line --stats=5m --size-only --s3-no-check-bucket \
  --fast-list --tpslimit=10 --tpslimit-burst=20

Why this matters / suggested direction

--timeout's doc string and most users' mental model of it is "kill a stalled
transfer" -- but as implemented it only detects a fully silent connection,
not a slow-but-technically-alive one. In practice this means a wedged
multipart upload (e.g. an orphaned/desynced multipart session on the S3 side,
or a misbehaving proxy) can run forever with zero visibility, since nothing
in rclone's error/retry machinery ever gets a chance to fire.

A minimum-throughput / stall-detection guard (in the spirit of curl's
--speed-limit/--speed-time) on top of the existing idle-timeout would
close this: track bytes moved per timeout window and error out (surfacing to
the normal --retries path) if that falls under some threshold, rather than
only checking "was anything read/written at all."

Happy to work up a PR for this (I already have the repro and read through
fs/fshttp and the S3 backend's HTTP client wiring to trace the mechanism),
but wanted to raise it here first in case there's already a preferred design,
a reason this hasn't been done, or a "yes but do X instead" before I sink time
into an implementation. Let me know what shape of fix (if any) you'd want to
see, and I'll send a PR.