Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fs/accounting: add --min-bandwidth / --min-bandwidth-time to detect stalled transfers - #9895

Draft
halindrome wants to merge 1 commit into
rclone:masterfrom
halindrome:feat/min-bandwidth-accounting
Draft

fs/accounting: add --min-bandwidth / --min-bandwidth-time to detect stalled transfers#9895
halindrome wants to merge 1 commit into
rclone:masterfrom
halindrome:feat/min-bandwidth-accounting

Conversation

@halindrome

@halindrome halindrome commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Fixes #9841

Draft. An earlier revision of this description claimed the CompleteMultipartUpload case was handled. It is not — see this comment and the "Known limitation" section below. The description has been corrected; the comment is kept for the record.

Adds --min-bandwidth (bytes/sec, default 0/disabled) and --min-bandwidth-time (default 60s): if a transfer averages below --min-bandwidth for a full --min-bandwidth-time window it is cancelled with a retryable error, so --retries can engage. Zero behaviour change when unset.

This is the two-knob shape @ncw described in #9841, matching curl's --speed-limit/--speed-time and google-drive-ocamlfuse's low_speed_limit/low_speed_time — the semantics asked for in #887 and #2122.

Known limitation — the CompleteMultipartUpload objection is NOT yet resolved

stallCheckLocked cancels purely on the EWMA falling below the threshold. It has no notion of a legitimate zero-progress phase — no completion gate, no source-EOF gate, no not-yet-started gate. So:

  • CompleteMultipartUpload: once the destination has consumed the whole source reader and is waiting on the server to finalise, no further AccountRead occurs, so avg decays toward 0 while updateOrPut is passing inAcc.Context() to that very finalisation request. At --min-bandwidth=1M on a 5 MB/s upload the cancel lands ~85s in, orphaning the multipart upload.
  • Server-side copy: bytes are booked in one lump at ServerSideCopyEnd, so avg is 0 throughout. It does not abort the copy (that runs under a different context), so the effect is a spurious stall report.
  • A transfer that has not yet moved its first byte arms the timer on the first tick.

Moving from timeoutConn to Account does fix keep-alive pooling and the measures-the-wrong-thing problem, but not this: the check fires on the absence of accounted bytes, and legitimate finalisation is exactly that absence. Fixing it needs an explicit exempt state rather than a guard, which changes what the feature promises — hence draft, pending @ncw's view on the shape.

Why fs/accounting and not timeoutConn

Per @ncw's review on #9841, and two of the three objections do hold here:

  • HTTP keep-alive pooling — the check is per-transfer, not per-connection, so a pooled connection cannot carry stall state into an unrelated request.
  • Measuring the wrong thing — Account tracks per-transfer progress rather than bytes-on-the-wire, and is fed real network bytes for multipart chunk uploads via rw.SetAccounting(acc.AccountRead) (lib/multipart.UploadMultipart).
  • Legitimate trickles — not resolved; see above.

Mechanism

Checked once per completed --min-bandwidth-time window against Account's existing EWMA (acc.values.avg — the same figure --stats uses), not the instantaneous rate, so a short dip does not trip it. Raised at most once per attempt (latched, reset in UpdateReader).

A sustained shortfall cancels the transfer's own context via context.WithCancelCause, so the cause (ErrorTransferStalled, carrying the measured speed) survives. ErrorTransferStalled is wrapped in fserrors.RetryError so StatsInfo.Error classifies it as retryable and --retries re-drives the transfer. updateOrPut maps context.Cause back onto the returned error, because the HTTP transport propagates only ctx.Err() and the reason would otherwise surface as context canceled.

Note this does not reach --low-level-retries: that loop runs inside the backend on fserrors.ShouldRetry, which consults Timeout()/Temporary() rather than Retry(), and the backend sees the transport's context.Canceled rather than this error. An earlier revision of this description claimed otherwise.

Account.Context() is new. Callers that create an Account must pass that — not the original ctx — to the subsequent Open/Put/Update, because the cancellation fires from averageLoop's background goroutine and only reaches an in-flight request running under the same context.

Scope

Wired into fs/operations/copy.go's updateOrPut, the single-file Put/Update path that the incident in #9841 hit.

Not migrated, and this is a defect rather than merely unfinished work — on these paths a stall is detected and logged but the abort cannot land:

Either those get migrated, or the stall check should not run for Accounts whose context is not wired through. Happy to do either in this PR.

Also worth a decision: the check hangs off every Account, not just sync transfers — vfs/read.go open handles, serve http/webdav GETs, HashSum, catObject. Being idle is normal for those, so a global --min-bandwidth would cancel an idle mount read handle after --min-bandwidth-time. Probably wants scoping or explicit documentation.

Tests

fs/accounting/min_bandwidth_test.go, driven through the real AccountRead path, each on a private config via fs.AddConfig (a shared global fs.ConfigInfo races averageLoop, which is what made CI red on an earlier revision):

  • TestAccountMinBandwidthDetectsStall — 10 bytes/sec against a 1000 bytes/sec minimum; surfaced via Context() and a subsequent AccountRead, and asserts the stall latches rather than re-firing every tick.
  • TestStallCheckResetsWindowOnRecovery — drives stallCheckLocked at chosen instants: a dip that recovers must not leave a stale window behind, so a later dip gets a full --min-bandwidth-time of its own.
  • TestAccountMinBandwidthRecovers — a dip that clears is not cancelled, run past the point where a non-resetting implementation would have fired.
  • TestAccountMinBandwidthStallIsRetryable — the error is marked retryable and survives wrapping.
  • TestAccountMinBandwidthDisabledByDefault — no behaviour change when unset.

Verified on go1.26: go vet ./fs/... and gofmt -l fs clean; go test -race ./fs/accounting/ ./fs/operations/ ./fs/ clean over the whole packages. make racequicktest shows the same nine failing packages as pristine master in the same container (root user defeats the permission-denial tests, no FUSE, NFS symlinks unsupported) and no others.

Each fix is mutation-checked: deleting the recovery reset, removing the whole recovery branch, unlatching the stall, and un-marking the error retryable each turn the corresponding test red.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cp6djGjYDg518RujgZwMsr

@halindrome
halindrome marked this pull request as draft September 9, 2026 19:46
@halindrome

halindrome commented Sep 9, 2026

Copy link
Copy Markdown
Author

Moving this to draft. I ran a proper review pass over my own branch after opening it and found that the PR description above overclaims in two places. Correcting that here rather than leaving it to waste your time, @ncw.

1. AC/objection 2 — CompleteMultipartUpload — is NOT actually handled

The description says "control traffic never goes through AccountRead, so it doesn't fire on that whitespace-drip." That is true and it is beside the point, which I missed: a minimum-bandwidth check does not need the finalisation traffic to be accounted — it needs it to be absent, which it is.

stallCheckLocked cancels purely on the EWMA falling below --min-bandwidth for --min-bandwidth-time. It has no notion of "this transfer is legitimately not moving accounting bytes right now" — no completion gate, no source-EOF gate, no not-yet-started gate. So:

  • CompleteMultipartUpload: once the destination has consumed the whole source reader and is waiting on the server to finalise, no further AccountRead occurs, so acc.values.avg decays 15/16 per tick toward 0. fs/operations/copy.go passes inAcc.Context() to dst.Update/f.Put — i.e. to the in-flight finalisation request. With --min-bandwidth=1M on a 5 MB/s upload, avg crosses the threshold in ~25s of silence and the cancel lands ~85s into the Complete — well inside the "minutes" you described. That is precisely the orphaned-multipart outcome the objection exists to prevent.
  • Server-side copy: bytes are booked in one lump at ServerSideCopyEnd, so avg is 0 for the whole operation. Every server-side copy longer than --min-bandwidth-time trips it. It does not abort the copy (that runs under a different context), so the visible effect is one ERROR line per second for the copy's duration.
  • Symmetrically, a transfer that has not yet moved its first byte has avg == 0 at construction and arms the timer on the first tick.

I think this affects the design as proposed, not just my implementation of it. Your reasoning was that the accounting layer "doesn't fire on control requests or server keep-alive whitespace (those aren't accounted transfer bytes)" — which is right about what is accounted, but the check triggers on the absence of accounting, and legitimate finalisation is exactly that absence. Moving from timeoutConn to Account fixes keep-alive pooling and the measure-the-wrong-thing problem (both real improvements), but it does not by itself fix this one.

So it needs an explicit exempt state rather than a one-line guard — something like: mark the Account as "source exhausted" when the reader hits EOF and stop checking from that point, plus a grace period before the first byte. Before I build that I'd rather hear whether you agree that's the right shape, since it changes what the feature promises: it would bound stalled data transfer, not a wedged request, and the wedged-Complete case then stays firmly in the separate per-request-deadline fix you already scoped out.

2. The test I cited as evidence for that objection cannot fail

TestAccountMinBandwidthRecovers — which I put forward as the guard against exactly this false-positive — is vacuous. Its lifetime is ~1.8s and averageLoop uses a 1-second ticker, so exactly one tick occurs, and that tick can only set belowMinSince. Cancellation requires now - belowMinSince >= MinBandwidthTime, i.e. t≈3s, which never arrives. The recovery-reset branch its docstring names as "what's under test" never executes; the test passes verbatim with stallCheckLocked deleted. My inline comment about "the 2s mark" is arithmetically wrong — the earliest possible cancel is ~3s.

TestAccountMinBandwidthDisabledByDefault has the same defect (~100ms lifetime, so the check never runs at all).

3. CI is red, and it is my test bug

go test -race ./fs/accounting/ over the whole package: 4x WARNING: DATA RACE plus --- FAIL: TestAccountMinBandwidthRecovers. The new tests set and defer-restore the shared global fs.ConfigInfo, which averageLoop reads every second from sibling tests' Accounts. Same bug class you had me fix with fs.AddConfig in #9843 — I reintroduced it in the new file. Sibling tests are visibly polluted with ERROR transfer stalled … lines.

My "passes under -race" claim was true only for a -run MinBandwidth subset, not the package. My mistake for not running the whole package under -race before opening this.

(The android-all failure is unrelated — a transient sum.golang.org error during gomobile bind.)

Also found, worth your view

  • ErrorTransferStalled is not retryable. It is a bare errors.New with no Retrier, so fserrors.ShouldRetry is false and --low-level-retries never engages; --retries engages only incidentally via IsNoRetryError. And because Go's transport surfaces ctx.Err() rather than context.Cause, the user sees Failed to copy: context canceled rather than the stall reason. So "retryable error, reason preserved" is currently half true. Both halves are cheap to fix.
  • Multi-thread copy detects a stall but cannot abort it. multiThreadCopy creates its Account with tr.Account(gCtx, nil) and never uses acc.Context(), so the cancel cannot reach the in-flight request — it logs once per second, indefinitely, and never aborts. Since multi-thread is the default above --multi-thread-cutoff 256Mi, that is exactly the wedged large-file upload case from s3/fshttp: --timeout only enforces an idle deadline, so a slow trickle (e.g. wedged multipart upload) can hang indefinitely with no error and no retry #9841. I had listed this call site as "not yet migrated"; that was accurate but understated — the shipped behaviour there is worse than not running the check at all.
  • Scope. The check hangs off every Account, not just sync transfers — vfs/read.go open handles, serve http/webdav GETs, HashSum, catObject. A paused video in a mount is an idle read handle, so with a global --min-bandwidth set it gets cancelled after --min-bandwidth-time. Being idle is normal for those. Probably wants scoping to non-checking transfer Accounts, or explicit documentation.

I'll fix the mechanical items (the race, the two vacuous tests, retryability, the once-per-second logging) regardless. Holding off on the exempt-state design until you've had a chance to say whether the shape above is what you'd want — and whether, given point 1, you still think Account is the right layer or whether this wants to be bounded differently.

@shanemccarron-maker @claude
fs/accounting: add --min-bandwidth / --min-bandwidth-time to detect s…
cc327ea 
…talled transfers

--timeout only resets an idle deadline on the raw connection whenever ANY
Read/Write moves n > 0 bytes (fs/fshttp timeoutConn.nudgeDeadline). A peer
dribbling even a single byte more often than --timeout can hold a connection
open indefinitely: nothing ever errors, so --retries / --low-level-retries
never get a chance to engage either. See issue rclone#9841 for the production
incident.

--min-bandwidth (bytes/sec, default 0/disabled) and --min-bandwidth-time
(default 60s) add a minimum-throughput check on a transfer's Account: if the
moving average stays below --min-bandwidth for a full --min-bandwidth-time
window, the transfer's own context is cancelled with ErrorTransferStalled,
which carries the measured speed and is marked retryable so --retries
re-drives it. Zero behaviour change when unset.

The check lives in fs/accounting rather than at the connection layer so it is
immune to HTTP keep-alive pooling, measures transfer progress rather than
bytes-on-the-wire, and works for backends rclone does not Dial. It is
evaluated once per completed window against the EWMA that --stats already
maintains, so a brief dip does not trip it, and it is raised at most once per
attempt.

Account.Context() is new: callers that create an Account must pass THIS (not
the original ctx) to the actual Open/Put/Update call, since the cancellation
is fired from averageLoop's background goroutine and can only reach an
in-flight request running under the same context. Wired into
fs/operations/copy.go's updateOrPut, which also maps context.Cause back onto
the returned error -- the HTTP transport propagates only ctx.Err(), so the
reason would otherwise reach the user as "context canceled".

KNOWN LIMITATION, not yet addressed: the check has no notion of a legitimate
zero-progress phase. A transfer whose source reader is exhausted while the
server finalises (S3 CompleteMultipartUpload), and a server-side copy whose
bytes are booked in one lump at the end, both look identical to a stall. This
needs an explicit source-EOF/not-started exempt state; see the discussion on
the pull request before relying on this with a server-side workload.

Account call sites other than updateOrPut (multithread copy, check) are not
migrated to Account.Context(), so on those paths a stall is detected but the
abort cannot land.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01Cp6djGjYDg518RujgZwMsr
@halindrome
halindrome force-pushed the feat/min-bandwidth-accounting branch from 69e7a7c to cc327ea Compare September 10, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ncw ncw Awaiting requested review from ncw ncw is a code owner
@nielash nielash Awaiting requested review from nielash nielash is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

s3/fshttp: --timeout only enforces an idle deadline, so a slow trickle (e.g. wedged multipart upload) can hang indefinitely with no error and no retry

2 participants

@halindrome @shanemccarron-maker