rt: revert #7757 to fix regression in spawn_blocking#8057
Merged
Conversation
This reverts commit 1604bc3. Unfortunately, this commit introduced a regression that causes programs using `spawn_blocking` to hang (see #8056). To fix the regression, we need to undo this change and publish a v1.52.1 release as soon as possible. In the future, we may wish to bring back a sharded queue for `spawn_blocking` tasks, either based on the implementation added in #7757 or a new one. However, since this is a substantial change to the runtime internals, I think such a change should probably be done as an unstable, opt-in `tokio::runtime::Builder` setting initially, so that we don't regress existing users. I had hoped we could do this now, but unfortunately, the sharded queue implementation from #7757 is kind of tightly coupled with the rest of the `spawn_blocking` machinery and cannot be easily swapped out --- and the hang still occurs with `NUM_SHARDS` set to 1, so there isn't an easy way to turn it on and off. Therefore, in the interest of getting a fix out ASAP, this is just a simple revert.
Member
Author
|
The remaining CI failures are due to CI picking up |
hawkw
added a commit
that referenced
this pull request
Apr 16, 2026
hawkw
added a commit
to oxidecomputer/omicron
that referenced
this pull request
Apr 16, 2026
Commit #10272 updated our dependency on `oxide-tokio-rt` to v0.1.4 and our `tokio` dependency to v1.52.0. This allowed us to pick up two of my fixes for Tokio issues that have been a thorn in our side for a long time, tokio-rs/tokio#7431 and tokio-rs/tokio#8010, which fix #8334 and #9619, respectively. The nature of these fixes is described in greater detail in #10272. Unfortunately, #10272 had to be reverted (in #10279), since @iliana discovered an unrelated regression in Tokio v1.52.0, tokio-rs/tokio#8056 (our issue #10277). This regression caused `spawn_blocking` to occasionally hang, and was introduced in tokio-rs/tokio@1604bc3 (PR tokio-rs/tokio#7757). I've since reverted this change upstream (tokio-rs/tokio#8057), and published a patch release ([v1.52.1]), which fixes the regression. Therefore, it is now once again safe to update our Tokio dependency to pick up the other fixes. This commit does that. I've also confirmed that the issue described in #10277 is no longer present in Tokio v1.52.1, as demonstrated by the fact `cargo nextest run -p omicron-sled-agent --stress-count 100 -- --exact artifact_store::test::issue_7796` now succeeds without hanging once again. Fixes #10272
eleboucher
pushed a commit
to eleboucher/towonel
that referenced
this pull request
Apr 18, 2026
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | workspace.dependencies | minor | `1.51.1` → `1.52.1` | --- ### Release Notes <details> <summary>tokio-rs/tokio (tokio)</summary> ### [`v1.52.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.1): Tokio v1.52.1 [Compare Source](tokio-rs/tokio@tokio-1.52.0...tokio-1.52.1) ### 1.52.1 (April 16th, 2026) #### Fixed - runtime: revert [#​7757] to fix [a regression][#​8056] that causes `spawn_blocking` to hang ([#​8057]) [#​7757]: tokio-rs/tokio#7757 [#​8056]: tokio-rs/tokio#8056 [#​8057]: tokio-rs/tokio#8057 ### [`v1.52.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0): Tokio v1.52.0 [Compare Source](tokio-rs/tokio@tokio-1.51.1...tokio-1.52.0) ### 1.52.0 (April 14th, 2026) #### Added - io: `AioSource::register_borrowed` for I/O safety support ([#​7992]) - net: add `try_io` function to `unix::pipe` sender and receiver types ([#​8030]) #### Added (unstable) - runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#​8010]) - taskdump: add `trace_with()` for customized task dumps ([#​8025]) - taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#​8040]) - fs: support `io_uring` in `AsyncRead` for `File` ([#​7907]) #### Changed - runtime: improve `spawn_blocking` scalability with sharded queue ([#​7757]) - runtime: use `compare_exchange_weak()` in worker queue ([#​8028]) #### Fixed - runtime: overflow second half of tasks when local queue is filled instead of first half ([#​8029]) #### Documented - docs: fix typo in `oneshot::Sender::send` docs ([#​8026]) - docs: hide #\[tokio::main] attribute in the docs of `sync::watch` ([#​8035]) - net: add docs on `ConnectionRefused` errors with UDP sockets ([#​7870]) [#​7757]: tokio-rs/tokio#7757 [#​7870]: tokio-rs/tokio#7870 [#​7907]: tokio-rs/tokio#7907 [#​7992]: tokio-rs/tokio#7992 [#​8010]: tokio-rs/tokio#8010 [#​8025]: tokio-rs/tokio#8025 [#​8026]: tokio-rs/tokio#8026 [#​8028]: tokio-rs/tokio#8028 [#​8029]: tokio-rs/tokio#8029 [#​8030]: tokio-rs/tokio#8030 [#​8035]: tokio-rs/tokio#8035 [#​8040]: tokio-rs/tokio#8040 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJ0eXBlL21pbm9yIl19--> Reviewed-on: https://git.erwanleboucher.dev/eleboucher/towonel/pulls/11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This reverts commit 1604bc3.
Unfortunately, this commit introduced a regression that causes programs using
spawn_blockingto hang (see #8056). To fix the regression, we need to undo this change and publish a v1.52.1 release as soon as possible.In the future, we may wish to bring back a sharded queue for
spawn_blockingtasks, either based on the implementation added in #7757 or a new one. However, since this is a substantial change to the runtime internals, I think such a change should probably be done as an unstable, opt-intokio::runtime::Buildersetting initially, so that we don't regress existing users. I had hoped we could do this now, but unfortunately, the sharded queue implementation from #7757 is kind of tightly coupled with the rest of thespawn_blockingmachinery and cannot be easily swapped out --- and the hang still occurs withNUM_SHARDSset to 1, so there isn't an easy way to turn it on and off. Therefore, in the interest of getting a fix out ASAP, this is just a simple revert.Fixes #8056