rqbit v9.0.0-beta.1 major release

Major changes since v9.0.0-beta.0 (where uTP support is the main change)

• Full IPv6 support. Rqbit now runs everything in dualstack IPv6 + IPv4 mode by default. It can also work in pure-IPv6 mode just fine. This could discover and connect to more peers.
• rqbit download is now completely stateless and doesn't use the same ports as rqbit server does. This makes it very easy to run multiple concurrent instances.

Why isn't this a stable release yet?

I want to bundle as many breaking changes as possible into v9, but I want to enable uTP by default, and it still requires a bit of work and observations. One other way to do it is to make v9 and v10, but I'd rather do it once for simplicity. Otherwise this beta should be fully working and better than v8

All PRs merged (mostly above + refactorings)

• Fix a very rare bug where DHT paniced by @ikatson in #380
• [feature] allow creating torrents via HTTP API by @ikatson in #387
• [bugfix] WebUI: padding files are not checked by default by @ikatson in #388
• improvement: backend ignores only files if they are padding files + code reuse by @ikatson in #389
• [enhancement]: create sparse files on Windows by @ikatson in #390
• [feature] TCP and uTP: listen on IPv6 dualstack instead of IPv4 only by @ikatson in #392
• [feature]: HTTP trackers now support IPv6 peers by @ikatson in #393
• [feature]: UDP trackers IPv6 support by @ikatson in #394
• [enhancement]: optimize tracker log verbosity + a improvement in HTTP retrying by @ikatson in #395
• [feature]: DHT - dualstack over IPv6 by @ikatson in #396
• DHT: patch persistence file for the upgraded DHT (to enable v6) by @ikatson in #397
• [fix]: utp - better wait algorithm for TCP failures by @ikatson in #398
• [refactor] Refactor compact ips code reuse by @ikatson in #399
• [fix] invalid torrent names to use from_utf8_lossy #316 by @ikatson in #400
• [fix] Do not try to delete padding files as they don't exist by @ikatson in #401
• [refactor] small optimizations to new compact_ip.rs by @ikatson in #404
• [enhancement] DHT improvements by @ikatson in #405
• [refactor]: optimize bencode error handling and perf by @ikatson in #406
• [breaking] rqbit download: make it a lot simpler (fixes #402) by @ikatson in #403
• [refactor / perf] various (potential) perf tweaks by @ikatson in #409

Full Changelog: v9.0.0-beta.0...v9.0.0-beta.1

patch release 8.1.1

Backported a few changes from main to stable

• #388 WebUI: padding files are not checked by default
• #380 Fix a very rare bug where DHT paniced
• #400 [fix] invalid torrent names to use from_utf8_lossy
• #401 Do not try to delete padding files as they don't exist

Full Changelog: v8.1.0...v8.1.1

Major new rqbit release 9.0.0-beta.0

The are some major new features and some breaking changes in the API / CLI.

TLDR

• rqbit now supports uTP protocol, which will help discovering more peers. Enable via --experimental-enable-utp-listen. I'll probably enable it by default once 9.0.0 full is released.
• breaking CLI change: you need to replace --tcp-min(max)-port with --listen-port if you used those.
• breaking librqbit changes: some SessionOptions are moved around

uTP support

After a long wait, we finally have support for uTP (uTorrent transport protocol) done through specially created librqbit-utp library.

What does that mean for end users? Mostly, that you'll get more peer connections, especially on torrents with low seed count. It turns out that in the wild there's lots of peers that only have uTP enabled, but they don't listen on TCP. For those torrents previously rqbit couldn't connect to those peers at all.

This turned out to be one of the hardest pieces of code I've ever written, it's pretty much as complex as creating a TCP stack from scratch.

There are some caveats though, which is why uTP is disabled by default and you need to enable it explicitly via --experimental-enable-utp-listen.

uTP caveats

First, the original reason for uTP creation was LEDBAT congestion control algorithm, which is notably missing from the current implementation, and we use CUBIC (the modern TCP congestion control algorithm) instead. I'll probably add LEDBAT someday, it shouldn't be hard. The reasoning behind LEDBAT is to move torrent traffic to background so that it doesn't interfere with normal network operation on the machine. In practice, in my network conditions this is never an issue: I never notice rqbit slowing down my internet to make it noticeable.

Second, the librqbit-utp code is complex and there might be bugs. It seems stable for me, hence I'm making this release: I was running my rqbit for a while in uTP-only mode. But I'm asking all users to please try it out and report issues.

Third, although I tried hard optimizing it, it can't beat kernel's TCP stacks that were being optimized for tens of years by very talented engineers. TCP runs in the kernel, while uTP in tokio userspace. For this reason, rqbit first tries connecting over TCP, and only if it doesn't work after 1 second, a uTP connection attempt is made.

uTP can saturate my gigabit connection however on Apple devices, so it's not slow, just slower than TCP at the limits, and also noticeable on slower devices. E.g. my OpenWrt router can squeeze ~50 megabytes/s with TCP, but only about 20-25 over uTP. Of course it can be optimized further and it's just the beginning.

One more caveat is UDP socket read buffer is shared among all connections, unlike TCP (which is per connection). librqbit-utp tries to increase SO_RCVBUF, but it's likely to fail increasing it enough at least on Linux. This can result in lost incoming packets which will make uTP work considerably worse with fast download speeds, so to prevent this you might need to increase /proc/sys/net/rmem_max on Linux.

Prometheus metrics exporter

While creating librqbit-utp it proved extremely handy to monitor it using metrics, as logs would be too verbose and have too much overhead.

So I added prometheus metrics available under /metrics HTTP endpoint. For now there's only uTP metrics, but we can add more to rqbit itself.

Other changes including breaking

• added --listen-port, --listen-ip, --enable-prometheus-exporter options
• breaking: env / CLI: RQBIT_TCP_LISTEN_MIN_PORT (--tcp-min-port) and RQBIT_TCP_LISTEN_MAX_PORT (--tcp-max-port) moved to RQBIT_LISTEN_PORT
• breaking: librqbit SessionOptions: peer_opts and socks_proxy_url moved to opts: Option<ConnectionOptions>
• breaking: librqbit SessionOptions: enable_upnp_port_forwarding moved to listen: Option<ListenerOptions>

What's Changed

• Use Rust edition 2024 by @ikatson in #377
• librqbit: backoff crate (unmaintained) -> backon by @ikatson in #378
• uTP support (disabled by default) by @ikatson in #322
• Preparing 9.0.0-beta.0 release by @ikatson in #379

Full Changelog: v8.1.0...v9.0.0-beta.0

What's Changed

• [feature] Initial support for private torrents by @ikatson in #299
• Can unpause torrents even if there's no peers by @ikatson in #300
• [Feature] Can put rqbit Web UI behind proxy server by @ikatson in #301
• [Feature] Redirect to Web UI on root when looked from browser by @ikatson in #302
• [refactor] HTTP API maintanability by @ikatson in #303
• Fix typo by @ChrisAntaki in #312
• [perf] don't use tokio::io::split by @ikatson in #320
• Make Session ratelimits accessible from outside of crate by @InoUno in #329
• Add support for P2P blocklists by @xobust in #330
• [feature] List of trackers from file by @ikatson in #337
• [enhancement] UDP tracker socket reuse by @ikatson in #338
• Increase min metadata size by @ikatson in #339
• CI: Use binstall, Dynamic dockerhub user and push by @luigi311 in #282
• fix: make SessionStatsSnapshot fields pub by @KevinT3Hu in #360
• CI: Tauri build linux by @luigi311 in #365
• Do not exchange peers or torrent metadata in private swarms/ by @av-gal in #364
• watch .magnet files too by @FrancescElies in #367
• fix: improve path traversal check by @illfygli in #370
• ⬆️ (librqbit) Update dep requirement for url to compatible w/ current version by @djmango in #358
• Include crate version in client fingerprint portion of peer_id by @av-gal in #374
• Prepare for releasing 8.1.0 by @ikatson in #376

New Contributors

• @ChrisAntaki made their first contribution in #312
• @InoUno made their first contribution in #329
• @xobust made their first contribution in #330
• @KevinT3Hu made their first contribution in #360
• @av-gal made their first contribution in #364
• @FrancescElies made their first contribution in #367
• @illfygli made their first contribution in #370
• @djmango made their first contribution in #358

Full Changelog: v8.0.0...v8.1.0

rqbit v8.0.0

Why a major release? Lots of merged PRs since v7.0.1, minor breaking changes in librqbit API and CLI naming.
Other than that, it's a checkpoint to release all the work by many contributors.

Breaking changes

• Minimal Rust version bumped from 1.75 to 1.78
• CLI parameters changes:
  • renamed disable-upnp -> disable-upnp-port-forward
  • removed upnp-server-hostname
  • added enable-upnp-server
• librqbit API changes: if any, they are minor, but might be some depending on usage. All should be easy to fix just by resolving compilation errors.

UI features

• There's a banner in the bottom showing total up/down speed and uploaded/downloaded bytes since restart
• Upload multiple files through UI

Some Other features

• upload/download rate limits
• watching a directory for .torrent files and adding them automatically (usage in #237)
• Improved UPNP server support. Now works on some Samsung TVs, and should be more reliable overall.

HTTP API changes

• /playlist URLs
• basic auth support (see #279 for usage)

BitTorrent protocol features

• BEP-47 padding files
• BEP-53 select only files from magnet
• BEP-11 Peer EXchange

What's Changed

• feat: add on_piece_completed method on TorrentStorage by @cocool97 in #219
• PEX - Peer Exchange initial impl by @izderadicka by @ikatson in #221
• [Feature] UPNP MediaServer: send notifies on all interfaces, no need to specify hostname by @ikatson in #222
• [Feature] option to disable upload by @ikatson in #223
• Desktop same workspace by @ikatson in #225
• [Feature] [UPnP] connection manager stub by @ikatson in #226
• Various tweaks to peer protocol information by @ikatson in #227
• Smarter fastresume by @ikatson in #228
• Standardized m3u8 formatting by @Artrix9095 in #229
• Remove error when adding duplicate torrents by @Artrix9095 in #230
• [UPnP / DLNA] Updates for Samsung to work by @ikatson in #231
• UPNP: join all multicast groups from all interfaces (not just the default one picked by the kernel) by @ikatson in #232
• Fastresume: check at least one piece from each file + windows fix by @ikatson in #235
• In PEX we should also look at dropped peers by @izderadicka in #233
• Upload multiple files through UI (fix #68) by @ikatson in #236
• [Feature] watching a directory for .torrent files and adding them automatically by @ikatson in #237
• [Feature] SSDP IPv6 support by @ikatson in #238
• [Refactor] Re-use code in "merge_streams" by @ikatson in #240
• Change to restarting of dead peers by @izderadicka in #239
• Add support for adding custom trackers by @Moeweb647252 in #243
• Correct peer outgoing address for incomming peers by @izderadicka in #244
• [Refactor] Generic peer IP by @ikatson in #246
• [Perf] Increase http_api stream capacity (perf optimisation) by @ikatson in #247
• feat: upgrade to tauri 2.0 by @PastaPastaPasta in #249
• feat: implement BEP-53 support by @PastaPastaPasta in #248
• refactor: use byteorder crate to enhance portability (to BE systems), and enhance code readability, avoiding manual byte manipulations by @PastaPastaPasta in #250
• Disable upload: do not expose in default builds, move under a feature flag. by @ikatson in #251
• Migrate linux configuration to a better named path. by @ikatson in #253
• Fix: transactional pause by @Mrreadiness in #254
• Add CLI parameter to allow send downloads to remote server by @izderadicka in #257
• Added destination_fodler to TorrentDetailsResponse by @markolo25 in #263
• Endpoint to show all torrents and all stats by @ikatson in #268
• BEP-47 padding files + refactor related code by @ikatson in #269
• BEP-47 - UI support and updates by @ikatson in #270
• [feature] support 40-byte infohash (not a magnet) as a way to add torrents by @ikatson in #271
• Update minimal Rust version to 1.76 by @ikatson in #278
• [feature] upload/download rate limits by @ikatson in #275
• Basic auth in HTTP API by @ikatson in #279
• Enable thin LTO release but not release-github by @luigi311 in #283
• Clippy updates + update Rust deps by @ikatson in #289
• PEX - sharing peers with othes [continue #261] by @ikatson in #284
• [feature] HTTP API timeouts by @ikatson in #290
• [breaking] multiple refactorings in preparation for deferring torrent metadata resolution by @ikatson in #292
• Prepare v8.0.0 by @ikatson in #294

New Contributors

• @cocool97 made their first contribution in #219
• @Artrix9095 made their first contribution in #229
• @Moeweb647252 made their first contribution in #243
• @PastaPastaPasta made their first contribution in #249
• @Mrreadiness made their first contribution in #254
• @markolo25 made their first contribution in #263
• @luigi311 made their first contribution in #283

Full Changelog: v7.0.1...v8.0.0

What's Changed since 7.1.0-beta.0

• UPNP: join all multicast groups from all interfaces (not just the default one picked by the kernel) by @ikatson in #232
• Fastresume: check at least one piece from each file + windows fix by @ikatson in #235
• In PEX we should also look at dropped peers by @izderadicka in #233

Full Changelog: v7.1.0-beta.0...v7.1.0-beta.1

What's Changed

• feat: add on_piece_completed method on TorrentStorage by @cocool97 in #219
• PEX - Peer Exchange initial impl by @izderadicka by @ikatson in #221
• [Feature] UPNP MediaServer: send notifies on all interfaces, no need to specify hostname by @ikatson in #222
• [Feature] option to disable upload by @ikatson in #223
• Desktop same workspace by @ikatson in #225
• [Feature] [UPnP] connection manager stub by @ikatson in #226
• Various tweaks to peer protocol information by @ikatson in #227
• Smarter fastresume by @ikatson in #228
• Standardized m3u8 formatting by @Artrix9095 in #229
• Remove error when adding duplicate torrents by @Artrix9095 in #230
• [UPnP / DLNA] Updates for Samsung to work by @ikatson in #231

New Contributors

• @cocool97 made their first contribution in #219
• @Artrix9095 made their first contribution in #229

Full Changelog: v7.0.1...v7.1.0-beta.0

7.0.1 - hotfix release

The binaries were ok, but the cargo crate had an issue were it couldn't parse torrents - fixed here c063cc5

NOTE: also for 7.0.0 and 7.0.1 the docker images and linux binaries were compiled to use ring for SHA1, instead of openssl, which turned out to be very slow. I reverted to openssl and rebuilt these without upgrading the version as nothing in rqbit changed, only the build parameters (enabled features).

7.0.0 release notes

Full Changelog: v7.0.0...v7.0.1

rqbit 7

Major new features:

• Integrated UPnP server to browse and stream directly to your LAN devices (e.g. TVs) more here
• SOCKS5 proxy support docs
• Now publishing multi-arch Docker images
• Playlist generation support through API and UI links
• Aggregate statistics in API and visible in UI
• MANY reliability improvements
• Fastresume - to restart quickly without rehashing

There's minor breaking changes in the CLI and librqbit API.

Merged PRs

• Playlist for playable media by @izderadicka in #167
• SOCKS5 proxy support by @ikatson in #168
• Copy playlist to clipboard, native UI (not alert) by @ikatson in #169
• Sort playlist and fix playlist URL copy by @izderadicka in #171
• Global playlist, not just per torrent by @ikatson in #172
• Playlist ordering and mime by @izderadicka in #173
• Fix playlist content type issues by @ikatson in #176
• Workaround default download dir #175 by @ikatson in #178
• Fix #154 - desktop now can upload files properly by @ikatson in #179
• Generate webui dist files as part of build.rs by @ikatson in #180
• Add an HTTP endopoint to resolve magnet URL to bytes (address #177) by @ikatson in #181
• Use bytes crate for zerocopy and memory re-use by @ikatson in #182
• Speed up e2e test by @ikatson in #183
• [Feature] support sending metadata to peers who request it (via extended request) by @ikatson in #184
• [Features] Support for pluggable session persistence storages by @ikatson in #185
• [Feature] postgres backend for session persistence by @ikatson in #186
• Hash-based API in addition to integer based by @ikatson in #187
• Split up librqbit http_api and tracing_subscriber into separate features by @ikatson in #188
• Limit concurrency of torrent initialization (fix #139) by @ikatson in #189
• Remove node-modules/.package-lock.json by @izderadicka in #191
• Implement Borrow<[u8]> for ByteBuf types by @izderadicka in #193
• Fix e2e test to break less by @ikatson in #195
• Fix a bug in merge_two_streams by @ikatson in #196
• Other fixes found by e2e test by @ikatson in #197
• Further e2e fixes for reliability by @ikatson in #198
• Fix unchoke broken in previous PR by @ikatson in #199
• [Feature] Decode torrent as JSON in the /torrents/resolve_magnet API by @ikatson in #200
• Fix a bug in torrent deletion by @ikatson in #201
• Update e2e test to test deletion by @ikatson in #202
• [Feature] Fast resume - quick restart without rehashing by @ikatson in #203
• [Feature] session stats by @ikatson in #204
• Cleanups by @ikatson in #205
• Fill yourip in extended handshake by @izderadicka in #206
• Fix persistence bugs by @ikatson in #207
• [Major feature] UPNP server integrated into rqbit by @ikatson in #208
• [Feature] UPnP server configurable from UI by @ikatson in #209
• [Feature] Add environment variables support to rqbit binary by @ikatson in #212
• Switch x86_64 build to use musl by @ikatson in #213
• [Feature] add umask option by @ikatson in #214
• Add docker build for linux/amd64 by @ikatson in #215
• Graceful shutdown + send ssdp:bye on termination by @ikatson in #216
• SO_REUSEADDR on upnp socket by @ikatson in #217
• Mediaserver response proper st by @ikatson in #218

Full Changelog: v6.0.0...v7.0.0

Major feature

UPnP server integrated into rqbit - it can advertise all known torrents to local network, and you can browse and stream torrents from compatible devices (checked on my LG and Sony TVs).

Configurable both from UI and CLI.

CLI usage

rqbit --upnp-server-hostname 192.168.0.112 --upnp-server-friendly-name "rqbit is awesome" server start ...

UI

What's Changed

• Fill yourip in extended handshake by @izderadicka in #206
• Fix persistence bugs by @ikatson in #207
• [Major feature] UPNP server integrated into rqbit by @ikatson in #208
• [Feature] UPnP server configurable from UI by @ikatson in #209

Full Changelog: v7.0.0-beta.2...v7.0.0-beta.3