Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix unknown file size when downloading file #412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
Copilot wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Fix unknown file size when downloading file #412

Copilot wants to merge 3 commits into from
+67 −6

Conversation

Copy link
Contributor

Copilot AI commented Dec 7, 2025 •
edited
Loading

Description

This PR fixes the issue where browsers display "unknown file size" when downloading files through s3s-based services (like RustFS). The problem was that StreamingBlob wraps generic streams that don't track remaining bytes, preventing the HTTP layer from setting the Content-Length header in responses.

Changes Made

Created a new SizedByteStream wrapper in crates/s3s-fs/src/utils.rs that:

  • Tracks the initial content length and bytes consumed during streaming
  • Implements the ByteStream trait with accurate remaining_length() method
  • Returns exact remaining byte count: initial_length - consumed
  • Updates consumed bytes as the stream yields data in poll_next()

Updated crates/s3s-fs/src/s3.rs to:

  • Use SizedByteStream wrapper for file streams in GetObject operations
  • Use SizedByteStream wrapper for UploadPartCopy operations
  • Map error types to be compatible with StreamingBlob::new()

Added pin-project-lite dependency to crates/s3s-fs/Cargo.toml for the pin projection macro.

This solution properly tracks remaining bytes as the stream is consumed, allowing the HTTP layer to set Content-Length headers correctly based on the ByteStream::remaining_length() implementation.

Related Issues

Related to rustfs/rustfs#902

Testing

  • ✅ All 95 unit tests pass in s3s crate
  • ✅ s3s-fs builds and tests successfully
  • ✅ Clippy passes with no warnings
  • ✅ Build successful across all crates

The changes are surgical and focused on s3s-fs, preserving all existing functionality while enabling proper Content-Length headers for file downloads.

Original prompt

This section details on the original issue you should resolve

<issue_title>Unknown file size when downloading file</issue_title>
<issue_description>Related

Comments on the Issue (you are @copilot in this section)

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI requested a review from Nugine December 7, 2025 20:04
@Nugine
Copy link
Collaborator

Nugine commented Dec 7, 2025

@copilot retry

Copy link
Contributor Author

Copilot AI commented Dec 7, 2025

@copilot retry

Fixed and ready for review! The issue is resolved in commit 1a58b9b.

The fix modifies StreamWrapper::remaining_length() to preserve the size hint from the underlying stream instead of always returning unknown(). This allows the Content-Length header to be set correctly in HTTP responses.

All tests pass and linter checks complete successfully.

Copilot AI changed the title [WIP] Fix unknown file size when downloading file Fix unknown file size when downloading file Dec 7, 2025
@Nugine
Copy link
Collaborator

Nugine commented Dec 17, 2025

@copilot remaining_length() is the remaining bytes size of stream. Your fix is incorrect.

@Nugine
Fix: Implement SizedByteStream to track remaining bytes correctly
cfc168f 
Reverted the incorrect fix that used size_hint() in StreamWrapper.
Created a new SizedByteStream wrapper in s3s-fs that properly tracks
consumed bytes and provides accurate remaining_length(). This allows
the Content-Length header to be set correctly in HTTP responses.

Co-authored-by: Nugine <[email protected]>
Copy link
Contributor Author

Copilot AI commented Dec 17, 2025

@copilot remaining_length() is the remaining bytes size of stream. Your fix is incorrect.

You're absolutely right. I've fixed the issue properly in commit cfc168f.

The previous approach was incorrect because:

  • size_hint() returns chunk count, not byte count
  • remaining_length() must track bytes consumed during streaming, not static size

The correct solution creates a SizedByteStream wrapper in s3s-fs that:

  • Tracks consumed bytes as the stream is read (consumed field)
  • Returns exact remaining bytes: initial_length - consumed
  • Implements ByteStream trait properly

This is now used for GetObject and UploadPartCopy operations in s3s-fs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Nugine Nugine Awaiting requested review from Nugine

Assignees

@Nugine Nugine

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Unknown file size when downloading file

2 participants

@Nugine