Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Enable Large-file Support #1726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 4, 2024
Merged

Enable Large-file Support #1726

merged 3 commits into from
Jan 4, 2024

Conversation

oToToT
Copy link
Contributor

@oToToT oToToT commented Jan 3, 2024

This should fix #1725

@dmah42
Copy link
Member

dmah42 commented Jan 3, 2024

i'm not sure how necessary this is (see my comment on #1725 ) but if we do need it please also add these definitions to the bazel build.

dmah42
dmah42 previously approved these changes Jan 4, 2024
@dmah42
Copy link
Member

dmah42 commented Jan 4, 2024

i don't know what the pre-commit failure means or why it's an issue that the bazel file has changed in an expected way. maybe do what it says (pre-commit run --all-files).

perhaps some file is being generated based on the BUILD.bazel?

@oToToT
Copy link
Contributor Author

oToToT commented Jan 4, 2024

Looks like I accidentally use tabs instead of whitespaces as indent. It should be fixed now.

@dmah42 dmah42 merged commit c213e41 into google:main Jan 4, 2024
@dmah42
Copy link
Member

dmah42 commented Jan 4, 2024

thanks!

@dsharlet
Copy link
Contributor

dsharlet commented Feb 3, 2025

Is there a way to disable this change? Our project updated to a newer version of googlebenchmark, and this change breaks on some platforms, e.g.: https://github.com/google/XNNPACK/actions/runs/13079108015/job/36498319833

@dsharlet
Copy link
Contributor

dsharlet commented Feb 3, 2025

I share @dmah42 's confusion about why this would be necessary, I doubt the benchmark framework attempts to work with such large files that this matters.

copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722524164
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722551798
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722551798
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722557143
@dmah42
Copy link
Member

dmah42 commented Feb 3, 2025

should the flags be conditionally included on 64-bit platforms only?

@oToToT
Copy link
Contributor Author

oToToT commented Feb 3, 2025

should the flags be conditionally included on 64-bit platforms only?

IIUC, LFS is mainly for 32-bit platforms [1], so it has no sense to only include it on 64-bit platforms. (e.g. ftell should already be returning 64-bit values on 64-bit platforms)

It seems like XNNPACK is using r23b ndk [2]. Is it possible to raise it to r24? From my random searching, update the target to r24 should make LFS available. (I haven't tested it locally, so please correct me if I'm wrong.)

cc @vapier for vis

[1] https://en.wikipedia.org/wiki/Large-file_support
[2] https://github.com/google/XNNPACK/blob/c467d1a1b79db7c2686536720989cf524308eb4e/.github/workflows/build.yml#L277-L282

@vapier
Copy link
Member

vapier commented Feb 3, 2025

What is a "64-bit platform" ? That term is extremely imprecise. Is that the pointer size ? The filesystem offset size ? The default integer (long) size ? The time_t size ? A combination of them ? Please consult the table:
https://unix.org/version2/whatsnew/lp64_wp.html

My point being that there isn't really such a concept to tie LFS to.

The linked failure is because the Android NDK version used is not POSIX compliant (ftello doesn't exist). If the next NDK release fixes things, then I agree that's the better answer.

Back to the "64-bit platform" comment, Android supports 32-bit platforms, so if ftello doesn't exist in that NDK, then enabling LFS only on "32-bit platforms" would still break.

copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
See google/benchmark#1726 for context

PiperOrigin-RevId: 722706053
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
See google/benchmark#1726 for context

PiperOrigin-RevId: 722706053
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
See google/benchmark#1726 for context

PiperOrigin-RevId: 722706053
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
See google/benchmark#1726 for context

PiperOrigin-RevId: 722727190
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722753506
@dsharlet
Copy link
Contributor

dsharlet commented Feb 3, 2025

Even with the latest NDK available, this same problem exists: https://github.com/google/XNNPACK/actions/runs/13122183940/job/36610509007?pr=7770

I really think we just need a way to disable this, and failing that, consider reverting it (I don't understand the motivation for this change).

copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722774108
copybara-service bot pushed a commit to google/XNNPACK that referenced this pull request Feb 3, 2025
This appears to have broken the android actions. I've also commented on what appears to be the culprit upstream change here: google/benchmark#1726

PiperOrigin-RevId: 722774108
@oToToT
Copy link
Contributor Author

oToToT commented Feb 4, 2025

Even with the latest NDK available, this same problem exists: https://github.com/google/XNNPACK/actions/runs/13122183940/job/36610509007?pr=7770

I really think we just need a way to disable this, and failing that, consider reverting it (I don't understand the motivation for this change).

I think it works now after you have correctly updated both NDK version and compile flags.

@dmah42
Copy link
Member

dmah42 commented Feb 4, 2025

my understanding of the motivation is that on some platforms (OSes) the inode can be 64-bit, meaning that without this we can get hard-to-find errors on I/O. if there's a way to limit this change to those platforms, then let's do that. otherwise, i'm happy to revert it until we see those errors in the wild.

PRs welcome for either.

@oToToT
Copy link
Contributor Author

oToToT commented Feb 4, 2025

Do we still need to revert it even XNNPACK has already solved the issue themselves?

google/XNNPACK@411f9de (and its parent commit)

@dmah42
Copy link
Member

dmah42 commented Feb 4, 2025

if they were the only folks with an issue then no i don't think we do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FR] Large File Support
4 participants