-
Notifications
You must be signed in to change notification settings - Fork 699
openbsd: fix build #3428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
openbsd: fix build #3428
Conversation
| @@ -1,4 +1,4 @@ | |||
| #[cfg(all(not(target_os = "macos"), not(target_os = "windows")))] | |||
| #[cfg(all(not(target_os = "macos"), not(target_os = "windows"), not(target_os = "openbsd")))] | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you tell me what problem this resolves exactly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I understand OpenBSD uses its own allocator, third-party allocators (jemalloc, tcmalloc, mimalloc) are intentionally not supported,
OpenBSD has a very strong security focus.
tikv_jemallocator is known to build failure in OpenBSD
- missing symbols
- unsupported platform
- allocator conflicts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which allocator does OpenBSD use?
We're using jemallocator here because in the past users reported that on Linux the default allocator (glibc) didn't release memory promptly, but I'm not sure if OpenBSD has the same problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OpenBSD doesn't use glibc malloc, instead uses a libc malloc optimized for security.
- Hardened by default
- Memory randomization
- Minimal Thread arenas
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you seen a continuous increase in memory usage while running Yazi, especially when switching back and forth between multiple image previews?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yazi image preview, memory speaking, in OpenBSD is not eficient as Linux,
but experimenting my self with large hundreds of photos, it is still good,
went from the initial 20M to (after large hundred of images) about 165M.
But yes, the memory will very slightly increase, that's not a problem of Yazi,
many packages in OpenBSD has these problem, they are very
conservative and always prioritizes security over... anything else. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will the memory drop from 165M to 20M after some time? Or does it stay at 165M until the program exits?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately, it doesn't! I don't know if there is a way to solve it in OpenBSD.
And my technical habilities in OpenBSD are also limited, I must confess, well not just in OpenBSD! :)
For me at least is not a problem, maybe is a problem if someone going through
large thousands of images, but yes, the problem although subtle, is there.
|
|
||
| #[cfg(target_os = "openbsd")] | ||
| fn final_path(path: &Path) -> io::Result<PathBuf> { | ||
| std::fs::canonicalize(path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not right as final_path should not follow symlinks but canonicalize does
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right — canonicalize() follows symlinks and changes semantics.
I used it as a workaround for OpenBSD.
I can replace it with a component-based normalization that preserves symlinks and matches the original behavior.
#[cfg(target_os = "openbsd")]
fn final_path(path: &Path) -> io::Result<PathBuf> {
use std::path::{Component, PathBuf};
let mut out = PathBuf::new();
for comp in path.components() {
match comp {
Component::Prefix(_) => out.push(comp),
Component::RootDir => out.push(comp),
Component::CurDir => {}
Component::ParentDir => {
out.pop();
}
Component::Normal(c) => out.push(c),
}
}
Ok(out)
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
component-based normalization won't work in this case unfortunately since final_path needs to obtain the actual path of the given path on the filesystem without following symlinks, which means that when passed /root/abc, if the filesystem is case‑insensitive and the real entry is named ABC, it should return /root/ABC instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I see, maybe this way instead?
#[cfg(target_os = "openbsd")]
fn final_path(path: &Path) -> io::Result<PathBuf> {
use std::{fs, os::unix::fs::MetadataExt, path::PathBuf};
let meta = fs::symlink_metadata(path)?;
let parent = path.parent().ok_or_else(|| std::io::Error::new(
std::io::ErrorKind::Other,
"Cannot get parent directory",
))?;
// scan parent dir
for entry in fs::read_dir(parent)? {
let entry = entry?;
let entry_meta = entry.metadata()?;
if entry_meta.ino() == meta.ino() && entry_meta.dev() == meta.dev() {
return Ok(entry.path());
}
}
Err(std::io::Error::new(std::io::ErrorKind::NotFound, "Path not found"))
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would need to walk all files in the directory, which is costly for large directories, so I'd like to avoid it if possible. What does the current implementation error on OpenBSD exactly? Perhaps we can find a fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I must say, is a bit challenge in OpenBSD.
OpenBSD prioritizes correctness and security over raw speed.
On OpenBSD there is unfortunately no syscall equivalent to F_GETPATH or /proc/self/fd,
that allows recovering the canonical on-disk path without following symlinks.
What do you think about this alternative:
#[cfg(target_os = "openbsd")]
fn final_path(path: &Path) -> io::Result<PathBuf> {
use std::{fs, os::unix::fs::DirEntryExt, os::unix::fs::MetadataExt};
let meta = fs::symlink_metadata(path)?;
let parent = path.parent()
.ok_or_else(|| io::Error::other("Cannot get parent directory"))?;
let target = path.file_name();
let mut fallback = None;
for entry in fs::read_dir(parent)? {
let entry = entry?;
// inode match = same file, no stat() needed
if entry.ino() != meta.ino() {
continue;
}
let entry_path = entry.path();
// exact match → done (fast exit)
if target == entry_path.file_name() {
return Ok(entry_path);
}
// remember first inode match for case-insensitive FS
if fallback.is_none() {
fallback = Some(entry_path);
}
}
fallback.ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, "Path not found"))
}
The implementation therefore falls back to scanning the parent directory only when necessary. To minimize cost:
- It compares inodes using DirEntryExt::ino() to avoid stat() calls.
- It exits early on an exact filename match.
- It only walks the parent directory, never recursively.
This appears to be the minimal correct solution given the APIs OpenBSD exposes today.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Traversing parent directories and comparing file inodes is not as "correct" or "safe" as using F_GETPATH or /proc/self/fd, because multiple different filenames can share the same inode.
For example, if abc has inode X and Abc is a hard link to abc, it will also have inode X. This becomes problematic on case-sensitive filesystems because we always perform case-insensitive filename comparisons and that's also why the Linux implementation first tries to get the path via /proc/self/fd which covers most cases without having to fall back to walking parent directories.
Is there a way on OpenBSD to detect whether the filesystem is case-sensitive, and in that case return the path directly to avoid the problems caused by case-insensitive fuzzy matching of inode and filename?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the clarification. On OpenBSD, native FFS filesystems as far as I know, are always case-sensitive, so in most cases the exact filename exists and the fallback scan is safe.
Since there is no equivalent of Linux’s /proc/self/fd or macOS F_GETPATH on OpenBSD, the implementation first tries the fast path (symlink_metadata) and only scans the parent directory if necessary. The scan uses inode comparison to minimize syscalls, and the first inode match is returned as a fallback, which covers the rare case of foreign or case-insensitive filesystems.
So while it isn’t as “correct” as /proc/self/fd, it is the best practical approach for OpenBSD.
I tried, within my technical habilities to make some improvements as you requested:
#[cfg(target_os = "openbsd")]
fn final_path(path: &std::path::Path) -> std::io::Result<std::path::PathBuf> {
use std::{fs, os::unix::fs::DirEntryExt, os::unix::fs::MetadataExt};
// 1. Fast path: exact spelling exists
if fs::symlink_metadata(path).is_ok() {
return Ok(path.to_path_buf());
}
// 2. Fallback: directory scan
// On OpenBSD, unlike Linux's /proc/self/fd or macOS's F_GETPATH,
// there is no API to get the exact cased path without scanning.
// For most cases on FFS (case-sensitive), the first inode match will
// be the same file, so this is usually safe. On rare case-insensitive
// or foreign filesystems, this handles the best-effort matching.
let meta = fs::symlink_metadata(path)?;
let parent = path.parent()
.ok_or_else(|| std::io::Error::other("Cannot get parent directory"))?;
let target = path.file_name();
let mut fallback = None;
for entry in fs::read_dir(parent)? {
let entry = entry?;
// inode match = same file, no stat() needed
if entry.ino() != meta.ino() {
continue;
}
let entry_path = entry.path();
// exact filename match → preferred
if target == entry_path.file_name() {
return Ok(entry_path);
}
// remember first inode match as fallback for rare case-insensitive FS
if fallback.is_none() {
fallback = Some(entry_path);
}
}
fallback.ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "Path not found")
})
}
Let me know if that approach aligns with what you would recommend.
And I hope to have answer your questions clearly.
I am trying to follow, my sincere apologies.
| self.inner.pre_exec(move || { | ||
| let rlp = libc::rlimit { rlim_cur: max as _, rlim_max: max as _ }; | ||
| #[cfg(target_os = "openbsd")] | ||
| libc::setrlimit(libc::RLIMIT_DATA, &rlp); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason for using RLIMIT_DATA instead of RLIMIT_AS?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RLIMIT_AS on OpenBSD is not supported.
OpenBSD intentionally avoids relying on global address-space limits for security and correctness reasons.
Instead, OpenBSD favors:
- Per-subsystem limits
- Conservative enforcement
- Predictable failure modes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure whether using RLIMIT_DATA to limit the memory usage of spawned child processes makes sense - is this a common practice on OpenBSD? Or is there an OpenBSD-specific API that should be used for this purpose?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't answer correctly your question, my knowledge is "dumb by default" in some specifics,
but if I try to build with RLIMIT_AS it fails, because undeed it's not supported.
If there is an API I don't know, but I don't think OpenBSD has that, they are
very rigid with the code.
Cargo.toml
Outdated
| [profile.release] | ||
| codegen-units = 1 | ||
| lto = true | ||
| lto = false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be changed upstream as we (as downstream) need to have lto enabled for better optimized code
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, you're right, in OpenBSD lld is intentionally conservative, the best I could do it it was using lto = "thin",
I supposed the alternative is some exception for OpenBSD, I don't know.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you revert this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reverted lto to true.
Although, to build it successufully in OpenBSD,
the person would have to change it, at least to "thin".
|
Hi, I've fixed a build issue on NetBSD in #3506, could you try to see if it also fixes the OpenBSD problem? |
Which issue does this PR resolve?
Currently yazi fails to build in OpenBSD, this PR solves the problem, and I been able
to use yazi in my OpenBSD 7.8-Current AMD64 machine.
Rationale of this PR
It was a simple solution, some target_os specifics, also turn libc::RLIMIT_AS intto libc::RLIMIT_DATA.
Also disable tikv-jemallocator for OpenBSD target.
I had also to change lto to false, that is crucial for a standard OpenBSD, I hope some better idea to enable lto
in a standard OpenBSD.
I am not a developer, just trying to contribute... keep the awesome work.