Conversation
bc28e30 to
b371326
Compare
Our error marking code: ``` function foobar ^~~~~~~^ ``` runs fish_wcswidth to figure out how wide the squiggly line should be. That function returns -1 when it runs into a codepoint that wcwidth returns -1 for, so the marking would stop at a single `^`. In some cases, this happens because the error range includes a newline. Since we already find the end of the line, and can only mark one line, we clamp the squiggles at the end of that line. This improves some markings. See discussion in fish-shell#12171
Our error marking code: ``` function foobar ^~~~~~~^ ``` runs fish_wcswidth to figure out how wide the squiggly line should be. That function returns -1 when it runs into a codepoint that wcwidth returns -1 for, so the marking would stop at a single `^`. In some cases, this happens because the error range includes a newline. Since we already find the end of the line, and can only mark one line, we clamp the squiggles at the end of that line. This improves some markings. See discussion in #12171
Run fish_indent on test scripts that will be modified in a later commit Not running it on all the files because a quarter of them need fixing, some of which are badly formatted on purpose
- Use the different strings for different checks to more easily narrow down where a failure happens - Move CHECK comments outside a `if...else...end` to avoid giving the impression that the check only runs in the `if` case.
b371326 to
2d60f17
Compare
| __fish_test_thrash_cd | | ||
| __fish_test_thrash_cd | | ||
| __fish_test_thrash_cd | | ||
| __fish_test_thrash_cd |
There was a problem hiding this comment.
haven't gotten around to reading more than the first commit, but we can finally fix the formatting once and for all,
by having build_tools/style.fish check as much as possible.
(even if we don't add the fish_indent on/off directive)
Feel free to include this (or not, I'm not yet sure how big the rest of this PR is).
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 7f4c584037..d69b89434d 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -110,7 +110,7 @@
For formatting, we use:
- ``rustfmt`` for Rust
-- ``fish_indent`` (shipped with fish) for fish script
+- ``build_tools/fish_indent_wrapper.fish`` (shipped with fish) for fish script
- ``ruff format`` for Python
To reformat files, there is a script
@@ -124,7 +124,7 @@
-----------------------
1. All fish scripts, such as those in the *share/functions* and *tests*
- directories, should be formatted using the ``fish_indent`` command.
+ directories, should be formatted using the ``build_tools/fish_indent_wrapper.fish`` command.
2. Function names should be in all lowercase with words separated by
underscores. Private functions should begin with an underscore. The
diff --git a/build_tools/fish_indent_wrapper.fish b/build_tools/fish_indent_wrapper.fish
new file mode 100755
index 0000000000..01056ec2eb
--- /dev/null
+++ b/build_tools/fish_indent_wrapper.fish
@@ -0,0 +1,17 @@
+#!/usr/bin/env fish
+
+set -l workspace_root (status dirname)/..
+if string match -qr -- '\n' $argv
+ echo "$(status filename): arguments must not contain newline"
+ exit 1
+end
+builtin fish_indent (
+ for arg in $argv
+ switch (realpath -- $arg)
+ case $workspace_root/tests/checks/some-intentionally-unformatted-file.fish # TODO...
+ # Don't format it.
+ case '*'
+ echo $arg
+ end
+ end
+)
diff --git a/build_tools/style.fish b/build_tools/style.fish
index f26305f20f..c13298ed76 100755
--- a/build_tools/style.fish
+++ b/build_tools/style.fish
@@ -43,7 +43,9 @@
end
end
end
- set fish_files $workspace_root/{benchmarks,build_tools,etc,share}/**.fish
+ set fish_files \
+ $workspace_root/{benchmarks,build_tools,etc,share}/**.fish \
+ tests/checks/basic.fish
set python_files $workspace_root
else
# Format the files specified as arguments.
@@ -71,10 +73,10 @@
end
echo === Running "$green"fish_indent"$normal"
if set -l -q _flag_check
- fish_indent --check -- $fish_files
- or die "Fish files are not formatted correctly."
+ $workspace_root/build_tools/fish_indent_wrapper.fish --check -- $fish_files
+ or die "Fish files are not formatted correctly (Did you forget to set up build_tools/fish_indent_wrapper.fish ?)"
else
- fish_indent -w -- $fish_files
+ $workspace_root/build_tools/fish_indent_wrapper.fish -w -- $fish_files
end
end
|
Hmm, not sure why the automated test freezes. It happened to me as well when I pushed to a temporary branch in my repo. I then restarted the action and it passed. |
When the output is redirected, Python buffer its whole output, unlike a TTY output where only lines are buffered. In GitHub actions in particular, it means that we can't see any progress after each test. And if a test blocks forever, there is no output at all. So flush the output after printing each result to see the progress being made
2d60f17 to
d604398
Compare
|
I've added a |
|
Looks like |
Those tests are unreliable and sometimes even block forever on Cygwin/MSYS.
d604398 to
f302181
Compare
| # Not using setup-msys2 `install` option to make it easier to copy/paste | ||
| run: | | ||
| pacman --noconfirm -S --needed git rust | ||
| pacman --noconfirm -S --needed git rust python3 diffutils tmux |
There was a problem hiding this comment.
looks good to me overall.
I'll probably use actions-tmate to ssh into the runners to debug any future failures
| set existing_parent (dirname $existing_parent) | ||
| end | ||
|
|
||
| if mount | string match "*on $(stat -c %m -- $existing_parent) *" | string match -qr "[(,]noacl[),]" |
There was a problem hiding this comment.
Since we now have two of these checks,
can we extract something like
share/functions/__fish_in_cygwin.fish and
share/functions/__fish_in_cygwin_noacl.fish (or make it a flag like "__fish_in_cygwin --noacl=$dir")
Even if it's a suboptimal name, it's probably better than duplicating the regexes?
I think __fish_in_cygwin_noacl would have the same interface as cygwin_noacl,
since only __fish_make_cache_dir needs to traverse parent directories
(so the path traversal logic probably better stay)
I guess that would make the call to __fish_in_cygwin_noacl check for MSYS|CYGWIN redundantly but if we use the cached value fish __fish_uname that seems fine?
There was a problem hiding this comment.
I thought about factorizing the code before. But it didn't feel worthwhile to ship more function files only for for that one __fish_make_cache_dir function, especially given how simple those cygwin functions are.
Moreover, I do not like the idea of shipping the cygwin_noacl in particular since I've put an exit call in it and I couldn't think of a better way to do that part (I forgot to pass the path too many times for the removal of the arg check to be an option; using a return instead of exit would complicate the tests code much more than duplicating the function does; and using an exit call in ship code just feels plain wrong, even if it's never supposed to be triggered with proper code).
So in the end, it felt that duplicating the code was the better option, at least until more shipped code require to check for cygwin and noacl.
There was a problem hiding this comment.
We've had a lot of issues due to unnecessarily duplicated code in the past;
don't worry about shipping extra function files - nowadays those don't actually end up on the user's file system;
instead they are embedded into the fish binary. We could also put multiple such functions into a single source file, but that's for later.
Also, all (well, almost all) functions that start with __ are internal functions, so it's not a commitment to expose anything to the user.
The exit call seems nice, so we can have a test-only wrapper that has it.
(If such a programming error happens in shipped code, we should not exit but print something about "internal error" and return nonzero. Error handling in our fish scripts is kind of a best-effort thing, which happens to be fine because most errors basically never happen).
diff --git a/share/functions/__fish_cygwin_noacl.fish b/share/functions/__fish_cygwin_noacl.fish
new file mode 100644
index 0000000000..dea7c30d09
--- /dev/null
+++ b/share/functions/__fish_cygwin_noacl.fish
@@ -0,0 +1,7 @@
+# localization: skip(private)
+function __fish_cygwin_noacl
+ __fish_is_cygwin
+ # MSYS (default) and Cygwin (non-default) mounts may not support POSIX
+ # permissions.
+ and mount | string match "*on $(stat -c %m -- $argv[1]) *" | string match -qr "[(,]noacl[),]"
+end
diff --git a/tests/test_functions/is_cygwin.fish b/share/functions/__fish_is_cygwin.fish
rename from tests/test_functions/is_cygwin.fish
rename to share/functions/__fish_is_cygwin.fish
index dd3ee532f0..a81f248248 100644
--- a/tests/test_functions/is_cygwin.fish
+++ b/share/functions/__fish_is_cygwin.fish
@@ -1,3 +1,4 @@
-function is_cygwin
+# localization: skip(private)
+function __fish_is_cygwin
__fish_uname | string match -qr "^(MSYS|CYGWIN)"
end
diff --git a/share/functions/__fish_make_cache_dir.fish b/share/functions/__fish_make_cache_dir.fish
index 95c8b35e65..245d3da6c8 100644
--- a/share/functions/__fish_make_cache_dir.fish
+++ b/share/functions/__fish_make_cache_dir.fish
@@ -12,14 +12,14 @@
set -l mkdir_options -m 700
# Can't set the permission in Cygwin on a `noacl` mount
- if __fish_uname | string match -qr "^(MSYS|CYGWIN)"
+ if __fish_is_cygwin
# Find the first existing parent so we can `stat` it and get its mountpoint
set -l existing_parent $xdg_cache_home/fish/"$argv[1]"
while not path is -d $existing_parent
set existing_parent (path dirname $existing_parent)
end
- if mount | string match "*on $(stat -c %m -- $existing_parent) *" | string match -qr "[(,]noacl[),]"
+ if __fish_cygwin_noacl
set mkdir_options
end
end
diff --git a/tests/checks/command-not-found.fish b/tests/checks/command-not-found.fish
index 09749cd510..c5bb3e1f76 100644
--- a/tests/checks/command-not-found.fish
+++ b/tests/checks/command-not-found.fish
@@ -1,6 +1,6 @@
#RUN: fish=%fish %fish %s
-if is_cygwin
+if __fish_is_cygwin
# The Cygwin/MSYS DLLs must be in the path, otherwise Fish cannot be
# executed
set -g PATH /usr/bin
diff --git a/tests/checks/complete-cygwin.fish b/tests/checks/complete-cygwin.fish
index 812dd51b93..90beaf722e 100644
--- a/tests/checks/complete-cygwin.fish
+++ b/tests/checks/complete-cygwin.fish
@@ -1,6 +1,6 @@
#RUN: fish=%fish %fish %s
-# REQUIRES: %fish -c "is_cygwin"
+# REQUIRES: %fish -c "__fish_is_cygwin"
mkdir dir
echo "#!/bin/sh" >dir/foo.exe
diff --git a/tests/checks/default-setup-path.fish b/tests/checks/default-setup-path.fish
index 5ce8dbad63..2862e38fd4 100644
--- a/tests/checks/default-setup-path.fish
+++ b/tests/checks/default-setup-path.fish
@@ -1,7 +1,7 @@
#RUN: fish=%fish %fish %s
# Cygwin/MSYS PATH automatically inherits the Windows PATH
-# REQUIRES: %fish -c "not is_cygwin"
+# REQUIRES: %fish -c "not __fish_is_cygwin"
if command -q getconf
# (no env -u, some systems don't support that)
diff --git a/tests/checks/jobs.fish b/tests/checks/jobs.fish
index 5c7d3ad9ee..f47c7f4950 100644
--- a/tests/checks/jobs.fish
+++ b/tests/checks/jobs.fish
@@ -16,7 +16,7 @@
# simultaneously that causes a zombie to spawn, so limit the output only to processes started by
# this fish instance.
# Cygwin does not have zombies (or at least, `ps` cannot show if they are or not)
-if not is_cygwin
+if not __fish_is_cygwin
if not contains (uname) SunOS
ps -o ppid,stat
else
diff --git a/tests/checks/tmux-prompt.fish b/tests/checks/tmux-prompt.fish
index 705dd79f40..8b7e085415 100644
--- a/tests/checks/tmux-prompt.fish
+++ b/tests/checks/tmux-prompt.fish
@@ -18,7 +18,7 @@
isolated-tmux capture-pane -p
# CHECK: prompt 0> <status=0> <>
-if is_cygwin
+if __fish_is_cygwin
# See issue #12074
echo "prompt 0> <status=0> <changed>"
else
diff --git a/tests/test_functions/cygwin_noacl.fish b/tests/test_functions/cygwin_noacl.fish
index f3ccecc93f..da0af0fd76 100644
--- a/tests/test_functions/cygwin_noacl.fish
+++ b/tests/test_functions/cygwin_noacl.fish
@@ -4,10 +4,6 @@
set -q argv[1] || { echo "cygwin_noacl: missing path" >&2; exit 2 }
- # MSYS (default) and Cygwin (non-default) mounts may not support POSIX
- # permissions. Define a `noacl` variable if that's not the case, so we can cheat
- # on the test
- is_cygwin || return 1
-
- mount | string match "*on $(stat -c %m -- $argv[1]) *" | string match -qr "[(,]noacl[),]"
+ # Define a `noacl` variable if that's not the case, so we can cheat on the test
+ __fish_cygwin_noacl $argv[1]
endThere was a problem hiding this comment.
BTW we could also query status build-info to test if we're on Cygwin.
Not sure if that's better or worse; I guess it could in theory save a call to uname.
It's not perfectly machine-readable output but we could fix that by adding a status build-info target sub-sub-command, similar to what we did in #12106
That would also increase robustness of some existing uses of status build-info.
There was a problem hiding this comment.
We've had a lot of issues due to unnecessarily duplicated code in the past;
I get that and generally I agree. It's just that in this particular case, I do not think it's likely to be an issue and worth the cost. But you're the boss, I'll change it.
it could in theory save a call to uname.
That's micro-optimization, especially since __fish_uname caches the result anyway. If so, creating function files is also costly because it means fish may have to regularly (instead of one-time) check the disk in multiple places to see if the function has been updated/replaced (and on Cygwin, stat and the like are quite expensive... but then uname itself is impressively slow too, at 20ms per uname.exe call and 7ms just for the syscall on my system, vs 1ms and 0.2us respectively in WSL).
status build-info
This assumes that the build environment and the runtime one are identical/similar enough. Currently they have to be. Later, there is no guarantee there won't be some cross-compilation option (built on Linux for Cygwin/MSYS2, built on Cygwin for MSYS2 or vice-versa, ...) or even cross-runtime (same binary for both Cygwin and MSYS2) since MSYS2 is trying to close the gap between it and Cygwin.
Also while for noacl it doesn't matter if it's Cygwin or MSYS2 since they would both need to run the same commands, it wouldn't be the same for the nosymlink check, which does need to distinguish between the two. And I think it's better if both noacl and nosymlink use the same method.
If you want to avoid a call to uname, a status uname would be a better option IMHO, and semantically more appropriate.
There was a problem hiding this comment.
I do not think it's likely to be an issue
Duplication does add small maintenance cost.
But you're the boss, I'll change it.
meh :( ideally you change it because I'm right because we have resolved all your concerns:
If so, creating function files is also costly because it means fish may have to regularly (instead of one-time) check the disk in multiple places to see if the function has been updated/replaced
Ugh, I forgot about that.
If this is not totally insignificant, then we should fix it.
We should really not have too choose between making maintenance harder or making the software slower for the user,
when we can avoid both relatively easily.
Based on a superficial look it doesn't seem to be a problem at all for first-party function files.
For one, I'd expect that stat $fish_function_path is basically always cached so it should be fast.
But most importantly, we don't ever reload embedded functions.
I did
$ __fish_apropos nosuchcommand
$ echo >~/.config/fish/functions/__fish_apropos.fish '
function __fish_apropos; echo pwned; end'
$ __fish_apropos nosuchcommandand got the same result from __fish_apropos.
If we look at autoload.rs, we'll probably find that fish only looks at the source file where the function was originally loaded from (and not any other entries in $fish_function_path), and for embedded paths, that "file" is immutable, at least in release builds where the files are actually embedded. In debug builds, we currently also don't ever reload embedded paths even though we could (because they are read from disk); we might want to change this if it helps developers (I typically restart target/debug/fish).
20ms per uname.exe call
right. As of today, fish should only call it once at startup. Some infrequently-called places still use the uncached uname; we could change all of them to use __fish_uname I guess.
There was a problem hiding this comment.
I see, so uname is probably more correct than build-info for this.
There was a problem hiding this comment.
+ if __fish_cygwin_noacl
LOL, missing "$argv[1]"
Duplication does add small maintenance cost
Never said otherwise. I only said that deduplication also has a cost and that in this case I'm not convinced about which one is highest.
ideally you change it because we have resolved all your concerns
Ideally. And if one can't convince the other, you're the boss and your opinion wins the day :)
I'd expect that stat $fish_function_path is basically always cached
Only if the function was used recently enough and there isn't a lot of memory churning on the system (and if I remember correctly, Windows is also keen on preloading stuff so it may choose to evict old stuff even if there isn't any memory pressure)
Also in the case of Cygwin, I don't know how much it caches the result on its own, and when not cached there, how costly it is to reconstruct the POSIX data even when all the pieces are in the OS cache but it's not "cheap" (e.g. the x permission bit in noacl requires reading the beginning of the file, i.e. multiple roundtrips between Cygwin and Windows).
For perspective, to show how bad Windows can be, it takes 900ms to run time ls -l /usr/share/fish/functions/ on a cold cache (40ms when hot). Now I believe fish will only check a specific file, not the whole lot, so it won't be that bad for this PR, but I think it's worth keeping in mind when talking about "files on Cygwin". It's just really hard to overestimate how bad things can be here.
If we look at autoload.rs, we'll probably find that fish only looks at the source file where the function was originally loaded from
I didn't realize that. This is confusing and not how I interpreted the Autoloading functions part of the documentation. "If the function definition is changed" to me includes "having a new definition in a higher priority directory" but I understand it's probably better for performance reason.
| echo "called on line 123 of file fake/cd.fish" >&2 | ||
| else | ||
| begin | ||
| set -lx CDPATH other |
There was a problem hiding this comment.
(unrelated but exporting CDPATH is an anti-pattern because it
will change the behavior of "cd" in (noninteractive!) shells that
inherit it - for example they will print the destination if they used CDPATH
AFACT this test shouldn't need to export it.)
| test_env = makeenv(script_path, home) | ||
| if test_file_path.endswith(".fish"): | ||
| if IS_CYGWIN and is_tmux_test(test_file_path): | ||
| return TestSkip(arg) |
There was a problem hiding this comment.
nice; we could also use this TestSkip approach for the FISH_CI_SAN workaround in future.
(Though I'll try to set aside some time to fix the tmux tests soon.. no need to spend too much time on them)
| "Return 0 if Cygwin fakes symlinks, return 1 otherwise" | ||
|
|
||
| if string match -q "MSYS*" (uname) | ||
| string match -qv "*winsymlinks*" "$MSYS" |
There was a problem hiding this comment.
probably not string match -q ... would be easier to read than string match -vq.
(Also the if/else cascade could be a switch with `case 'MSYS*'`` etc. but either works for me)
The main changes are: - disabling some checks related to POSIX file permissions when a filesystem is mounted with "noacl" (default on MSYS2) - disabling some checks related to symlinks when using fake ones (file copy) Windows with acl hasn't been tested because 1) Cygwin itself does not have any Rust package yet to compile fish, and 2) MSYS2 defaults to `noacl`
f302181 to
5256ac3
Compare
|
I don't know why the windows test failed. I pushed to a temp branch on my repo, which passed, then push the PR branch, which also passed. |
maybe run it in a loop, in case it's reproducible, |
|
I've merged the first three commits as-is, hope that's okay (I just noticed this was still marked as draft); let's look into the SEGV issue some more. |
Yes, that's fine. I'm not the one who mark the PR as draft, it was never a draft in my eyes1 so I didn't know if I should remove the mark or not. Footnotes
|
(I marked it as a draft when the tests were hanging and it was unclear why, to prevent accidental merging) |
|
Some data:
So it's likely the crash from before is another consequence of something bad going on in |
|
I wonder if the |
|
The |
|
The 6h timeout is not because the tests take time, but because one froze. The job only really ran for ~2h before getting stuck. |
|
The |
Description
A bit of cleanup and a lot of check disabling to account for Cygwin/MSYS2 shortcomings.
All test checks pass on Cygwin now, but there are still issues preventing the use of
build_tools/check.sh. So for now, I'm still relying oncargo testand addedtests/test_driver.py.The main fixed issues are symlinks and POSIX permissions. The CI will run the tests with and without proper symlinks, but for now, only
noaclis tested (default mount option for MSYS2).As for
check.sh, there are least two issues with it:ruffis not available in the MSYS environment (it's only available for the MINGW64/UCRT64).fish_gettext_extraction-<hex>.exefrom loading (std-<hex>.dllnot found error)I didn't try further than that, so I do not know if there are more problems in
check.sh