refactor(scripts): check_urls.vim to Lua + Treesitter #35593
Conversation
ad3beb5 to
18a7b04
Compare
Spawn it in a child job. I don't think vim.async is needed here, this script is not intended for interactive use, it's a build-time step. |
scripts/check_urls.lua
Outdated
| local function curl(url, cb) | ||
| local cmd = { | ||
| 'curl', | ||
| '--silent', |
There was a problem hiding this comment.
can vim.net.request be used or what is missing from it?
oh, i guess it's missing "async" behavior.
There was a problem hiding this comment.
the redirect -L flag and that the error code value is a string, but I think that checking err ~= nil might be fine too.
There was a problem hiding this comment.
Okay most importantly it's missing a way to add a timeout, or I'm not seeing it. This could either be fixed by passing --max-time 3 to curl, or for example the approach proposed by Lewis: #34140 (comment)
I added |
that makes sense, just wondering if it matters since this is mostly a CI job and we can always enhance it later. the |
3ddc7c9 to
626daf8
Compare
scripts/check_urls.lua
Outdated
| -- if output is not specified, err will always be nil (seems curl-specific) | ||
| vim.net.request(url, { retry = 1, outpath = '/dev/null' }, function(err, _) | ||
| if err then | ||
| vim.print(('Unreachable url in %s: %s'):format(filename, url)) |
There was a problem hiding this comment.
@justinmk Preference for the output format? (here filename is the helpfile's name)
2ee7644 to
2a210c4
Compare
Problem: scripts/check_urls.vim manually matches urls in the help pages and then synchronously checks them via curl/wget/powershell. This is extremely slow (~5 minutes for Nvims runtime on my machine) and prone to errors in how the urls are matched. Solution: Use Treesitter to find the urls in the help pages. Asynchronously call curl for their response. Takes around 10s for the same docs.
2a210c4 to
d36f8f3
Compare
| @@ -25,7 +25,7 @@ function M.request(url, opts, on_response) | |||
| local retry = opts.retry or 3 | |||
|
|
|||
| -- Build curl command | |||
| local args = { 'curl' } | |||
| local args = { 'curl', '--max-time', '5' } | |||
There was a problem hiding this comment.
Not the final solution, but need to find a way to have this behaviour. Either by allowing curl arguments to vim.net.request or some callback-wizardry.
| -- Usage: | ||
| -- $ ./scripts/check_urls.lua [DIR...] | ||
| -- | ||
| -- [DIR...] defaults to all 'doc' directories in the runtimepath. |
There was a problem hiding this comment.
Current behaviour also checks runtime/pack/dist/opt (seems okay) but that also includes netrw (booo). I still think it should be like this: $VIMRUNTIME/doc is easy to supply as script argument, 'all doc directories in the rtp' is not.
Problem:
scripts/check_urls.vimmanually matches urls in the help pages and then synchronously checks them via curl/wget/powershell. This is extremely slow (~5 minutes for Nvims runtime on my machine) and prone to errors in how the urls are matched.Solution:
Use Treesitter to find the urls in the help pages. Asynchronously call curl for their response. Takes around 20s for the same docs (with a timeout of 5s per request).
Current limitations:
vim.net.request(curl), not wget and powershell