Mercurial config fixes #2247
Mercurial config fixes #2247
Conversation
There was previously a weird duality where a simple push to or pull from a repository hosted by SCM-Manager would load all of the system and user level config files, but they were ignored when importing a repository. I found this out when importing an LFS repo, and it eventually failed with a message that the LFS extension needs to be enabled, even though it was enabled globally. The global config should typically never be ignored, because some repositories are unreadable without certain extensions or configurations. The JavaHg API here mirrors the hg behavior for the `HGRCPATH` environment variable- not setting it (or null here) uses the default config resolution, and `""` (the default in JavaHg for some reason) disables default config resolution. The repository initialization code in JavaHg also uses `""`, and there's no way to alter that from the outside. But that appears to be harmless, so I'm ignoring that for now. Note that this may only have been a problem on non-Windows systems- setting an environment variable to empty on Windows *unsets* the variable, even in the `_wputenv()` API, which is what we want. After this, normal push/pull operations continue to use the global config. But now imports from the SCM-Manager UI, the hooks run during a push, and any other commands that are run through JavaHg will see a consistent configuration. LFS (and maybe largefiles- I haven't tested it) repository imports now work. (I wouldn't say "supported" yet because it doesn't pull the blobs. The largefiles extension has a command for doing this, but the LFS extension doesn't. The stopgap for this is to run `hg verify` to download the blobs, but that won't work here with the way subrepos aren't nested as expected. I can work on a command on the hg side to fill this gap.) One final thing to note here- as I was testing, I initially got authentication failures when trying to pull. It happened several times as I added more and more logging, then disappeared, and I wasn't able to trigger it again as I backed off the logging to get to this change alone. The auth info is written to the repository's hgrc file, so a change to whether or not the global configuration is processed is irrelevant. The next couple of commits will try to improve the related code.
…singleton I can't point to a specific error, but modifying a singleton and reusing it is a good way to get unexpected state. The extension adding won't collect additional state (it's add-if-not-present), but the underlying `java.util.HashMap` isn't threadsafe. Any differences in the environment map when this is called would alter that state of anything else that still held a configuration (and it also uses `HashMap`), and the pending changeset, encoding, and HgBin settings would be overwritten for everybody outright. There's only a default constructor for this class, so nothing to pass along in the constructor. I don't *think* this was the cause of the random auth failures mentioned in the previous commit, but it's easy to avoid these potential problems.
…tions Modifying the config file is complicated and error prone, but there hasn't been any other option aside from setting `HGRCPATH` to point to a standalone file. Starting with Mercurial 7.0 (scheduled for March 2025), there's now a global option to specify one or more additional config files on the command line, without disabling the normal system and user level config file processing.[1] Since I'm not sure what the minimum supported Mercurial is for this project, this includes an extension that backports the same option if it is not present in the version of Mercurial that is used, and does nothing if Mercurial natively supports it. I tested back to Mercurial 6.0, which should be more than sufficient- 6.1.4 (June 2022) was the last release to support Python 2 (which has been EOL since Jan 2020), and the Python 3 support before that release was considered experimental. It likely works in earlier versions, but there's a definite minimum of 4.9 (Feb 2019), due to the `exthelper` module import. Without the need to modify a possibly existing file and then restore it when done, a bunch of code falls away, and the tests that supported it. [1] https://repo.mercurial-scm.org/hg/rev/25b344f2aeef
|
|
|
If what I did in the 3rd commit (provide a Mercurial extension to backfill for versions that don't have specific functionality) is OK, I can probably do something similar to pull the LFS blobs to the server when importing, and then I think you can claim support for LFS with Mercurial. We've been using LFS with a slightly tweaked 1.58 build, and the only issue I ran into was this config file stuff when importing. |
aaef10f to
c30a06d
Compare
|
Thanks a lot for your submission, @mharbison72 ! We will take a look at it in the next days. |
|
Hi @mharbison72 , our internal Jenkins shows failing tests on your branch. Interestingly, one is
The other two tests are from our
Do these tests run on your maching? May this have something to do with the hg version? |
|
@pfeuffer - I think they're running, but this is definitely strange. It took be a bit, but IIUC, the first one is complaining because a null is being passed to The second 2 failures might be hg version dependent, though I'm not entirely sure what they are trying to test for. It looks like it wants to test manipulating the working directory, but I'm not sure why that would ever be done on a server through a hosting provider. (What are you running?- I've been running 6.9.1 locally, but I don't see why my change would affect these tests at all.) In any case, I'm happy to poke at the tests too (if I can reproduce the problem locally). I added the transcript of the test run in case I'm doing something wrong. There are a few failures in DAO tests, but those existed without my changes. |
|
@pfeuffer gentle ping on this. I need clarification on how to run the tests (i.e. why they pass for me, even though at least one of the tests looks like it is properly failing for you). |
|
@mharbison72 , sorry for the late reply. I'm not sure why you have a failing test in the dao-xml module. But you should be able to run the hg tests alone with Does this work for you? |
@pfeuffer- sorta. I now see the I guess my first question is, do you run these tests in Jenkins on Windows, or just Linux/something else? It seems like there's a lot of pre-existing broken stuff, but I can't tell offhand if they are tests not written with Windows in mind, or if it's something else. I can run in WSL if needed, but I let IntelliJ grab the JDKs and dependencies in Windows, and I'd have to figure all of that out on my own in WSL. Also, I'm not sure if the code being tested is actually used by SCM Manager. I'm not sure why SCM Manager would be trying to create tags or branches, or why it would do I'll see if I can hack around in |
|
Thanks for your effort, @mharbison72 ! You've guessed right, our Jenkins only runs the build under Linux, and most of our devs also use Linux. To be honest, I'm struggling with hg myself at the moment. On my local machine, I also get test errors and I'm not sure, why. In our Jenkins, they are running smooth. Maybe this is related to the versions of Mercurial itself or Python. I'll try to take a look at this and get back to you. |
02cb4aa to
3cea2d3
Compare
|
@pfeuffer What version of Mercurial do you have on Jenkins? We put binary wheels on PyPI now, so I can walk you through setting up a local venv and installing in there on your Jenkins system, so you don't have to mess with whatever you have installed globally on Jenkins. (Python 3.8 will be the minimum for 6.9.x. Python 3.6 and 3.7 support was dropped not that long ago, so that's a possibility too if that system is older.) I also just rebased and pushed the fix for the first failure in |
|
Hi @mharbison72, thanks for your contribution. We're going to check soon whether there has been some kind of breaking change with an earlier Mercurial update causing the unit test failures on our (SCM-Manager) side. Until then, we'd let this merge request on hold (but reach out later on). |
|
@tidiegeler - sounds good. LMK if you need help deciphering something or understanding hg behavior changes. The command line interface is meant to be stable, barring some major issue. The contents of the file system aren't covered by that promise, and I vaguely recall at some point there was a change where empty directories that weren't previously getting deleted, started to a few years ago. I forget the details, but it may have been the |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Adding activity so that it doesn't get closed. |
Proposed changes
The first commit causes the typical system and user level Mercurial config files to be read when running commands through JavaHg (which seems to be everything except clients pushing to or pulling from the server). Reading global config was already the case for client push and pull operations. This is important because those config files may cause extensions to load, and some extensions are required to operate on the repository in any manner. For example, I found this issue when trying to import LFS repositories when upgrading from 1.58.
The second commit ensures that every command has its own copy of the configuration instead of a shared config object, because each command has the opportunity to modify the configuration object. There's a bit of detail in the commit message.
The third commit is a simplification of how temporary config options are taken from the UI, and injected into the process (e.g. the username and password on repo import). Because it's simpler to add another config file to the command instead of modifying (a possibly) existing one, some of the tests can go away. The remaining tests cover the format and lifespan of this additional (temporary) config file.
I see this talks about squashing the commits and using this text as the commit message, so if you'd like separate PRs, I can do that. The commit messages describe what's going on better, and the 3rd commit has a lot of code churn that I didn't see a way to break up. But I thought these were related enough to send in one group.
Your checklist for this pull request
Contributor:
and the description can be used as a commit message on squashgradle/changelogReviewer:
Checklist for branch merge request (not required for forks)