I have no more open issues with this PR, so removing the not-for-unstable label. Now waiting for the conceptual review by @twpol and @cjakeman .

Hi Chris @cjakeman,

Yes two of my PRs (this and #538) conflict with each other, and whichever gets merged first into the master, the other will have a conflict that needs to be resolved, I am aware of that. However I'm not sure how to resolve it at this stage. I cannot commit the fix into either of the PRs, because then that could not be merged without the other.

The best I can do now is that I mark #538 as not-for-unstable, however that is also ready for the review and the merge.

Hi Peter, @pzgulyas

However I'm not sure how to resolve it at this stage. I cannot commit the fix into either of the PRs, because then that could not be merged without the other.

I am happy to approve PR 538 once I've removed from the Manual the instruction to delete binary files by hand.
Hope that will unblock this mutual conflict.
Looking at that now.

Hi Peter, @pzgulyas

I am happy to approve PR 538

PR 538 is now approved and merged into the master branch.
You should be able to update your local master branch and then resolve the conflict.

Best wishes

Conflict resolved, back on track.

Conflict resolved, back on track.

That's great. I see the PR has made it into the Unstable build.

In trying things out, I came across a case which vfs.FileExists() rejects:

"\MSTS\trains\trainset\MT_DD_CLASS_27_27102\CABVIEW\..\..\MT_Common.cabs\HP\Class26\Class26.cvf"

although the file exists in that location. I hope you can come up with a simple fix.

Best wishes

• All modification actions (write/delete) must be removed, they are completely banned on content files/directories; any code that needs these must be fixed or removed separately

This might really be a valid argument, but currently the *.or-binpat files are written to the content directory, and had to choose to either break that functionality, or implement writing. I've chosen the latter. I think changing that is not in the scope of the VFS.

• The plain text file is a poor choice IMHO and a burden I do not want to go forward with; I would be happy to go forward with no VFS configuration initially (so only mounting directories with or without archives)

Please note that the configuration file is optional only, one can switch on automount and go with no configuration file. But in cases like the Openrails's downloadable DemoModel1.zip it is necessary, because it is not zipped to have the MSTS root-level folders into the zip's own root-level. Some means are necessary in such cases for binding the right archive directory to the right VFS directory.

I don't adhere to the plain text file, however besides its contras it has pros too. For example, because the configuration need to contain only pairs of strings, a json structure is actually more than what is needed here. I had the unix /etc/fstab in my mind, but of course, json didn't yet exist when that got invented. An example json structure could look like this:

{ "vfs": [ { "source": "C:/something/MSTS_Base.zip/Train Simulator/", "mountPoint": "/MSTS/" }, { "source": "C:/something/DemoModel1.zip/Demo Model 1/", "mountPoint": "/MSTS/" } ] }

• Switching to my main MSTS installation profile in the menu has slowed from four seconds to sixty seconds! We should figure out if we can selectively scan for VFS files or otherwise improve this

Interesting point, I didn't experience any comparable slowdown, but your concern might be valid. But I don't have an idea currently how to solve this exactly.

This might really be a valid argument, but currently the *.or-binpat files are written to the content directory, and had to choose to either break that functionality, or implement writing. I've chosen the latter. I think changing that is not in the scope of the VFS.

Yeah, I don't expect you to fix that here. I will get the 7-year-old (!) code which uses these files fixed.

It has been a foundational principal of Open Rails that it never modifies your content and, while I think these files skirt around that a bit by being cache files, I am quite sure most people would be surprised that we're doing it. :)

I don't adhere to the plain text file, however besides its contras it has pros too. For example, because the configuration need to contain only pairs of strings, a json structure is actually more than what is needed here. I had the unix /etc/fstab in my mind, but of course, json didn't yet exist when that got invented. An example json structure could look like this:

It's a combination of:

1. We have agreed as a project to always use JSON for new (non-binary) file formats
2. Plain text requires you to write a parser which is where all the bugs slip in :)

The update code with a JSON file looks better, thanks for that

Interesting point, I didn't experience any comparable slowdown, but your concern might be valid. But I don't have an idea currently how to solve this exactly.

I've had a look, and my main MSTS install (which is where A LOT of content has been installed) has the following properties compared to a clean MSTS install (with all base routes):

• 75.7GB vs 1.6GB
• 880k vs 20k files
• 17k vs 1k directories
• 75 ZIP files not in the root (0 in the root)

If I understood the code right, and you did scan recursively through for ZIP files to mount, the VFS would have had to scan through all those directories and then try to mount all those ZIPs - it takes dir /s > nul a full 25 seconds on a warmed-up SSD :)

And at this point, I'm not even sure mounting ZIPs anywhere without opt-in is a good plan, but let's at least cut out the recursive part to start with :)

Thanks very much for the updates, Peter, we're getting closer!

With PR #647 merged, we can now remove all the write/delete-related VFS code

My biggest concern is still that this is doing a lot; and consequently, this PR is too big for me to comfortably review (I know I've missed a load of things)

The aim of the first version was always to implement the VFS with no new features and no visible impact to users; hopefully, the auto-mount off will make the impact minimal (but it still scans 100% of files to initialise the VFS which people will notice), but the UI changes to allow for the configuration file are above and beyond what I'm comfortable with right now

Thanks James for the fix you've committed! I couldn't yet take the chance to check that out, and I will be unavailable for the weekend, so I am just answering shortly now, but I will answer all the other comments in detail later this month.

To me it looks like your biggest two concerns are the configuration file and the automounter. Shortly speaking, I've added these into my initial development to serve the same purpose: to support existing non-precisely packaged content.

Take for example the following two downloadable files:

1. The "Demo Model 1" from our own website: http://openrails.org/download/content/
2. I picked the first ET locomotive download by searching files with keyword "Open Rails": http://www.elvastower.com/forums/index.php?/files/file/3117-bo-4-4-0-no270-mason/

As you can see, neither of them can be mounted into the VFS without either using a config file or the automounter.

(1) is – lets say – "too long" in its packaged path. There must be a way to say to ORTS to "omit the first directory level, and mount it as if was packaged without the \Demo Model 1\ directory". This can be achieved by a configuration file.

(2) is "too short". It must be mounted under /MSTS/TRAINS/TRAINSET/. This can be achieved either by a config file or by the automounter (without config!).

So the automounter is one way to get rid of the config file, but it can help only in cases the packaged path is "too short". The "too long" cases only the config file can solve. You are repeatedly arguing against the config file and the automounter, but never explained a possible replacement solution for these problems. What would be your suggestion? (I can tell in advance, that a solution like "do not use mispackaged or before-this-date content packages" is not the thing that was in my mind when I developed this feature, but rather is exactly the opposite.)

You are repeatedly arguing against the config file and the automounter, but never explained a possible replacement solution for these problems. What would be your suggestion?

For the automounter, it is an acceptable feature to include, but its performance is not (#541 (review)), so we must either reduce its functionality or turn it off by default - which you've done, and that's good with me :)

For the configuration file, I have only requested that we do not include it in this version; I wanted/expected the VFS to be implemented in smaller steps than you've implemented here, and since we have not actually discussed anything about the configuration file, it is premature to include it IMHO

I have spent far too much of my involvement with Open Rails untangling the messes from prematurely implemented features, so I am extremely wary of things like the configuration file as a result - it needs to be discussed on the forums and agreed upon before code is written, or we might be stuck with something bad for years and years

Honestly, I don't like the idea of stripping the configurability at all, because without that most of the existing downloadable packages will be unusable. Also the VFS cannot be demonstrated with either of the examples (1) or (2) I've mentioned above. Actually your answer didn't address my concerns stated above about removing the configuration file. But I'm not against delaying the inclusion of the VFS into the official version until we all feel comfortable with how it works, it is a bad feeling to be stuck with premature functions for years.

Honestly, I don't like the idea of stripping the configurability at all, because without that most of the existing downloadable packages will be unusable.

If you don't want to remove the configuration part of this PR, we had better start discussing the design and format in the forum :)

@pzgulyas Hello! We (ORMT) have been discussing this feature and how it relates to existing pain points and downloadable content and are writing up a new proposal - would you be up for reviving this work?

The new proposal, which I will be publishing fully, is not too different:

• Installation profiles continue being isolated (same)
• Supports packages as ZIP files and directories (same)
• Package ZIP files and package directories are functionally identical (same)
• Keeps packages in a single location to avoid recursive scanning (improvement)
• Mounts packages using documented logic for too-deep and too-shallow structures (this hasn't been included in any previous proposal IIRC but is a nice feature of this implementation which we believe can be kept and expanded, subject to extensive testing of course)
• Special mapping and loading code (if necessary) to ensure global directories (e.g. GLOBAL and SOUND) are treated as route-specific (improvement)