Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@nathanlesage
Copy link
Member

Description

Okay, I know, this is getting ridiculous, but here we go: Yet another FSAL refactor. If my count is right, this is the fourth one since I created the FSAL. It's almost as if my conceptions about how to handle the file system were completely wrong when I started! :D

But hey, I learned things.

Anyways, this PR has a simple to explain, yet difficult to achieve goal:

This PR aims to decouple the file discovery step from the indexing step.

In other words, this PR aims to change the entire app in such a way that parsing files (which can be extremely time-consuming) can be deferred. Essentially, it restores the app to a prehistoric time before I started to cache some information from the files, when the file manager would only show you a file name, and no additional information for directories. Essentially, a time before even the invention of the .ztr-directory, hence long before 1.0.

However, if a file or folder actually has been indexed at some point, it still should show all info as we all have come to know and enjoy.

Why?

I have been prompted to do this, since more and more people are complaining that opening new workspaces or the app can take an excruciating amount of time. Indeed, as we stored and collected more and more info, the indexing process became slower and slower. At some point, I added a splash screen to show users the progress. However, in my infinite wisdom I didn't account for one particular group: Avid Markdown users who already possess huge amounts of Markdown files, and who would first have to load their files into workspaces. Until now, the entire design of Zettlr only accounts for recurring users (so that, when the FSAL cache is cleared, they will see the splash screen), or for new users who are entirely virgin to the concept of Markdown (hence, loading workspaces initially doesn't take a lot of time because they're essentially empty).

So, what I aim to do with this PR is to decouple the mere file discovery ("Does a file exist?") from the indexing (= extracting data by parsing the file content). File discovery should still be fast (since we're only querying the file system) so that even if you load a humongous amount of files, we can at least already start displaying files in your sidebar, even if the titles and headings have not yet been loaded. This may induce some flickering, as the index builds in the background, but since this should still be a fairly rare operation (unless I start releasing a new version every week), this should be fine.

Changes

This PR is conceptualized in three steps:

  • Phase One (done when opening the PR): Adjust the FSAL descriptor types so that all consumers of FS descriptors expect both "complete" and "incomplete" descriptors. "Complete" descriptors are already parsed, whereas "incomplete" ones only come with the bare-bones data. This means that the consumers of descriptors don't care whether they get "full" descriptors or incomplete ones. GUI elements have been disabled for incomplete descriptors where necessary (e.g., you can't change the sorting of a folder if the directory descriptor hasn't yet been loaded.)
  • Phase Two: Complete remove the indexing step from the boot sequence, and the loading step. This will essentially make the app look very bland, but this allows us to double-check that everything still works.
  • Phase Three: Re-implement indexing, but as a background process that essentially just works through a queue in a worker thread.

Nota bene: The only real change in the main process would be that the FSAL would no longer force-parse files if they didn't exist in the cache, and instead return bare-bones descriptors.

Additional information

The end result of this PR should be in a perfect world:

  • Make the splash screen unnecessary. App boot should be (almost) instantaneous. The largest amount of the loading is indeed the indexing, and if that is deferred, we can already show the app much, much quicker. Coincidentally, this will also shave off about 5 MB of the binary (each window has about 5 MB of minimal bundle size, which is ridiculous, but hey, insert some bad JavaScript takes here)
  • Load and unload workspaces almost instantaneously. Users can load humongous folders as workspaces, and begin working with them immediately, as the app indexes the data.
  • Indexing the data can then finally be properly conceptualized as a LRT (see my other PR), depending on whether there are more than, idk, 10 files to index.
  • Last but not least — and this is something I'm personally excited about — this also means that we can infer a lot more data from the files that we can then use. We don't have to be time-efficient anymore, but can tilt the scale more towards quality. (I'm planning for a semantic search in the future, which would involve a bit of very heavy computations during indexing, but that's for the future.)

Tested on: macOS Tahoe 26.2

@nathanlesage
Copy link
Member Author

Most important finding of this refactoring endeavor so far: TypeScript types are difficult, and I have implemented the FSAL descriptors a tad brittle :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants