Codecov Report

❌ Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.10%. Comparing base (96be4fb) to head (e98864c).
⚠️ Report is 35 commits behind head on master.

@@            Coverage Diff             @@
##           master    #1131      +/-   ##
==========================================
- Coverage   28.93%   28.10%   -0.84%     
==========================================
  Files         266      266              
  Lines      178909   202414   +23505     
  Branches   178909   202414   +23505     
==========================================
+ Hits        51774    56889    +5115     
- Misses      54496    64159    +9663     
- Partials    72639    81366    +8727     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This is somewhat related to #1121, which completely overhauls Tectonic's bundles (and thus conflicts with the changes here)

Notably, #1121 adds a new bundle format, and setting a variable for just the url prefix won't work. It may be a better idea (even without extra bundles) to have the variable contain the whole URL.

Also, I don't think I understand the difference between the LOCKED and PREFIX variables. Is there any reason to have two? I think one variable that overrides the default bundle url should be enough.

A few more notes about the issue in #1002:

• Should this be a cli option or a env variable?
• Why does tectonic new touch the network in the first place? I'd argue that the command should create a skeleton project completely offline (with the default url in tectonic.toml), which can be changed later by the user. This is clearer and doesn't need any extra config options.

Hi @rm-dr! Thank you for the comment!

This is somewhat related to #1121, which completely overhauls Tectonic's bundles (and thus conflicts with the changes here)

Wow amazing! I didn't know this before! This is a huge change though. Are you aware of #1122? I think being a member would probably make it easier to land these changes.

Notably, #1121 adds a new bundle format, and setting a variable for just the url prefix won't work. It may be a better idea (even without extra bundles) to have the variable contain the whole URL.

Also, I don't think I understand the difference between the LOCKED and PREFIX variables. Is there any reason to have two? I think one variable that overrides the default bundle url should be enough.

Currently, the bundle url depends on the format_version:

Should this be a cli option or a env variable?

They are different! Let me clarify:

• the env variable provides a build time override for distro packagers, such that the tectonic bundle they distributed will match the biber version they distributed. This is related to Biblatex fails due to version mismatch #893.
• the --web-bundle option is for runtime override by the end user. This relates to Provide options for V2 cli to override the default bundle_url #1002.

I think for the runtime override, we should only provide the CLI option, because env variables are more implicit and thus reduce reproducibility. For build time, however, I think env variables are the only way to pass around the customized build constants, so I have to use it.

Why does tectonic new touch the network in the first place? I'd argue that the command should create a skeleton project completely offline (with the default url in tectonic.toml), which can be changed later by the user. This is clearer and doesn't need any extra config options.

AFAIK, tectonic -X new and init:

• takes a generic web bundle link https://relay.fullyjustified.net/default_bundle_v33.tar,
• redirects it to a version-locked one, such like https://data1.fullyjustified.net/tlextras-2022.0r0.tar,
• and finally pins it down in the Tectonic.toml file.

I think this behavior is nice as it maximizes reproducibility. There are some suggestions in issues to put the locked url ...tlextras-2022.0r0.tar in a dedicated Tectonic.lock file. When and if that happens, tectonic -X new and init can be completely offline, as it only needs to write the unlocked link ...default_bundle_v33.tar.

$ tectonic -X init
note: "version 2" Tectonic command-line interface activated
note: creating new document in this directory ($HOME/apps/tectest)
note: connecting to https://relay.fullyjustified.net/default_bundle_v33.tar
note: resolved to https://data1.fullyjustified.net/tlextras-2022.0r0.tar

$ cat Tectonic.toml
[doc]
name = "tectest"
bundle = "https://data1.fullyjustified.net/tlextras-2022.0r0.tar"
...

Note how the url gets redirected and locked down.

I really like the idea of having an environment variable for this, but I also don't understand why we need 2 environment variables, and not only 1.

Besides that, I also support having this environment variable read in runtime and not build time, as this would also help users to test different bundles in a different way, I think there is no advantage in terms of reproducibility between reading this environment variable in runtime vs build time; since Tectonic from different distributions may have different environments anywhere.

I also think it'd be fair for Tectonic to not guarantee reproducibility in case it reads a dedicated environment variable, and there's a conflict with the URL in the Tectonic.toml file. In such a case I think it'd be best to simply print a big warning, saying that we used the URL from the environment variable and not the the URL from Tectonic.toml file.

Hi @doronbehar! Thank you very much for the comment!

1. I am absolutely okay with a single env variable such as TECTONIC_WEB_BUNDLE_URL. As explained above, the PREFIX variant is only introduced for backward compatibility and ease of mirroring. If we decide that it is confusing, I can easily remove it!

2. When it comes to runtime, I would still prefer explicit flags such as --web-bundle, which can be overridden by a second --web-bundle. This is not yet the current behavior, and I can look into fixing it it has been fixed with the latest commit in this PR.

I think there is no advantage in terms of reproducibility between reading this environment variable in runtime vs build time; since Tectonic from different distributions may have different environments anywhere.

Yes, this is true in reality, but ideally, I would like Tectonic to not read any environment variables at all (at runtime), by design. It's like the new flake-y nix no longer reads the NIXPKGS_* env variables; sure, it makes things a bit more inconvenient, but I like the purity in the UI.

I know @pkgw is probably busy, but since this is now more than a bugfix and related to the UI design, I would very much like to hear your opinion, and I can fix the code accordingly!

2. When it comes to runtime, I would still prefer explicit flags such as --web-bundle, which can be overridden by a second --web-bundle. This is not yet the current behavior, and I can look into fixing it.

I did noticed that the 2nd commit of this PR as of currently is referring to this? But not completely? Maybe it'd be easier for upstream to digest this change only in a PR of its own.

They are different! Let me clarify:

Ah, I see. We really do need a compile-time option for this.
The two-link config is a bit messy, but I see why it's necessary...

and I don't think I know a better way to do this, if we want easy bundle updates through a "default" redirect.

Following the suggestion of @doronbehar I have moved the CLI changes to a separate PR #1132. These two PRs are both related to the web bundle interface, but they are technically independent, so it would be easier to review this way.

I've merged #1132, so that this PR becomes a lot smaller.

It does seem reasonable to me to have the default bundle URL be build-time configurable, for specialized applications.

My only comment upon first glance is one of the evergreen ones: these changes should come with corresponding documentation somewhere!

My only comment upon first glance is one of the evergreen ones: these changes should come with corresponding documentation somewhere!

Yes indeed! But since #1132 (run time web bundle override) has been merged, this one is no longer a blocking issue for me 😆 But I will try to document it sometime!

The top level bundle override tectonic --web-bundle -X ... broke after the bundle work:

• Bundle rework #1218

... so we need this again for downstream reproducible packaging / distribution of tectonic. I have added the relevant documentation in the tectonic book.

It would be great if this patch is considered before the next release; that would make downstream packaging much easier 😆 Thank you! @rm-dr @CraftSpider @pkgw

Update:

• added a new commit that prints a message when then bundle variable is set; this is good for debugging
• the test failure seems unrelated; in particular it didn't happen for the previous commit rebased and CI happy now.

I'll have to take some time to familiarize myself with our bundle code, but I'll try to take a look at this.

I'd remove this whole build script, and just do an unwrap_or(TECTONIC_BUNDLE_PREFIX_DEFAULT) in lib.rs. The environment variables will be passed through to rustc by default.

This is now done! I added a commit that moves the changes to lib.rs. Note that:

• Now I remember the reason that I chose to put the variables in build.rs instead of lib.rs is for discoverability: back then I have not yet written docs for these new compile time env vars, so I want it to be in a more visible place. However, docs are now included in the PR, so I guess it's easier for them to reside in lib.rs
• In the final commit I added some tests for these environment variables. Since we are testing compile time behaviors I created a mini sub crate under /tests/bundle_env_overrides_test and it's compiled by tests/bundle_env_overrides.rs to verify these behaviors.
  • The test turns out to be a bit too complicated than I would like it to be, so feel free to drop it if it feels unnecessarily complex!

Thank you for keeping up on this after a whole year and a half of waiting - hopefully a 0.16 release can be cut soon.

I should thank you for taking care of the project! Once I experienced Tectonic, I don't want to go back to texlive anymore, so I am just trying my best to help it live on 😆

There may be a way to simplify it with the ui_test crate or similar in the future, but for now I think this is good to merge. It works on all our many CI setups.

I didn't know about ui_test, I will investigate if it can be simplified with that!