OpamFile.OPAM API: No longer assume that internal opam repositories are directories #6679

kit-ty-kate · 2025-09-09T10:02:35Z

Extracted from #6625
Queued on

To allow to change the type of repository from a directory to something else, we need some of the internal functions that expects to be dealing with a directory to be rearranged a little.
In this PR, we do so for OpamFile.OPAM.get_extra_files which currently takes a function returning a directory (repository root), to return instead the list of files and content that users of the function actually expect.

rjbou · 2025-09-18T10:44:50Z

tests/reftests/dot-install.test

 === install 1 package
  - install lot-of-files ~dev
 SYSTEM                          mkdir ${BASEDIR}/OPAM/inst/.opam-switch/build/lot-of-files.~dev
-SYSTEM                          copy ${BASEDIR}/OPAM/repo/default/packages/lot-of-files/lot-of-files.~dev/files/lot-of-files.install.in -> ${BASEDIR}/OPAM/inst/.opam-switch/build/lot-of-files.~dev/lot-of-files.install.in


To investigate...

It's simply that OpamSystem.write has no log

maybe todo: have a look why action-disk test is not affected

Added logging for read & write. To extract in its own PR

rjbou · 2025-09-18T11:07:50Z

tests/reftests/extrafile.test

-<default>/no-checksum.1: Errors.
-             error  3: File format error in 'extra-files' at line 11, column 2: expected [file checksum]
-# Return code 1 #
+<default>/no-checksum.1: Passed.


This change is due to the fact that now we take the opam file from repo state instead of the plain file from opam internal repo copy. Stored opam file in repo state removes the mismatching extra files, so there is no longer the error (extra-files field removed).

We may change --package behaviour to be used only with pinned package...

From dev meeting: we decided to put back the mechanism to retrieve the file from repo root. At first, we wanted to avoid it because extracting a file from a tarred repository is a consuming operation.
For the usages that are done in this PR (opam lint --package and opam pin with version and / or edit), it is conceivable to have this consuming operation as they are not operation that are done often, and it permit to keep the same behaviour (respectively linting without preprocessing and preserve format of opam files).

rjbou · 2025-09-18T11:13:30Z

src/state/opamRepositoryState.ml

 let get_repo_root rt repo =
  get_root_raw rt.repos_global.root rt.repos_tmp repo.repo_name

+let get_repo_files rt name dir =


Suggested change

let get_repo_files rt name dir =

(* We need to keep the argument as string to be able to have the

non resolved path to concatenate. *)

let get_repo_files rt name dir =

or use a basename

rjbou · 2025-09-18T11:13:59Z

src/state/opamPinned.mli

-
-(** Finds back the location of the opam file this package definition was loaded
-    from *)
-val orig_opam_file:


renamed and moved to switch state to fit its new specification

kept, cf #6679 (comment)

src/core/opamCompat.ml

master_changes.md

src/format/opamFile.ml

src/client/opamSwitchCommand.ml

src/state/opamSwitchAction.ml

src/state/opamUpdate.ml

kit-ty-kate · 2025-09-24T23:05:17Z

src/client/opamCommands.ml

-           let opam = OpamSwitchState.opam st nv in
-           match OpamPinned.orig_opam_file st (OpamPackage.name nv) opam with
-           | None -> raise Not_found
+           match OpamSwitchState.overlay_opam_file st name with


Something feels a bit off with the use of overlay_opam_file vs. the previous orig_opam_file, in here and with its use via ?format_from. I think we should probably avoid breaking compatibility on this and instead do something like copying the file to a temporary directory, and simply remove the ability for orig_opam_file to fetch things from the repository and only fallback to overlay if the original file isn't accessible. It was in OpamPinned after all

This will keep backward compat only for already pinned packages. Pinned package from repo won't have the format kept as now.

kit-ty-kate · 2025-09-24T23:09:36Z

tests/reftests/dot-install.test

 === install 1 package
  - install lot-of-files ~dev
 SYSTEM                          mkdir ${BASEDIR}/OPAM/inst/.opam-switch/build/lot-of-files.~dev
-SYSTEM                          copy ${BASEDIR}/OPAM/repo/default/packages/lot-of-files/lot-of-files.~dev/files/lot-of-files.install.in -> ${BASEDIR}/OPAM/inst/.opam-switch/build/lot-of-files.~dev/lot-of-files.install.in


It's simply that OpamSystem.write has no log

rjbou · 2025-10-08T18:11:55Z

tests/reftests/extrafile.test

 # Return code 31 #
 ### opam source escape-good-md5
-Successfully extracted to ${BASEDIR}/escape-good-md5.1
+[WARNING] Some errors extracting to ${BASEDIR}/escape-good-md5.1:


Ok, in fact it resolves an error that we had 👍

rjbou

Nice 4 hands job :D

…re directories Co-authored-by: Raja Boujbel <[email protected]>

kit-ty-kate added this to the 2.5.0~alpha1 milestone Sep 9, 2025

kit-ty-kate added PR: QUEUED Pending pull request, waiting for other work to be merged or closed PRIORITY AREA: API labels Sep 9, 2025

This was referenced Sep 9, 2025

[API] Abstract over repository roots to simplify the addition of new repository formats #6680

Open

[WIP] Improve performance of opam update/init by changing the structure of the internal http opam repositories (use the tar.gz as-is) #6625

Draft

rjbou force-pushed the get_repo_files branch 2 times, most recently from a1a2d47 to 4d6c1c5 Compare September 18, 2025 11:06

rjbou reviewed Sep 18, 2025

View reviewed changes

rjbou force-pushed the get_repo_files branch from 4d6c1c5 to 40d71f6 Compare September 23, 2025 15:20

rjbou added PR: NEEDS UPDATE PR: WAITING FOR REVIEW and removed PR: QUEUED Pending pull request, waiting for other work to be merged or closed labels Sep 23, 2025

kit-ty-kate commented Sep 24, 2025

View reviewed changes

kit-ty-kate requested a review from rjbou October 1, 2025 15:28

rjbou force-pushed the get_repo_files branch from 3d3c2e2 to 8172625 Compare October 8, 2025 18:10

rjbou reviewed Oct 8, 2025

View reviewed changes

compat: add Lazy.map_val

75043fd

rjbou force-pushed the get_repo_files branch from 8172625 to 8c913af Compare October 9, 2025 17:10

kit-ty-kate force-pushed the get_repo_files branch from 8c913af to ae70eeb Compare October 9, 2025 18:46

kit-ty-kate removed the PR: NEEDS UPDATE label Oct 9, 2025

kit-ty-kate force-pushed the get_repo_files branch from ae70eeb to 59ffd38 Compare October 9, 2025 20:49

kit-ty-kate changed the title ~~Change the OpamFile.OPAM API to suite different formats of repositories~~ OpamFile.OPAM API: No longer assume that internal opam repositories are directories Oct 9, 2025

kit-ty-kate force-pushed the get_repo_files branch from 08edd0b to 6eb5032 Compare October 10, 2025 16:45

kit-ty-kate requested a review from rjbou October 10, 2025 16:45

kit-ty-kate force-pushed the get_repo_files branch from 6eb5032 to 7fd7117 Compare October 10, 2025 16:47

rjbou approved these changes Oct 10, 2025

View reviewed changes

rjbou removed the PR: WAITING FOR REVIEW label Oct 10, 2025

rjbou added the STATE: READY TO MERGE label Oct 10, 2025

rjbou assigned kit-ty-kate and rjbou Oct 10, 2025

rjbou and others added 4 commits October 10, 2025 18:40

Add logging for write & read files

86bac20

reftest: complete action-disk.test with the behaviour of extra-files

ceff73d

reftest: complete action-disk.test with the behaviour of extra-source

7a504c0

OpamFile.OPAM API: No longer assume that internal opam repositories a…

1c71f5e

…re directories Co-authored-by: Raja Boujbel <[email protected]>

kit-ty-kate force-pushed the get_repo_files branch from 7fd7117 to 1c71f5e Compare October 10, 2025 17:40

kit-ty-kate merged commit 6c51e0e into ocaml:master Oct 10, 2025
46 checks passed

kit-ty-kate deleted the get_repo_files branch October 10, 2025 19:38

rjbou removed PRIORITY STATE: READY TO MERGE labels Oct 17, 2025

OpamFile.OPAM API: No longer assume that internal opam repositories are directories #6679

OpamFile.OPAM API: No longer assume that internal opam repositories are directories #6679

Uh oh!

Conversation

kit-ty-kate commented Sep 9, 2025 • edited by rjbou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rjbou left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kit-ty-kate commented Sep 9, 2025 •

edited by rjbou

Loading

rjbou left a comment •

edited

Loading