Requests for comments: how does opam-repository scale?

[hannesm]

I've observed over the years that there's the sentiment that "no package shall be removed from opam-repository" (I still don't quite understand the resoning behind it -- maybe it is lock files that fail to register the git commit of the opam-repository? Maybe it is for some platforms that require the opam-repository for removing opam packages?

So, I'd like to raise the question why this sentiment exists. Answers (or pointers to arguments) are highly welcome.

Why am I asking this? Well, several years back Louis worked on "removing packages that aren't ever being installed" (with the reasoning that if foo in version 6.0.0 and in version 6.0.1 (with the very same dependencies) are available, 6.0.1 is always chosen (which makes sense due to it may very well be a bugfix).

Now, I also observe that rarely, really rarely old package releases get bumped their minor version -- i.e. the "long term support" of opam packages does not really exist (happy if you prove me wrong): bug fixes are incorporated with new features and API changes, and so new (major) versions are released.

Taking a step back, it is pretty clear that collecting more and more packages will lead to larger amount of work for the solver (which needs all the packages being parsed, and then find a solution). This means that the solver needs to improve speed-wise roughly every year. This is rather challenging, and in the end leads to opam not being very usable on smaller computers (Raspberry PI, or even tinier computers...).

Also with carbon footprint in mind, I fail to see why opam-repository may not delete packages. In the end, it is a git repository -- if you wish to install compiler in an arcane version, it is fine to roll back your opam-repository git commit to an arcane commit that fits nicely. The amount of work for the opam-repo CI could as well be minimized by removing old opam packages (since a revdep run will be much smaller).

Please bear with me if you've already answered this, and have written up the design rationale (and maybe the answer how opam-repository will scale). Comments and feedback is welcome. Once I understand the reasoning, it'll be much easier for me to figure out how to move forward. Thanks a lot. //cc @AltGr @avsm @kit-ty-kate @mseri @samoht

[hannesm]

As a first observation: the opam repository is a git repository, so anyone who wants to install old releases can always use an old commit.

One idea I discussed with @mro is to add an expiration date on each opam package when publishing, i.e. "this is being maintained for 3 months". I was initially skeptical, but think this is a fine idea since various contributors (authors) and companies have different release cycles, and for me personally code that I don't use and touch for several years should either be released as stable (with a 1.x) and expiration of infinite; or will bitrot.

And for reference, I found these PRs and their discussion pretty insightful:

• Big new year cleanup #11194
• Remove uninstallable packages #11400
• Cleanup 'obsolete' packages #11559

Especially the comment #11400 (comment) is worth reading, where different people have different opinions (in respect to whether it "is good for reproducibility to remove packages"). And note to myself, there's opam admin check --obsolete.

[hannesm]

Looks like nobody else cares (and Louis cared earlier, as outlined in closed issues and pull request). Closing.

[mseri]

I do actually care, but I don't have a good solution yet and, worse, there is not yet a real consensus on what to do. I hope we will settle on some periodic stable releases and an archive repository where old packages go to rest, or some other compromise that will make things slimmer.

[mseri]

If you ask me we should leave this open as an opportunity to get more ideas. There was also a discuss (or even more than one) thread where we had a discussion about this. If I can find it, I'll link it

[samoht]

I also care.

But what I would like to see before deleting packages is a process to move these deprecated/unmaintained packages to a separate repository. Keeping them around is really helpful when you need to test large-scale language features (for new release, but also to check langage statistics for old releases).

I think the main blocker for this right now is to have the appropriate tooling (for instance to split an existing opam-repo easily).

[hannesm]

I doubt this is a tooling issue to be honest, since there's opam admin check --obsolete around (surprise: it may need some improvements), as well as opam admin check --installability.

It feels more like a policy decision about what should opam-repository achieve - together with what compilers to support (see related issues #24868 ocaml/opam#5748 -- but also ocaml/infrastructure#87 ocaml/infrastructure#48).

But what I would like to see before deleting packages is a process to move these deprecated/unmaintained packages to a separate repository.

What prevents you from adding a tag before deletion of anything in this repository, and establish a (e.g.) quarterly routine on what to delete (i.e. have a quarterly tag, 2023Q1..Q4)?

Maybe it is just me, using a ~10 year old laptop, who suffers from this issue -- but it looks like the "opam.ocaml.org update job" is having quite some trouble (related to the amount of packages).

To have a concrete proposal that can be discussed:

• Let's have a quarterly delete job. Each deletion will be preceeded by a git tag (so someone interested in earlier releases can just opam repo add old-stuff https://github.com/ocaml/opam-repository.git#2023Q4).
• For Q4/2023, there will be a tag, and only support OCaml 4.08 (see https://discuss.ocaml.org/t/raising-the-minimum-tested-version-of-ocaml-to-4-05-or-4-08-from-4-02/12464) and opam 2.1 (to be discussed what the minimum opam version should be, certainly 2.0)
• Also, all "available: false" packages will be removed
• opam admin check --installabe will be executed and evaluated whether the output list should be removed
• The upside is: smaller repository, fewer packages -> fewer work on all clients
• Additionally, for each package, only the latest patch version will be retained: i.e. if there's a package with versions 0.1.0, 0.1.1, 0.2.0, 0.2.1, 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1 - only 0.1.1, 0.2.1, 1.0.2, 1.1.1 are kept.
• Exemptions for the above rule are the following set of packages: ocamlformat
• After removal, the output of opam admin check --installable will be evaluated
• If there's breakage, old packages can be revived (and added to the above set of exemptions)
• since we now have a path how packages are removed, any maintainer/author can opt in to add their package (+version) to the set of "to-be-deleted" packages (at the end of each quarter)

Yes, this will break "opam lockfiles" - in theory at least. In practise, feel free to point to hyperlinks of lockfiles that are broken by such a mechanism (and yes, I really really think these lockfiles need to be fixed to include the repository commit so that they are sustainable).

How can we "discuss" such a policy? Is this issue good for it? How can we reach something that we then act on? Or will we just sit down and wait until it became so annoying that nobody is using it anymore?

[kit-ty-kate]

On opam’s side we’re thinking about adding a compressed format for the repository after 2.2.0 (ocaml/opam#5648). Such format would eliminate the issue for opam update at least.

For the solver/clutter/git-growth/… though the problem still remains and removing packages could be a solution. It is also to note that we currently have nobody knowledgeable enough to work on the solver on a regular manner.

[samoht]

What I do not really like with tags and saying that you can always look back at the history to install old packages is that we are constantly improving the quality of the metadata of opam-repository. So when you use an old opam-repo, you loose these improvements unless you have a (custom) merge of this old repo with the live repo.

My ideal workflow would be to have layered repositories with a well-defined merge strategy - so our automated system could easily test old packages with recent metadata fixes.

[samoht]

So a few more details on what I have in mind. I'm describing an hypothetical worfklow - we can get there incrementally or not.

• We have 3 official opam repositories instead of one: unstable/stable/archive
• optionnaly, there's a new opam field x-opam-repository-status that contains unstable/stable/archive (if we have want the opam client to do something clever about those)
• optionanly, there's a new opam field x-opam-repository-expiry with an expiry date for a package
• every opam repository has a different policy:
  • unstable: no gatekeepers - a very light CI ("it lints and it builds on ubuntu+latest OCaml release, no x-platform checks, no revdeps"). Strong encouragement to have an expiration date for the packages
  • stable: same as today but with all the unstable/old/deprecated packages removed. The CI runs all of todays' checks, but just on the stable repo.
  • archive: all the old/deprecated packages. No (or very few) CI checks - archive packages are supposed to always build.
• the lifecycle of a package could be:
  • people can submit new packages to unstable and stable (these correspond to different level of CI/process checks)
  • optionally, there's a weekly/monthly bulk build of unstable packages with extra CI check - the ones that pass the extra CI checks (including revdeps with stable packages) are submitted in bulk to the stable repository
  • there's a weekly/monthly bulk build of packages "on the deprecation list", that we want to delete from stable and submit to archive. The CI runs with stable+archive. We need to fix the constraint before adding anything into archive (ie. archive packages should always build). This is super important when there's a new OCaml release (we should check that it doesn't break archive packages and if yes, add the new constraints). When the packages are added to archive, they are removed from stable
• we could host the 3 repositories as different branches in ocaml/opam-repository

What I am trying to get is some kind of guarantees that unstable/old/deprecated package metadata continue to somehow be maintained but without putting too much hassle on our CI infrastucture and on opam-repo gatekeepers. And I'm also keen to have a system where "submitting a simple package for fun" doesn't add more load to our system/process.

What do you think?

[arbipher]

I am new and started volunteer work in opam admin a few weeks ago. I haven't done much yet except for just attending a few weekly meetings. I shall have more time from this week. I am so glad to see this issue discussing the scaling problem. Here are some topics appearing in your discussions and my opinion.

Retiring old packages

I don't agree with retiring old packages directly. The direct reason is we cannot guarantee the packages will observe the plausible invariant from semantic versioning so that e.g. projects or libraries depending on 6.0.0 can always run without problems after changing dependency to 6.0.1. Reproducibility is such a painful problem.

However, I do agree that we should suggest to use a better alternative of a package version. No matter when users try installing a package with a non-optimal version, or opam (.opam) detects a non-optimal version is specified. But does it require the user to always have a most recent view of the latest opam-repo?

So my point is the package versioning suggests a good package choice but not reliable. That's the reason lockfiles are widely used.

Using GitHub as the backend

Another unclear point to me is when evaluating a design suggestion above, shall we assume that opam repo will stick to using GitHub (git) backend or is it also discussable? It will be helpful to consider whether a suggest is mainly for better design or a better implementation.

We have 3 official opam repositories instead of one: unstable/stable/archive

It has some debian flavor but I have to say using opam where there is just one official central repo is much easier than using ubuntu, with which I have to manually tune it the src-list for many times. And there do have some popular repo e.g. https://coq.inria.fr/opam/released. Does the repo kind (unstable/stable/archive) also apply to other repoes, or is it actually a package metadata?

CI burden, Package author burden, Opam repo admin burden

I would like to observe for some more time and hand on more work to figure out where the CI burden and repo admin burden comes from.

I think x-opam-repository-expiry may increase the Package author burden because it's too complicated and rarely possible to anticipate. I may lose some context or experience for this.

I agree with the fact that CI sometimes runs slower to block things. However, in my very limited observation, some CI issues due to the service itself could be improved (accidental error). Some CI errors are caused by package incompatiblity (essential error), and they do rescue users from suffering from this problem. That's one big achievement of the current admin team and opam repo workflow outperforms many other package management tools I have used, in my opinion. I don't think changing the CI arrangement can reduce this problem. Or maybe it helps, if more packages are staying the unstable without moving into the stable then the overall CI check can be reduced.

If we can offload some checking on the publishing side, some burdens are transferred from the CI/admin side to the author side.

p.s. In recap, I am very interested this problem for long time. I have some plan to survey (and surveyed a little, but not sort them out yet) more package managers especially for programming languages.

[mro]

My take on scaling cares about growth and reliability. Growth There is no such thing as eternal growth, so if we design for a liveable future, we should reach a saturation in size. IMO this is simple via expiration. It can be 10 years for "LTS" versions (e.g. the last one before a major/minor jump) but it has to be less than "eternity". There is no maintenance promised during these 10 years, just mere existence. A HTTP 200 instead a 404. The eol date should be in the metadata inside the tarball (and a http header). Who needs it longer (we come to that in a second), should be able to easily mirror/vendor single version tarballs. Reliability IMO we should embrace reproducible builds. This means we need reliably retrieveable sources incl. toolchain. Certain versions may do, it doesn't have to be any version. But once declared such (think 'best before Jan 1st 2040) it must to be available under the same url (or redirected). No need for 5 nines availability and neither bandwidth nor latency shouldn't matter too much, as you can easily cache/download stuff. Maybe signing the tarballs may be useful. (Web of trust, uh) In the end it's an url registry with signed, expiring entries. For reliability (legal, technical, moral) we shouldn't tie ourself to any third party concerning the build toolchain. GitHub may have been fashionable recently, but IMO the opam repository should be consumable without touching billionaire-run infrastructure let alone namespaces. I would want my project to be buildable by peers in Teheran, Moskow, Paris, Beijing and New York alike. And if not, it should be our decision (ocaml.org) and not a billionaire-clerk's. Monitoring may be helpful. Hope this makes sense, Marcus

…

On Tue, 12 Dec 2023 21:43:20 -0800 Weng Shiwei 翁士伟 ***@***.***> wrote: I am new and started volunteer work in opam admin a few weeks ago. I haven't done much yet except for just attending a few weekly meetings. I shall have more time from this week. I am so glad to see this issue discussing the scaling problem. Here are some topics appearing in your discussions and my opinion. **Retiring old packages** I don't agree with retiring old packages directly. The direct reason is we cannot guarantee the packages will observe the plausible invariant from _semantic versioning_ so that e.g. projects or libraries depending on 6.0.0 can always run without problems after changing dependency to 6.0.1. Reproducibility is such a painful problem. However, I do agree that we should suggest to use a better alternative of a package version. No matter when users try installing a package with a non-optimal version, or opam (`.opam`) detects a non-optimal version is specified. But does it require the user to always have a most recent view of the latest opam-repo? So my point is the package versioning suggests a good package choice but not reliable. That's the reason lockfiles are widely used. **Using GitHub as the backend** Another unclear point to me is when evaluating a design suggestion above, shall we assume that opam repo will stick to using GitHub (git) backend or is it also discussable? It will be helpful to consider whether a suggest is mainly for better design or a better implementation. > We have 3 official opam repositories instead of one: unstable/stable/archive It has some debian flavor but I have to say using opam where there is just one official central repo is much easier than using ubuntu, with which I have to manually tune it the src-list for many times. And there do have some popular repo e.g. `https://coq.inria.fr/opam/released`. Does the repo kind (unstable/stable/archive) also apply to other repoes, or is it actually a package metadata? **CI burden, Package author burden, Opam repo admin burden** I would like to observe for some more time and hand on more work to figure out where the CI burden and repo admin burden comes from. I think `x-opam-repository-expiry` may increase the _Package author burden_ because it's too complicated and rarely possible to anticipate. I may lose some context or experience for this. I agree with the fact that CI sometimes runs slower to block things. However, in my _very limited observation_, some CI issues due to the service itself could be improved (_accidental error_). Some CI errors are caused by package incompatiblity (_essential error_), and they do rescue users from suffering from this problem. That's one big achievement of the current admin team and opam repo workflow outperforms many other package management tools I have used, in my opinion. I don't think changing the CI arrangement can reduce this problem. Or maybe it helps, if more packages are staying the unstable without moving into the stable then the overall CI check can be reduced. If we can offload some checking on the publishing side, some burdens are transferred from the CI/admin side to the author side. p.s. In recap, I am very interested this problem for long time. I have some plan to survey (and surveyed a little, but not sort them out yet) more package managers especially for programming languages. -- Reply to this email directly or view it on GitHub: #23789 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

-- mro ***@***.***>

[hannesm]

@arbipher you say "lockfiles are widely used" -- would you mind to elaborate on that? From what I can see, some people use lockfiles. Do you have a specific insight in which domain "lockfiles are widely used"?

There are some loud voices on "lockfiles" that prevent any progress on pressing issues since "lockfiles may break".

FWIW, if you're keen on reproducibility - lockfiles don't achieve this at all. You'll need to put some more effort (and include all the metadata into your build-information, together with environment variables and system packages that have been used). If you're fine with "sometimes you may be able to reproduce the same version / behaviour", obviously lockfiles are fine.