extra-deps' clones as many times the repository as there are subdirs

[blackheaven]

General summary/comments (optional)

When I specify, in extra-deps, a repository with many subdirs, the repository is clone many times.

Steps to reproduce

Build a project with this package.yml part:

extra-deps:
- git: https://github.com/theam/aws-lambda-haskell-runtime.git
  commit: 6aef017c1621c828a47dc98eae12d84818a8d45c
- git: https://github.com/brendanhay/amazonka.git
  commit: 020bc7bde47bb235e448c76088dc44d6cec13e9b
  subdirs:
    - gen
    - core
    - test
    - examples
    - amazonka

Expected

I expect only one cloning.

Actual

The repository is cloned once by subdir.

Partial output

Stack version

$ stack --version
Version 2.3.3, Git revision cb44d51bed48b723a5deb08c3348c0b3ccfc437e x86_64 hpack-0.33.0

Method of installation

curl -sSL https://get.haskellstack.org/ | sh

[mattaudesse]

Thanks for reporting @blackheaven.

Just a note: I've given this the type: enhancement label rather than type: bug because it's surprising and sub-optimal (but not workflow-breaking) behavior.

[qrilka]

Pantry caches repos but it looks like it does it per subdirectory - https://github.com/commercialhaskell/pantry/blob/master/src/Pantry/Storage.hs#L236

[blackheaven]

Nice catch.

I am curious, what are the rationals for that strategy?

[qrilka]

@snoyberg could you share them?

[snoyberg]

The model of pantry is that a tree represents a single package. This allows caching only what is needed for that package, and more frequent cache hits.

IMO, the best way to improve this is to tweak pantry to aggregate all packages coming from the same repo and collect all the trees at once. I don't think it would be too invasive a change, and shouldn't require any changes at all to the overall model.

[blackheaven]

It makes sense, thank you for that explanation.

[hasufell]

Another data point is cardano-wallet, which takes ~30 minutes just to clone all repos on my machine (1mb/s, say nothing 😢). Cabal takes about 5.

[hasufell]

Fixed via:

• Fix cloning per subdirectory wrt #5411 #5624
• Improve and expose fetchRepos/fetchReposRaw pantry#41