Why aren't constructors exported out of Internal libraries

[madgen]

Hi, I'm trying to derive Lift instances for HashMap and HashSet as done in th-lift-instances for Set and Map from containers (specifically here) so that I can use them with Typed Template Haskell.

However, deriving Lift as an orphan instance requires access to the constructors which I'd expect from the Internal version of the HashSet and HashMap modules but that is not the case. HashMap partially exposes it, however, it uses types such as HashMap.Array.Array which doesn't expose its constructors.

Is there a reason for this?

[treeowl]

We should fix this. We should also offer a Lift instance. That said, I'm rather curious how you came to need that instance.

[madgen]

@treeowl I'm staging a regular expression compiler. I switched to HashMap/HashSet for efficiency, but lost the ability to stage without doing horrible hacks. The following works if I monomorphise all my type signatures for example:

instance (Lift k, Lift v) => Lift (M.HashMap k v) where
  lift m = [| M.fromList m' |]
    where
    m' = M.toList m
  liftTyped = unsafeTExpCoerce . lift

I'm hoping if I instead just use DeriveLift, I can keep being polymorphic in my signatures and avoid cryptic TH errors.

[treeowl]

We won't be able to "just" derive Lift, because we'll have to explain how to lift the arrays. Not a big deal; any code we're missing can be copied from primitive. Your orphan workaround should probably be written

instance (Lift k, Lift v) => Lift (M.HashMap k v) where
  liftTyped m = [|| M.fromList m' ||]
    where
      m' = M.toList m

to avoid the ugly and unnecessary unsafety.

[treeowl]

@madgen I've opened a PR. As it is right now, the internal arrays are realized as lists and converted. I imagine that will lead to more compact code than trying to produce code to actually construct the arrays by poking elements into arrays one by one. However, I'd be happy to experiment with alternative approaches if you find it's taking too long to build the maps at run-time.

[phadej]

As I commented in the PR #343, it might well be that using containers with DeriveLifted instance will be faster then HashMap which cannot be lifted into static structure.

See https://well-typed.com/blog/2020/06/th-for-static-data/

Please benchmark whether HashMap would be faster, I'm be curious to know! GHC devs would appreciate a real world motivated push for https://gitlab.haskell.org/ghc/ghc/-/issues/16944

[madgen]

@phadej compared to the DeriveLift + containers from th-lift-instances, my stupid hack with HashMap Lift instance + monomorphisation of the staged function type signatures led to a 5x improvement in my microbenchmarks of the staged code. However, to be sure that monomorphisation of signatures didn't have any effect, I should try the containers variant with monomorphisation as well.

Unfortunately even with monomorphisation, I couldn't get HashSet to be staged, but it is WIP. I can update you once (if) I get that working too.

Note that the performance of unstaged versions of both container and unordered-container implementations are nowhere near the staged ones. There's a 100x perf difference.

I'll try benchmarking with the Lift instances you provided as well. Thank you.