Skip to content

Improve cache mechanism when modulepath contains a large amount of non-modulefiles #562

Closed

Description

@xdelaruelle

As a follow-up of a discussion on the modules-interest mailing-list, cachebuild process takes infinite time and memory when a modulepath contains a very large amount of non-modulefiles (millions).

This is due to the recording in the cache of invalid modulefiles with their path and the error message obtained when checking if these files are modulefiles.

It may be simpler to avoid recording such information in the cache and treat access to these non-modulefiles as a file not found error.

Activity

  1. xdelaruelle commented on Jan 15, 2025

    @xdelaruelle
    CollaboratorAuthor

    New commands modulepath-ignore/modulepath-only, described on #561, may also be used to avoid recording information regarding files that are set to be avoided.

    Copying specification written in #561 here.

  2. added this to the 5.7.0 milestone on Feb 18, 2026
  3. xdelaruelle commented on Aug 18, 2026

    @xdelaruelle
    CollaboratorAuthor

    New modulefile commands may be introduced to define the path pattern to ignore or to only take into account.

    A new modulepath-ignore command may take a list of file path pattern as arguments (1..N). When walking down the content of modulepath directory, entries that match any of these patterns are ignored (files are not evaluated to check if they are modulefiles, directories are not searched down to find other modulefiles)

    In the issue described on the mailing-list, modulefiles are a few files among large amount of other kind of files (installed software files). In this case another command may be useful, modulepath-only, to define a list of path pattern (1..N) to only take into account to check for modulefiles. When modulepath-only is used, patterns defined via modulepath-ignore are ignored as the logic is not to accept all ignore some but to ignore all accept some.

    Preferably, such commands should be set in the top .modulerc of modulepath, as it determines how to walk through its content. If used in a .modulerc file found deeper, current directory will be added to the pattern. For instance, if /path/to/modpath/modname/.modulerc defines foo* pattern to ignore, it will be converted into /path/to/modpath/modname/foo*.

    If user mentions a file that matches a modulepath-ignore pattern or does not match modulepath-only pattern when set:

    • it should get a file path not found error
    • such files should be excluded from generated .modulecache
  4. xdelaruelle commented on Aug 18, 2026

    @xdelaruelle
    CollaboratorAuthor

    One use case for the ignore/only feature:

    The use case I was hoping to address with the new feature is the following: For the sake of modularity we have some files in module directories that are e.g. read by modulefiles but which are not valid modulefiles themselves. Currently these are hidden files (i.e. their name starting with a dot, e.g. '.some_function.tcl') so that the module command does not 'see' them.

    When managing the module directory tree these hidden files sometimes cause confusion. If we were able to define a modulefile name filter (e.g. ignore all *.tcl files) we could make such files regular files w/o any impact on the traversing of the modulepath. Hence the idea of using the gitignore syntax as this would enable ignoring both (classes of) files as well as directories. Or explicitly including them via negation.

    Additional design information:

    • multiple pattern may be specified on one command and command may be defined several times
    • this will give a list of all pattern to ignore (or only include)
    • as soon as a "modulepath-only" command is set, "modulepath-ignore" patterns are all ignored
    • patterns are prefixed by the location of the modulerc where they are defined
      • for instance "modulepath-ignore 1*" in "modpath/foo/.modulerc" produces the "modpath/foo/1*" ignore pattern
    • findModulesFromDirsAndFiles procedure will be updated to check each element found in a directory to determine if it should be taken into account or ignored
  5. xdelaruelle commented on Aug 22, 2026

    @xdelaruelle
    CollaboratorAuthor

    Final design: one new modulefile command introduced: modulepath-ignore. This command accepts same patterns than .gitignore files (including negated patterns).

  6. added a commit that references this issue on Aug 22, 2026
    ee149f9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions