You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve cache mechanism when modulepath contains a large amount of non-modulefiles #562
As a follow-up of a discussion on the modules-interest mailing-list, cachebuild process takes infinite time and memory when a modulepath contains a very large amount of non-modulefiles (millions).
This is due to the recording in the cache of invalid modulefiles with their path and the error message obtained when checking if these files are modulefiles.
It may be simpler to avoid recording such information in the cache and treat access to these non-modulefiles as a file not found error.
New commands modulepath-ignore/modulepath-only, described on #561, may also be used to avoid recording information regarding files that are set to be avoided.
New modulefile commands may be introduced to define the path pattern to ignore or to only take into account.
A new modulepath-ignore command may take a list of file path pattern as arguments (1..N). When walking down the content of modulepath directory, entries that match any of these patterns are ignored (files are not evaluated to check if they are modulefiles, directories are not searched down to find other modulefiles)
In the issue described on the mailing-list, modulefiles are a few files among large amount of other kind of files (installed software files). In this case another command may be useful, modulepath-only, to define a list of path pattern (1..N) to only take into account to check for modulefiles. When modulepath-only is used, patterns defined via modulepath-ignore are ignored as the logic is not to accept all ignore some but to ignore all accept some.
Preferably, such commands should be set in the top .modulerc of modulepath, as it determines how to walk through its content. If used in a .modulerc file found deeper, current directory will be added to the pattern. For instance, if /path/to/modpath/modname/.modulerc defines foo* pattern to ignore, it will be converted into /path/to/modpath/modname/foo*.
If user mentions a file that matches a modulepath-ignore pattern or does not match modulepath-only pattern when set:
it should get a file path not found error
such files should be excluded from generated .modulecache
The use case I was hoping to address with the new feature is the following: For the sake of modularity we have some files in module directories that are e.g. read by modulefiles but which are not valid modulefiles themselves. Currently these are hidden files (i.e. their name starting with a dot, e.g. '.some_function.tcl') so that the module command does not 'see' them.
When managing the module directory tree these hidden files sometimes cause confusion. If we were able to define a modulefile name filter (e.g. ignore all *.tcl files) we could make such files regular files w/o any impact on the traversing of the modulepath. Hence the idea of using the gitignore syntax as this would enable ignoring both (classes of) files as well as directories. Or explicitly including them via negation.
Additional design information:
multiple pattern may be specified on one command and command may be defined several times
this will give a list of all pattern to ignore (or only include)
as soon as a "modulepath-only" command is set, "modulepath-ignore" patterns are all ignored
patterns are prefixed by the location of the modulerc where they are defined
for instance "modulepath-ignore 1*" in "modpath/foo/.modulerc" produces the "modpath/foo/1*" ignore pattern
findModulesFromDirsAndFiles procedure will be updated to check each element found in a directory to determine if it should be taken into account or ignored
Final design: one new modulefile command introduced: modulepath-ignore. This command accepts same patterns than .gitignore files (including negated patterns).
Activity
xdelaruelle commentedon Jan 15, 2025
New commands
modulepath-ignore/modulepath-only, described on #561, may also be used to avoid recording information regarding files that are set to be avoided.Copying specification written in #561 here.
xdelaruelle commentedon Aug 18, 2026
New modulefile commands may be introduced to define the path pattern to ignore or to only take into account.
A new
modulepath-ignorecommand may take a list of file path pattern as arguments (1..N). When walking down the content of modulepath directory, entries that match any of these patterns are ignored (files are not evaluated to check if they are modulefiles, directories are not searched down to find other modulefiles)In the issue described on the mailing-list, modulefiles are a few files among large amount of other kind of files (installed software files). In this case another command may be useful,
modulepath-only, to define a list of path pattern (1..N) to only take into account to check for modulefiles. Whenmodulepath-onlyis used, patterns defined viamodulepath-ignoreare ignored as the logic is not to accept all ignore some but to ignore all accept some.Preferably, such commands should be set in the top
.modulercof modulepath, as it determines how to walk through its content. If used in a.modulercfile found deeper, current directory will be added to the pattern. For instance, if/path/to/modpath/modname/.modulercdefinesfoo*pattern to ignore, it will be converted into/path/to/modpath/modname/foo*.If user mentions a file that matches a
modulepath-ignorepattern or does not matchmodulepath-onlypattern when set:.modulecachexdelaruelle commentedon Aug 18, 2026
One use case for the ignore/only feature:
Additional design information:
findModulesFromDirsAndFilesprocedure will be updated to check each element found in a directory to determine if it should be taken into account or ignoredxdelaruelle commentedon Aug 22, 2026
Final design: one new modulefile command introduced:
modulepath-ignore. This command accepts same patterns than.gitignorefiles (including negated patterns).doc: design ignore of files in modulepath
Add modulepath-ignore command to filter out modulepath content