Add option for dxtbx's libtbx_refresh to not modify conda_base

[phyy-nx]

Hi, this is related to #413, which added a check for a readonly conda_base before modifying it. This was quite helpful and resolved issues we were having with HPC systems. However, it remains an issue in some HPC, CI, and dev environments that libtbx_refresh is changing conda_base (since #382). Here are some example impacts:

• Development environments are not as stable, as conda_base is not immutable during the build step
• A single conda_base cannot be used for multiple builds as part of development environments or build systems
• Most importantly, it impacts the phenix build system. Indeed, phenix is now pinned to dials 3.2 to work around the problems caused by modifying conda_base during the build (see cctbx/cctbx_project@58a213e).

For reference I did several controls to see what the modifications are. Steps:

• Run bootstrap base to get a dials conda environment
• Control: copy the environment and run a cctbx-only build. Diff the original conda_base and the copy (no significant changes)
• Test: copy the environment and run a dxtbx build. Differences:
  • dxtbx dispatchers in conda_base/bin
  • dxtbx.egg-link and easy-install.pth in conda_base/lib/python3.7/site-packages

I then enforced a readonly build by setting the check in libtbx_refresh to False, verified the conda environment was unmodified, and ran the dxtbx tests. A'ok.

Solution: add an option to dxtbx/libtbx_refresh to use the readonly option even if the conda_base folder is writable. This could either be an environment variable or an option passed to libtbx.configure and referenced using libtbx.env.build_options. The former would be simpler but would require exporting it before running make reconf. The latter seems preferable. Any opinions?

My preference would be for this option to be set by default, and only disabled for builds destined for public release. In nearly every other case, to me it seems preferable to not modify the environment during the configure step, but please let me know of other use cases I'm not aware of.

Thanks!

[ndevenish]

It sounds like you want a virtualenv?

Alternatively, would an environment variable suit your cases?

[phyy-nx]

Well we use virtualenvs at several HPCs, including the EuXFEL Maxwell cluster. And you're right in that case as we don't care as much if it gets modified (except for my first 2 bullets above). But it doesn't help the phenix build, as they won't be switching to virtualenvs any time soon.

An environment variable that is checked in libtbx_refresh would be fine.

As an aside, this also affects iota and a similar solution will be needed there.