There's no way to make `cargo build` - with no env vars - build redistributable executable

[kornelski]

Cargo and many sys creates depend on environmental variables for configuration, but there's no standard way to configure default env vars for a project.

Most of this configuration (such as target-cpu, crt-static and whether libraries should be linked statically) is absolutely critical for making high quality redistributable executables.

For example, forgetting about set RUSTFLAGS="-C target-feature=+crt-static" on Windows makes executables that don't work without the latest Visual Studio runtime, which is not available on Windows by default. Users get a scary warning about missing DLL, and googling DLL's name finds lots of shady sites trying serve adware/malware.

Similarly, forgetting to set all dependencies static on macOS with Homebrew means sys creates will use pkg-config, which will hardcode Homebrew-specific paths in the executable, and the executable won't run on other people's computers. "Works only on developers' machines" problem is not easy to spot before getting complaints from users for shipping a broken executable.

Env vars are also shared global mutable state, so even if you remember to set them, they make switching between projects harder and more error-prone.

• Cargo should use Cargo.toml for all configuration, so that git clone; cargo build can be made to do the right thing, instead of having to wrap cargo in another build process and remember to use it every time instead of the usual, simplest — but invalid — cargo build by mistake.

• The features mechanism should be extended, or another mechanism added, to correctly support selection of static vs dynamic linking of libraries. (e.g. you use foo-rs crate, but want to configure foo-sys to use static linking).

• If "Cargo-native" changes to Cargo.toml are undesirable and env vars are here to stay, then Cargo.toml should at least have a section for specifying env vars for the project, when it's build as the top-level project (i.e. not as a dependency. Dependencies setting global variables would be bad).

[killercup]

This came up when discussion packaging and distributing CLI tools, but it might also concern @rust-lang-nursery/cargo

[epage]

The impression I get of cargo is that it leaves these kind of policies and practices to the end-user environment to take care which I think I agree with due to this personally feeling like a landmine of issues to try to integrate in.

I do think this would be something to consider for cargo-tarball and other packaging tools

[matklad]

(the following is out of cache for me so could be wrong)

Today, one totally can write a build script which is hermetic in a sense that it only touches info explicitly specified in the metadata section of Cargo toml(
https://doc.rust-lang.org/cargo/reference/manifest.html#the-metadata-table-optional)

The problem is that nobody writes such well behaved build scripts. The possible solution here is to write a library, which makes writing well-behaved build scripts easy, and to migrate crates one by one to this new library. I think that this is basically a question of implementation and getting sufficient mind-share of the community to make such library a default standard.

However, Cargo has an unstable feature to make this even easier:
rust-lang/cargo#14903

Metabuild is beasically: “use this package as build.rs”. That Metabuild is supposed to read input from Cargo.toml metadata(not to be confused with Cargo metadata) to make it actually do different things for different crate.

So the way forward here is to catch up on meta build discussion and implementing a Metabuild crate which works via metabuild Cargo feature on nightly and via “put fn main() {metabuild::main()} in build.rs” on stable.

[matklad]

Hm, re-reading this, it seems like metabuild won’t actually help here, because it does not introduce new mechanism for passing info from the top-level project to build scripts of the dependencies.

This indeed seems like a missing feature for Cargo! I think adding a key in .cargo/config to override env vars of build scripts could be a welcome solution. Using .cargo/config here because this is what is used to override rustflags.

[kornelski]

I've seen projects include .cargo/config in their git repo. Is that the recommended way? I have concerns:

• It's a hidden directory. Having config that affects the build try to hide itself is kind-of weird.

• I'm not sure what happens if there are multiple config files (project's .cargo/config, user's .cargo/config, system global). If they're all merged hierarchically that'd be OK-ish, but if Cargo searches for a config file and stops at the first one found, that could be disruptive.

• setting of compiler flags is a blunt, leaky abstraction and breaks stuff, e.g. flags for the final product may be invalid for build.rs compilation or compilation of proc macros. I'd rather see mapping of Cargo config to necessary compiler flags done intelligently by Cargo.

So I do agree some feature is missing here. There's a need for a project-wide configuration, with variants of the config for each target platform. If you were planning to add custom user profiles (rather than fixed debug/release/bench/test), then it'd be nice to also have them here.

I think it'd be OK if it was another section in Cargo.toml. If it was another file, it'd be unclear why e.g. release/debug profiles are in Cargo.toml, but other compilation settings are in another file.

How about:

# have profile config per target, following same pattern as deps
[target.'cfg(windows)'.profile.release]
# map more Rust flags to "native" Cargo profile options
crt = "static" 

[kornelski]

and then some way to specify feature flags and other config for dependencies that are several layers deep.

Perhaps similarly to the way patch works:

# select crate to configure
[configuration.crates-io.some_crate] 
# enable extra features
features = ["foo", "bar"]
# the rest will be passed as ENV vars to its build script:
key = "value" # sets SOME_CRATE_KEY="value"

[epage]

This indeed seems like a missing feature for Cargo! I think adding a key in .cargo/config to override env vars of build scripts could be a welcome solution. Using .cargo/config here because this is what is used to override rustflags.

Isn't what you set the variables to dependent on whom you are building for? For example, creating a tarball for distributing vs packaging for distributions? To me, it seems like this would belong in the packaging layer, like cargo-tarball.

[kornelski]

I think it's more complex than that. There are project-specific, target-specific and packaging-specific requirements for dependency configuration and building. They overlap. Ideally, the end result should be more of a negotiation between all of them.

As an example, whether a dependency should be dynamic or static is something that almost all -sys crates need to know, and configuring that right is much more complex than it seems:

• distro requirement: Debian wants to unbundle everything.
• target requirement: macOS frameworks expose symbols of Apple's fork of libjpeg, so if you dynamically link to both Cocoa and your libjpeg, stuff breaks.
• project requirement: my project runs into a bug in libpng 1.2. I can accept dynamic libpng if it's 1.4+, but have to have a static one otherwise.

So if you go with opinion of only one of them, something won't work. For example cargo-deb (or some metabuild tool) could enforce Debian's policy, but it has no way of knowing that this one project needs an exception, sometimes. It needs both OS's policy & project-specific information to package it properly.

On macOS pkg-config gives Homebrew-specific paths that may not be valid on other people's Macs. If you use a generic Unix-compatible build process, which even passes tests on macOS in CI, it may end up breaking without accounting for this OS-specific quirk.

So in the end I'd probably need to have a way to specify project-specific quirks and exceptions, which then a packaging tool could read, combine with its own policy, and use that to configure dependencies.

It would be great if that was part of Cargo, not specific packaging tools, so that if I make my project work with cargo-tarball, I wouldn't have to duplicate the config for a (hypothetical) cargo-zip.

[epage]

It would be great if that was part of Cargo, not specific packaging tools, so that if I make my project work with cargo-tarball, I wouldn't have to duplicate the config for a (hypothetical) cargo-zip.

To be clear, cargo-tarball is going to cover zip for windows but I get your point. You could have cargo-tarball, cargo-deb, cargo-wix, .... Each will have its own requirements. Some will overlap. Keeping all up to date will be interesting.

[raphaelcohn]

Anything that depends on the global configuration (aka 'environment') of the build machine is inherently broken and will inevitably be incapable of reproducible builds (reproducible here means even timestamps are controlled, such that the resultant binary is byte-for-byte identical).

The real solution here is the same as it has been for decades; one ships profile files, one for each build variant, that specify every possible switch and setting that the developer of the binary supports and can vouch for (eg with testing). In practice, though, profile files usually end up being shell scripts because of the mix of configuration and grep-awk-sed required - and that in itself makes an application build difficult to audit. As an additional, such profiles should always be for cross-compilation - even when the target is the same OS and arch as the current machine.

Such an approach is hard to do in practice, however, whenever a package depends either on libc or another C library; the C 'build' systems are brittle, broken, fragmented and frankly, crap. All of them. Primarily because they do feature detection (GNU auto-cruft) or use the local C compiler (all of which hard code a billion assumptions and paths, particularly). All are hard to make reproducible (if not impossible). I can say this quite legitimately as the developer of Libertine Linux - an attempt to have a build-from-source-in-git, auditable, reproducible build of Linux.