Introduce run-make V2 infrastructure, a run_make_support library and port over 2 tests as example

[jieyouxu]

Preface

See issue #40713: Switch run-make tests from Makefiles to rust for more context.

Basic Description of run-make V2

run-make V2 aims to eliminate the dependency on make and Makefiles for building run-make-style tests. Makefiles are replaced by recipes (rmake.rs). The current implementation runs run-make V2 tests in 3 steps:

1. We build the support library run_make_support which the rmake.rs recipes depend on as a tool lib.
2. We build the recipe rmake.rs and link in the support library.
3. We run the recipe to build and run the tests.

rmake.rs is basically a replacement for Makefile, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example run-make tests ported over to the new infrastructure.

Example run-make V2 test

// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}

Follow Up Work

• Adjust rustc-dev-guide docs

[rustbot]

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

[bjorn3]

Maybe do:

Suggested change

	let mut cmd = $crate::xshell::cmd!(sh, "{rustc}")
	.arg("--out-dir")
	.arg($scx.tmpdir())
	.arg("-L")
	.arg($scx.tmpdir());
	for arg in args.split_whitespace() {
	cmd = cmd.arg(arg);
	}
	let mut cmd = $crate::xshell::cmd!(sh, concat_str!("{rustc} ", $args))
	.arg("--out-dir")
	.arg($scx.tmpdir())
	.arg("-L")
	.arg($scx.tmpdir());

This should allow interpolations the same way as xshell itself and should correctly handle quoting arguments. I haven't tested it though.

[jieyouxu]

A bit unfortunate but cmd!() only accepts a literal, can concat_str! output a literal? I don't think so right? And yeah quoting arguments is a problem with the current incorrect behavior of split_whitespace.

[jieyouxu]

I could force the input to be a &[], since

let args = ["hello", "world"];
let c = cmd!(sh, "echo {args...}");

works I think. This however forces the use site to become something like

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);
let output = rustc!(
    scx,
    [
        "--emit=metadata",
        "--extern",
        &format!("stable={}/libstable.rmeta", scx.tmpdir()),
        "main.rs"
    ]
);

[bjorn3]

Of course. cmd!() would see the concat_str!() before expanding. @matklad would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance? Such that this could be cmd!(sh, $args) where sh is Shell::new().wrapper_command(rustc_path).

[jieyouxu]

While a proper fix to this requires a change to xshell, I will temporarily use the iterable form for the recipes since they're at least more correct than split_whitespace.

[matklad]

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance?

This would technically work, but I don't think that's an obvious API with a clear use-case. My answer would be, if you want to control the first arg, to use something like

cmd!(sh, "{rustc} --emit=metadata --crate-type=lib")

Like shells, xshell supports interpolation of the first argumetn (or at least it should support it).

[jieyouxu]

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

(Note that passing-as-an-array here is a temporary band-aid fix, the intention is to avoid needing an array literal.) That is, ideally, the API would look something more like

let _output = rustc!("--emit=metadata --crate-type=lib stable.rs");

so the test writer doesn't have to bother with an array literal.

[matklad]

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance?

This would technically work, but I don't think that's an obvious API with a clear use-case. My answer would be, if you want to control the first arg, to use something like

cmd!(sh, "{rustc} --emit=metadata --crate-type=lib")

Like shells, xshell supports interpolation of the first argumetn (or at least it should support it).

[jieyouxu]

I changed the implementation to rely on shell-words instead to split the input string into words that can be passed to Command::arg. The API now looks something like

extern crate run_make_support;

use run_make_support::{rustc, run, run_fail};

fn main() {
    let _output = rustc("a.rs --cfg x -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = rustc("b.rs -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = run("b");
    let _output = rustc("a.rs --cfg y -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = run_fail("b");
}

This unfortunately isn't as nice as having straight up string interpolation like xshell does, e.g. in CURRENT_RUSTC_VERSION:

extern crate run_make_support;

use run_make_support::{rustc, aux_build};

fn main() {
    let _output = aux_build("stable.rs");
    let output = rustc(format!("--emit=metadata --extern stable={}/libstable.rmeta main.rs", env!("TMPDIR")));

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}

[rustbot]

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

[matklad]

It might be interesting to combine "split-on-whitespace" idea with builder-style interface. This is the style of testing employed by Cargo:

https://github.com/rust-lang/cargo/blob/5febbe5587b74108165f748e79a4f8badbdf5e0e/tests/testsuite/build.rs#L4168-L4169

So,

let tmpdir = env!("TMPDIR");
let output = rustc("main.rs --emit=metadata")
    .arg(format!("--extern=stable={tempdir}/libstable.rmeta"))
    .stderr_utf8_lossy()?
;

Thinking more about this, I think "specific builders" are probably a better fit here than "generic shell".

I expect these tests to be relatively few, so we don't have to optimize suuuper hard for readability.

At the same time, I expect most of these tests to be similar, and to benefit from abstracting common functionality. Abstracting through a builder is easier than abstracting through macro-based DSL.

[bors]

☔ The latest upstream changes (presumably #113162) made this pull request unmergeable. Please resolve the merge conflicts.

[Mark-Simulacrum]

r? @bjorn3 - this looks good to me in broad strokes, but looks like you've done a bunch of review here already.

[bjorn3]

@bors r+

[bors]

📌 Commit 5727711 has been approved by bjorn3

It is now in the queue for this repository.

[bors]

⌛ Testing commit 5727711 with merge 6475c81...

[bors]

💔 Test failed - checks-actions

[jieyouxu]

Gonna edit ci.yml to try to test this on msvc beforehand.
@rustbot author

[Zalathar]

Oh, if you edit ci.yml, you also need to do x run src/tools/expand-yaml-anchors to update the real file that is generated from it.

(If you forget, x test tidy will complain and tell you to run that command.)

EDIT: Never mind, you figured it out 👍

[jieyouxu]

Oh, if you edit ci.yml, you also need to do x run src/tools/expand-yaml-anchors to update the real file that is generated from it.

Yeah, great advice on editing ci.yml, thank you so much 😄 Would waste so much more time trying to debug why this PR always die on MSVC otherwise.

[jieyouxu]

I fixed the rmake executable missing a .exe on Windows, and fixed the a-b-a-linker-guard test accidentally having an extra --cfg x passed in to the second rustc() invocation. Please, msvc, please work now.

[jieyouxu]

tools.mk special cases Windows and uses PATH=PATH:$TARGET_RPATH_DIR for executables instead of just using $TARGET_RPATH_ENV for non-Windows platforms...

[jieyouxu]

DLL_NOT_FOUND but am not sure which DLL is not found... One definite bug is PATH=$1;$2 seperator is ; on Windows and not : like *nixes.

[bjorn3]

It is the host that matters here, right? You can probably use std::env::join_paths here.

[jieyouxu]

Yep, that did it!

[jieyouxu]

Seems to finally work for msvc!
@rustbot ready

[bjorn3]

@bors r+

[bors]

📌 Commit 48e9f92 has been approved by bjorn3

It is now in the queue for this repository.

[bors]

⌛ Testing commit 48e9f92 with merge 17edace...

[bors]

☀️ Test successful - checks-actions
Approved by: bjorn3
Pushing 17edace to master...

[rust-timer]

Finished benchmarking commit (17edace): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.7%	[1.7%, 3.5%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 651.67s -> 651.533s (-0.02%)
Artifact size: 311.10 MiB -> 311.14 MiB (0.01%)