panic code is not deterministically lto'ed

[glandium]

This is something I noticed when comparing two builds of Firefox on automation. Both builds were done with the same flags, same toolchains, same paths, same everything. Without any sort of caching. And yet, they differed in exactly one way: the contents of std::panicking::default_hook.

See https://taskcluster-artifacts.net/IWpRV77rTgmt-9RKwTzJiA/0/public/diff.html

I tried multiple times, and there seems to be only two variants of the generated code, and that's the only code, in all of what's generated by rust in Firefox build, that differs.

It turns out this is reproducible at a much smaller scale:

$ mkdir -p foo/src; cd foo
$ cat > Cargo.toml <<EOF
[package]
name = "foo"
version = "0.1.0"

[lib]
crate-type = ["staticlib"]

[profile.release]
opt-level = 2
rpath = false
debug-assertions = false
panic = "abort"
codegen-units = 1
lto = true
EOF
$ cat > src/lib.rs <<EOF
#[no_mangle]
pub extern "C" fn foo() {
    panic!("foo");
}
EOF
$ cargo build --release
$ mv target/release/libfoo.a .
$ rm -rf target
$ cargo build --release
$ diff target/release/libfoo.a .
Binary files target/release/libfoo.a and ./libfoo.a differ

It can take a few attempts repeating the last three commands before differences show up, because there are only two variants, and you may end up getting the same variant multiple times in a row.

Cc: @alexcrichton @nikomatsakis

[alexcrichton]

Oh dear now this is a little terrifying. The following script also reproduces the error here and is a bit smaller in that it only uses rustc. This script generates only the IR and object file for the compilation at hand:

Details

#!/bin/bash

set -ex

input="
#[no_mangle]
pub extern fn foo() {
    panic!(\"foo\");
}
"

rustc="rustc /dev/stdin -O -C lto -C panic=abort -C codegen-units=1"
rustc="$rustc --emit llvm-ir,obj --crate-type staticlib"

for i in `seq 1 100`; do
  rm -rf a b
  mkdir a b
  $rustc --out-dir a <<< $input
  $rustc --out-dir b <<< $input
  a=$(md5sum a/stdin.ll | awk '{print $1}')
  b=$(md5sum b/stdin.ll | awk '{print $1}')
  if [ "$a" != "$b" ]; then
    echo IR is different
    exit 1
  fi
  a=$(md5sum a/stdin.o | awk '{print $1}')
  b=$(md5sum b/stdin.o | awk '{print $1}')
  if [ "$a" != "$b" ]; then
    echo object is different
    exit 1
  fi
done

The scary part here is that the IR is the same when the object files are different. This means that somehow LLVM's code generator looks like it's being non-deterministic. I can't reproduce this with llc either, LLVM's standalone code generator.

Somehow this means that the in-process state of LLVM is nondeterministic in just the right way between compilations that it affects the final output file. As to how... that's a bit of a mystery.

This isn't at least an obvious determinism bug in rustc in that the IR is the same so we're feeding the same input to LLVM both times. It's seemingly something else that's going awry!

[alexcrichton]

cc @michaelwoerister

[Mark-Simulacrum]

Trace diff for rustc_codegen_llvm is non-empty but surface level appears to be just pointers changing addresses and hashes being different (still odd, but at least reasonable): https://gist.github.com/Mark-Simulacrum/5d04c7ad104aa8936d45c4ac71f301f3.

Trace diff for the whole compilation (RUST_LOG=trace) is here: https://gist.github.com/Mark-Simulacrum/a1dcc978a6b65cd48307ad9407c60cf9. This shows that there is certainly non-determinism inside rustc, but also does not look like there's anything obvious here.

This seems likely to have something to do with recent changes to panic_implementation though IIRC they didn't touch codegen directly. However, perhaps we're seeing some side effect of the specific code being generated -- the BoxMeUp and similar code is relatively low-level for Rust.

I cannot reproduce with a minimal no_std program within the ~100 iterations:

#![no_std]
#![feature(panic_implementation)]

use core::panic::PanicInfo;

#[panic_implementation]
fn rust_begin_panic(_info: &PanicInfo) -> ! {
   loop {}
}

#[no_mangle]
pub extern fn foo() {
    panic!(\"foo\");
}

[glandium]

FWIW, this doesn't reproduce with 1.25, which was the first version with LLVM 6.

[Mark-Simulacrum]

I'll attempt a bisection tomorrow.

[glandium]

This reproduces with nightly-2018-06-04, which is the first one with #50338 (panic_implementation) merged. But it also does reproduce with nightly-2018-06-03.

[alexcrichton]

A local bisection points to a rollup, #49051, as the culprit. #48892 seems the most likely but even that seems like it'd be unrelated...

[glandium]

Both the merge of #49051 and the parent merge fail for me, so it seems your bisection went wrong.

[alexcrichton]

The merge of #49051, 36b6687, reproduces via the script I wrote above for me locally. The previous merge of #47813, 3926453, did not reproduce via the same script after 100 executions of trying to get a different hash. Additionally using 3926453 I am unable to reproduce via the example you gave in the OP.

How are you testing 3926453? I'm using rustup-toolchain-install-master to download the prebuilt binaries and execute locally. Did you build from scratch?

To hopefully weed out weird build issues as the problem I've tested the next previous merge as well of #48138, ff2d506, but it also isn't reproducing the bug here.

[glandium]

I built from scratch and tested both 36b6687 and 3926453, which both reproduced.

[glandium]

Local bisection with builds from scratch point me to the upgrade to LLVM 6, quite consistently. Has the compiler used on automation to build C/C++ code changed later?

[glandium]

Mmmm it fails both when llvm is built with GCC 5.4 (which is what I was originally bisecting with) and GCC 4.8 (which seems to be what rust automation used for some old builds I looked at the logs for). I wonder if the fact that this is on Ubuntu, where GCC defaults to have some features enabled that upstream GCC doesn't makes a difference.