Accessing Ubuntu debug info server is enabled by default after runner image update but extremely slow

[vadz]

Description

I'm not sure if it's really a bug, but it cost me a couple of hours, so I'd like to at least leave a record of it somewhere where it can be found and, maybe, understand why is accessing https://debuginfod.ubuntu.com from GitHub Actions runners so slow.

Anyhow, the facts that motivated this are:

1. A relatively recent run of the CI job using Valgrind to run tests finished in 106 seconds. In case the log times out, here is the important part:

    1/4 Test #1: soci_empty_test ..................   Passed    2.20 sec

i.e. the test using "empty" database backend not doing anything took ~2s. This run used ubuntu-24.04 image version 20260628.225.1 and ran on the host compute agent with version 20260624.560 and ran on July 2.

2. All the subsequent runs of exactly the same workflow, using the same commit, starting from July 24 and until the latest commit, timed out because Valgrind ran extremely slowly. Here is an excerpt from a recent run which ran today, August 11, on ubuntu-24.04 image version 20260720.247.2 and host compute agent version 20260707.563. The "empty" test still succeeds but look at the time it takes now:

    1/4 Test #1: soci_empty_test ..................   Passed  813.81 sec

And other, less trivial, tests just time out and fail.

The fix is simple enough, once you know you need it: I've added --enable-debuginfod=no and the tests pass again now. And I guess the reason it is necessary now, while it wasn't before, is that DEBUGINFOD_URLS is now set to https://debuginfod.ubuntu.com in the runner environment while this probably wasn't the case before. However it's still not clear why is accessing this server to excruciatingly slow. Is it intentionally throttled? The access times are difficult to measure precisely as they are so huge, but it's definitely several orders of magnitude slower than when accessing it from a local machine. Is this expected?

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 22.04
• Ubuntu 22.04 Arm64
• Ubuntu 24.04
• Ubuntu 24.04 Arm64
• Ubuntu 26.04
• Ubuntu 26.04 Arm64
• Ubuntu Slim
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• macOS 26
• macOS 26 Arm64
• Windows Server 2022
• Windows Server 2025
• Windows Server 2025 with Visual Studio 2026
• Windows Desktop 11
• Windows Desktop 11 with Visual Studio 2026

Image version and build link

20260720.247.2, see https://github.com/SOCI/soci/actions/runs/31523383040/job/93885711625

Is it regression?

Yes, worked fine in 20260628.225.1, see https://github.com/SOCI/soci/actions/runs/28614100942/job/84853488049

Expected behavior

Valgrind not taking ages to run.

Actual behavior

Valgrind spends hours trying to contact https://debuginfod.elfutils.org and times out

Repro steps

Even running valgrind /bin/ls is enough to reproduce the problem.

[v-davit-ioramashvili]

Hi,

We will take a look.

[v-davit-ioramashvili]

Thanks for the write-up — your diagnosis is correct, and --enable-debuginfod=no is the right fix.

Worth clarifying two things: this comes from Ubuntu's default debuginfod configuration, not something we set explicitly on the image — Ubuntu ships its debuginfod client pointed at its own symbol server. And we don't operate that server, so we can't say whether it's throttled; the slowness is on Canonical's side.

If it's useful, you can also disable it for the whole job instead of per-invocation, by unsetting the variable in an early step:

- run: echo "DEBUGINFOD_URLS=" >> "$GITHUB_ENV"

[vadz]

Worth clarifying two things: this comes from Ubuntu's default debuginfod configuration, not something we set explicitly on the image — Ubuntu ships its debuginfod client pointed at its own symbol server. And we don't operate that server, so we can't say whether it's throttled; the slowness is on Canonical's side.

Oh, of course, this makes so much more sense. Sorry, I don't know why did I think that the problem was coming from the GitHub side, but thanks for looking at it anyhow!