runtime: decorate anonymous memory mappings

[L3n41c]

Linux 5.17 has introduced an API to name the anonymous memory areas: prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, …).

This API is already leveraged by the glibc to give an insight of what each anonymous memory area is used for.
See:

• https://github.com/bminor/glibc/blob/fc058b46c78f090580c1ca5b785b54ae4092f0c1/malloc/malloc.c#L2434 and
• https://github.com/bminor/glibc/blob/fc058b46c78f090580c1ca5b785b54ae4092f0c1/sysdeps/unix/sysv/linux/setvmaname.c#L63-L64

Thanks to that, /proc/<pid>/maps and /proc/<pid>/smaps provide information about what each anonymous memory area is used for:

lenaic@coruscant:~$ GLIBC_TUNABLES=glibc.mem.decorate_maps=1 cat /proc/self/maps 
603d00bf8000-603d00bfa000 r--p 00000000 00:21 47929385                   /usr/bin/cat
603d00bfa000-603d00bff000 r-xp 00002000 00:21 47929385                   /usr/bin/cat
603d00bff000-603d00c01000 r--p 00007000 00:21 47929385                   /usr/bin/cat
603d00c01000-603d00c02000 r--p 00008000 00:21 47929385                   /usr/bin/cat
603d00c02000-603d00c03000 rw-p 00009000 00:21 47929385                   /usr/bin/cat
603d01d05000-603d01d26000 rw-p 00000000 00:00 0                          [heap]
7091f2800000-7091f2aec000 r--p 00000000 00:21 48077669                   /usr/lib/locale/locale-archive
7091f2cd1000-7091f2cd4000 rw-p 00000000 00:00 0                          [anon: glibc: loader malloc]
7091f2cd4000-7091f2cf8000 r--p 00000000 00:21 48076955                   /usr/lib/libc.so.6
7091f2cf8000-7091f2e69000 r-xp 00024000 00:21 48076955                   /usr/lib/libc.so.6
7091f2e69000-7091f2eb8000 r--p 00195000 00:21 48076955                   /usr/lib/libc.so.6
7091f2eb8000-7091f2ebc000 r--p 001e3000 00:21 48076955                   /usr/lib/libc.so.6
7091f2ebc000-7091f2ebe000 rw-p 001e7000 00:21 48076955                   /usr/lib/libc.so.6
7091f2ebe000-7091f2ec6000 rw-p 00000000 00:00 0 
7091f2ec6000-7091f2ec8000 rw-p 00000000 00:00 0                          [anon: glibc: loader malloc]
7091f2ed2000-7091f2f14000 rw-p 00000000 00:00 0                          [anon: glibc: malloc]
7091f2f14000-7091f2f16000 r--p 00000000 00:00 0                          [vvar]
7091f2f16000-7091f2f18000 r--p 00000000 00:00 0                          [vvar_vclock]
7091f2f18000-7091f2f1a000 r-xp 00000000 00:00 0                          [vdso]
7091f2f1a000-7091f2f1b000 r--p 00000000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
7091f2f1b000-7091f2f44000 r-xp 00001000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
7091f2f44000-7091f2f4f000 r--p 0002a000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
7091f2f4f000-7091f2f51000 r--p 00035000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
7091f2f51000-7091f2f52000 rw-p 00037000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
7091f2f52000-7091f2f53000 rw-p 00000000 00:00 0 
7ffe15321000-7ffe15342000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

See the lines with [anon: glibc: …]

It could be valuable to have something similar for Go programs.
One use-case that comes to my mind is profiling multi-languages programs that are mixing Go and native C libraries.
With a pure 100% Go program, knowing where the memory allocations are coming from can be done with pprof.
But pprof profiles only the Go code.
In a program mixing Go and C libraries, we first need to know if the memory allocation we’re hunting is coming from the Go part or the C part.
Labeling the anonymous memory regions would help for that.

I have already drafted a proof of concept in CL #646095: https://go-review.googlesource.com/c/go/+/646095 that makes the following Go program:

package main

import (
        "fmt"
        "log"
        "os"
)

/*
#include <stdlib.h>

void f(void)
{
  (void)malloc(1024*1024*1024);
}
*/
import "C"

func main() {
        C.f()

        data, err := os.ReadFile("/proc/self/maps")
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(string(data))
}

produce this output:

lenaic@coruscant:~/doc/devel/perso/test_vma_name$ GLIBC_TUNABLES=glibc.mem.decorate_maps=1 ~/doc/devel/open-source/go/bin/go run .
00400000-00402000 r--p 00000000 00:47 738                                /tmp/go-build3615860867/b001/exe/test_vma_name
00402000-004a4000 r-xp 00002000 00:47 738                                /tmp/go-build3615860867/b001/exe/test_vma_name
004a4000-00574000 r--p 000a4000 00:47 738                                /tmp/go-build3615860867/b001/exe/test_vma_name
00574000-00575000 r--p 00173000 00:47 738                                /tmp/go-build3615860867/b001/exe/test_vma_name
00575000-00580000 rw-p 00174000 00:47 738                                /tmp/go-build3615860867/b001/exe/test_vma_name
00580000-005a4000 rw-p 00000000 00:00 0 
05105000-05126000 rw-p 00000000 00:00 0                                  [heap]
c000000000-c000400000 rw-p 00000000 00:00 0                              [anon: Go: map]
c000400000-c004000000 ---p 00000000 00:00 0                              [anon: Go: reserve]
774f2c000000-774f2c021000 rw-p 00000000 00:00 0                          [anon: glibc: malloc arena]
774f2c021000-774f30000000 ---p 00000000 00:00 0 
774f34000000-774f34021000 rw-p 00000000 00:00 0                          [anon: glibc: malloc arena]
774f34021000-774f38000000 ---p 00000000 00:00 0 
774f38000000-774f38021000 rw-p 00000000 00:00 0                          [anon: glibc: malloc arena]
774f38021000-774f3c000000 ---p 00000000 00:00 0 
774f3c000000-774f3c021000 rw-p 00000000 00:00 0                          [anon: glibc: malloc arena]
774f3c021000-774f40000000 ---p 00000000 00:00 0 
774f433ff000-774f43400000 ---p 00000000 00:00 0 
774f43400000-774f43c00000 rw-p 00000000 00:00 0                          [anon: glibc: pthread stack: 4120]
774f44000000-774f44021000 rw-p 00000000 00:00 0                          [anon: glibc: malloc arena]
774f44021000-774f48000000 ---p 00000000 00:00 0 
774f4ac4b000-774f4ac4c000 ---p 00000000 00:00 0 
774f4ac4c000-774f4b44c000 rw-p 00000000 00:00 0                          [anon: glibc: pthread stack: 4122]
774f4b44c000-774f4b44d000 ---p 00000000 00:00 0 
774f4b44d000-774f4bc4d000 rw-p 00000000 00:00 0                          [anon: glibc: pthread stack: 4121]
774f4bc4d000-774f4bc8d000 rw-p 00000000 00:00 0                          [anon: Go: persistent alloc]
774f4bc8d000-774f4bc8e000 ---p 00000000 00:00 0 
774f4bc8e000-774f4c48e000 rw-p 00000000 00:00 0                          [anon: glibc: pthread stack: 4119]
774f4c48e000-774f4c4ce000 rw-p 00000000 00:00 0                          [anon: Go: persistent alloc]
774f4c4ce000-774f4c4cf000 ---p 00000000 00:00 0 
774f4c4cf000-774f4cccf000 rw-p 00000000 00:00 0                          [anon: glibc: pthread stack: 4118]
774f4cccf000-774f4ccdf000 rw-p 00000000 00:00 0                          [anon: Go: arena]
774f4ccdf000-774f4ccef000 rw-p 00000000 00:00 0                          [anon: Go: span]
774f4ccef000-774f4cdef000 rw-p 00000000 00:00 0                          [anon: Go: page alloc]
774f4cdef000-774f4ce00000 rw-p 00000000 00:00 0                          [anon: Go: persistent alloc]
774f4ce00000-774f4ee00000 rw-p 00000000 00:00 0                          [anon: Go: heap arena map]
774f4ee00000-774f5ef80000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f5ef80000-774f5ef81000 rw-p 00000000 00:00 0                          [anon: Go: map]
774f5ef81000-774f7ef80000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f7ef80000-774f7ef81000 rw-p 00000000 00:00 0                          [anon: Go: map]
774f7ef81000-774f90e30000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f90e30000-774f90e31000 rw-p 00000000 00:00 0                          [anon: Go: map]
774f90e31000-774f93206000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f93206000-774f93207000 rw-p 00000000 00:00 0                          [anon: Go: map]
774f93207000-774f93600000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f93605000-774f93685000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f93685000-774f93686000 rw-p 00000000 00:00 0                          [anon: Go: map]
774f93686000-774f93705000 ---p 00000000 00:00 0                          [anon: Go: reserve]
774f93705000-774f93725000 rw-p 00000000 00:00 0                          [anon: Go: map]
774f93725000-774f93728000 rw-p 00000000 00:00 0                          [anon: glibc: loader malloc]
774f93728000-774f9374c000 r--p 00000000 00:21 48076955                   /usr/lib/libc.so.6
774f9374c000-774f938bd000 r-xp 00024000 00:21 48076955                   /usr/lib/libc.so.6
774f938bd000-774f9390c000 r--p 00195000 00:21 48076955                   /usr/lib/libc.so.6
774f9390c000-774f93910000 r--p 001e3000 00:21 48076955                   /usr/lib/libc.so.6
774f93910000-774f93912000 rw-p 001e7000 00:21 48076955                   /usr/lib/libc.so.6
774f93912000-774f9391a000 rw-p 00000000 00:00 0 
774f9391a000-774f9391c000 rw-p 00000000 00:00 0                          [anon: glibc: loader malloc]
774f93928000-774f93968000 rw-p 00000000 00:00 0                          [anon: Go: persistent alloc]
774f93968000-774f9396a000 r--p 00000000 00:00 0                          [vvar]
774f9396a000-774f9396c000 r--p 00000000 00:00 0                          [vvar_vclock]
774f9396c000-774f9396e000 r-xp 00000000 00:00 0                          [vdso]
774f9396e000-774f9396f000 r--p 00000000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
774f9396f000-774f93998000 r-xp 00001000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
774f93998000-774f939a3000 r--p 0002a000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
774f939a3000-774f939a5000 r--p 00035000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
774f939a5000-774f939a6000 rw-p 00037000 00:21 48076946                   /usr/lib/ld-linux-x86-64.so.2
774f939a6000-774f939a7000 rw-p 00000000 00:00 0 
7fff8a80a000-7fff8a82b000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

See the lines with [anon: Go: …]

Thanks to CL #646095, we can know track which memory has been allocated by the Go runtime vs. by the C part.

Rather than labeling all anonymous areas with the same Go label, I decided to mimic the glibc behavior and to provide more information about which part of the Go runtime allocated them. But the exact labels can probably be improved.

[gopherbot]

Change https://go.dev/cl/646095 mentions this issue: runtime: decorate anonymous memory mappings

[gabyhelp]

Related Issues

• runtime: GC & debug.FreeOSMemory calls don't return memory to the OS #45000 (closed)
• proposal: runtime: manage off-heap memory lifecycle using the garbage collector #70224

Related Code Changes

• runtime: decorate anonymous memory mappings
• runtime,runtime/pprof: get memory mappings on darwin.

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

I don't think this has to go through the proposal process.