Core Dumps

Making the Kernel Write One

A core dump is the process memory image at the moment it died. Two things have to be true, a nonzero core size limit and a core_pattern you can find.

ulimit -c              # 0 means no core will ever be written
ulimit -c unlimited    # this shell and its children only
cat /proc/sys/kernel/core_pattern

# write plain files into the cwd, named core.<pid>
sudo sysctl -w kernel.core_pattern=core
sudo sysctl -w kernel.core_uses_pid=1

# persist it
echo 'kernel.core_pattern=core.%e.%p' | sudo tee /etc/sysctl.d/50-core.conf

If core_pattern starts with | the kernel pipes the core to that program instead of writing a file. That is the normal state on modern distros, where the handler is systemd-coredump or apport, so no core file appears in your directory even though the dump was captured.

ulimit -c unlimited
sysctl -w fs.suid_dumpable=2      # allow cores from setuid binaries

Other reasons no core appears: the process was in a read-only or full directory, it called setrlimit itself, it was in a container whose limits differ from the host, or RLIMIT_CORE was reset by a service manager. For a systemd service the limit is LimitCORE=infinity in the unit, not your shell's ulimit.

coredumpctl

On any systemd distro, systemd-coredump catches the dump and coredumpctl is how you get at it.

coredumpctl list                       # everything it has kept
coredumpctl list prog                  # only this executable
coredumpctl list --since "1 hour ago"
coredumpctl info                       # detail on the most recent one
coredumpctl info 4711                  # by pid
coredumpctl dump prog -o prog.core     # extract the raw core file
coredumpctl debug prog                 # open it in gdb right now
coredumpctl debug --debugger=lldb prog

TIME                        PID  UID  GID SIG     COREFILE EXE
Mon 2026-09-14 11:02:31 UTC 4711 1000 1000 SIGSEGV present  /home/me/prog

coredumpctl debug is the shortcut worth memorizing. It extracts the core, finds the matching executable, and launches gdb with both. A COREFILE column reading none means the dump was truncated or dropped, usually because it exceeded ProcessSizeMax in /etc/systemd/coredump.conf.

coredumpctl info | grep -i 'stack trace' -A 30
sudo systemctl status systemd-coredump@*.service

Opening a Core in gdb

gdb ./prog core.4711
gdb ./prog -c core.4711
gdb --core=core.4711 ./prog

(gdb) core-file core.4711     # load one from inside a session
(gdb) bt
(gdb) thread apply all bt     # every thread's stack at death
(gdb) info registers
(gdb) p $_siginfo.si_signo
(gdb) p $_siginfo._sifields._sigfault.si_addr
(gdb) info proc mappings      # works from the core's own note section
(gdb) detach                  # unload the core

A core is a corpse, so nothing runs. continue, next, call, and p f() all fail with a message about no running process. x, p, bt, info registers, and frame navigation all work.

$_siginfo tells you the fault address, which paired with info proc mappings is usually the whole diagnosis. A si_addr of 0x0 is a null dereference, and a small nonzero value is usually a null pointer plus a struct offset.

Snapshotting a Live Process

You do not have to wait for a crash. gcore writes a core from a running process and lets it continue.

gcore 4711                    # writes core.4711, standalone tool
gcore -o snap 4711            # writes snap.4711

(gdb) attach 4711
(gdb) gcore                   # or generate-core-file
Saved corefile core.4711
(gdb) detach

gcore is implemented on Linux, FreeBSD, Solaris, and S390 only. It writes a sparse file where the filesystem supports it, so ls -l can report a size far larger than the disk space used. Comparing two gcore snapshots minutes apart is a cheap way to see a leak or a stuck queue grow.

A Core From Another Machine

A core only makes sense with the exact binaries that produced it. Same build of the executable, and the same libc and shared libraries, byte for byte.

# collect on the crashing host
tar czf syms.tgz ./prog /lib/x86_64-linux-gnu/libc.so.6 /lib64/ld-linux*.so.2
mkdir -p /tmp/root && tar xzf syms.tgz -C /tmp/root

(gdb) file ./prog
(gdb) set sysroot /tmp/root
(gdb) set solib-search-path /tmp/root/lib:/tmp/root/usr/lib
(gdb) set debug-file-directory /tmp/root/usr/lib/debug
(gdb) core-file core.4711
(gdb) info sharedlibrary
(gdb) bt

Set sysroot before loading the core, because gdb resolves libraries when the core is read. In info sharedlibrary, a No in the Syms Read column or a path pointing at your local /lib means gdb silently used the wrong library, and every frame in that library will be nonsense.

For a container, the sysroot is just the image filesystem, so docker cp or podman unshare the relevant paths out and point sysroot at that tree. With debuginfod configured, matching debug info for distro libraries is fetched by build ID and none of this is needed for the libraries themselves.