Enabling Core Dumps in Embedded Systems

Core dumps are the standard Linux way for post-mortem analysis of crashed applications. Given some preconditions, the core dumps can provide a detailed back trace and shed some light about the last whereabouts of the crashed application.

The core dump itself is an executable file format, and it is generated by the Kernel. The core dump contains a list of memory sections which were accessible by the process, and their memory image. This allows you to analyze the core dump offline, on a host machine, using gdb which was compiled for the appropriate target.

Core dump is not enabled by default in embedded systems mainly due to memory limitations. In this article I will explain what is required in order to enable core dumps support on Embedded Linux/uClibc/Busybox platforms for a specific debug task.

As I mentioned, core dumps are not enabled by default in embedded systems, which usually have a very limited storage space and more RAM than non-volatile storage. The following steps describe what is needed in order to enable core dumps in your platform.

Enable Kernel support for Core Dumps

Core dump support in the Kernel can be disabled in compilation time in order to reduce the kernel size. Without this support, no core dump will ever be generated. In order to enable the Kernel’s support for Core dumps, enable the following option in the Kernel configuration menu:

General setup  --->
      Configure standard kernel features (for small systems)  --->
            Enable ELF core dumps

Enabling this option adds about 4KB to the final kernel size.

Enable debug mode in uClibc

The uClibc library must be configured differently in order to support the back trace. In case your application is multi-threaded, then you need to enable to following configuration options:

General Library Settings  --->
     Build pthreads debugging support

Also, configure the uClibc to skip the library stripping, in order to retain its symbols:

uClibc development/debugging options  --->
      Strip libraries and executables

Enable Core Dump support in Busybox

The Busybox provides the init process of the system, which is the father of all processes. The Busybox can be configured to enable core dumps to all its children by inheritance. Actually, it sets the init process as “dumpable”, and sets the core dump size limitation to “unlimited”. All the children of init will inherit these properties and allow core dumps. There are two actions required in order to enable the Busybox support for core dumps. First, enable this option in the Busybox configuration menu:

Init Utilities  --->
     Support dumping core for child processes (debugging only)

Second, create an empty file in the target’s filesystem’s root directory, called “.init_enable_core“.

Setup the Core dump size limitation

In systems without Busybox, it is required to setup the Core dump size limitation. By default, the size is 0, and it means that the Core dump is disabled. In order to get a readable (non-truncated) Core dump, setup the limitation to “unlimited” using the ulimit application:

# ulimit -c unlimited

Setup the Core file name and path

By default,  the Core dump file is named core, and it is created in the current working directory of the process. There are cases where a process’s current working directory points to a read only filesystem. In this case, the Core dump file could not be created. Furthermore, the default file name does not indicate what was the process name or id, and by listing the file, you can not which process created it. The /proc/sys/kernel/core_pattern file can be set to define a template that is used to name core dump files. The template can contain % specifiers which are substituted by the following values when a core file is created:

%%  A single % character
%p  PID of dumped process
%u  real UID of dumped process
%g  real GID of dumped process
%s  number of signal causing dump
%t  time of dump (seconds since 0:00h, 1 Jan 1970)
%h  hostname
%e  executable filename

This file allows you to setup a different output directory as well. This option allows you to configure the Kernel to generate the files in a writable filesystem, such as a RAM disk, which could have enough storage space to hold the core dump file. Use “echo “your_pattern” > /proc/sys/kernel/core_pattern” to change the Core dump file name pattern.

The /proc/sys/kernel/core_uses_pid file determines if the Core dump file name will end with .PID.

Setup the Compilation flags of your debugged Application(s)

In order to make an application debugable, it must be compiled a bit differently, using some compilation flags that create debug symbols and enable support for stack frame.

1. The applications needs to be compiled with the “-g -ggdb” flags in gcc, in order to enable debug symbols.
2. The optimization level: Optimization levels higher than -O1 trigger frame pointer omitting, which makes back tracing impossible. The reason is because the frame pointer can be reused for other purposes, and the fact that saving the frame pointer requires additional opcodes. In order to keep the frame pointer in higher optimization levels, use the “-fno-omit-frame-pointer” flag in gcc. If possible, reduce the optimization level to -O1 for the purpose of the debug. Higher optimizations may alter the final binary and omit or inline some functions. The -O0 optimization level might break your application, so I don’t recommend using it.
3. If the target platform is ARM and the binaries are in Thumb mode, add the following flags in gcc to keep the stack frame in the standard form: “-mtpcs-frame -mtpcs-leaf-frame“.

Enable Core Dumps manually in your Application(s)

It is possible to set your application as “Dumpable” and its limit as “Unlimited” using standard system calls.

The “Dumpable” state can be set or retrieved by the prctl system call, using the PR_SET_DUMPABLE and PR_GET_DUMPABLE flags accordingly. When using the set command, the second argument determines if the process is “Dumpable” (1-yes, 0-no). Here’s an example code snippet:

{
    #include <sys/prctl.h>

    int ret;

    /* Get the Dumpable state */
    ret = prctl( PR_GET_DUMPABLE, 0, 0, 0, 0 );
    printf( "PR_GET_DUMPABLE returned %d", ret );

    /* Set the Dumpable state to be enabled */
    ret = prctl( PR_SET_DUMPABLE, 1, 0, 0, 0 );
    printf( "PR_SET_DUMPABLE returned %d", ret );

    /* Get the Dumpable state again, make sure it was set */
    ret = prctl( PR_GET_DUMPABLE, 0, 0, 0, 0 );
    printf( "PR_GET_DUMPABLE returned %d", ret );
}

The limitation of a core dump can be set manually by the application using the setrlimit function call, using the RLIMIT_CORE flag. It receives a structure with two fields. The first sets the current limitation, and the second sets the maximum allowed limitation. The fields accept any number, and RLIM_INFINITY to specify “Unlimited”. Here’s an example code snippet:

{
    #include <sys/time.h>
    #include <sys/resource.h>

    int ret;
    struct rlimit rlim;

    /* Get the core dump limitation */
    ret = getrlimit(RLIMIT_CORE, &rlim);
    printf( "RLIMIT_CORE returned %d (%d, %d)", ret, rlim.rlim_cur, rlim.rlim_max );

    /* Set the core dump limitation to be unlimited */
    rlim.rlim_cur = RLIM_INFINITY;
    rlim.rlim_max = RLIM_INFINITY;
    ret = setrlimit(RLIMIT_CORE, &rlim);
    printf( "RLIMIT_CORE returned %d", ret );

    /* Get the core dump limitation again */
    ret = getrlimit(RLIMIT_CORE, &rlim);
    printf( "RLIMIT_CORE returned %d (%d, %d)", ret, rlim.rlim_cur, rlim.rlim_max );
}

Custom Signal Handlers

Unfortunately, if your application is equipped with a customized signal handler, no core dump will be generated, because it is generated only by the default signal handlers. In case your application has a custom signal handler, disable it before starting to debug, otherwise, no core dump will be generated. Some sources in the Internet mention that restoring the default signal handler inside the signal handler after the exception has occurred, and sending it again in a loopback can trigger a core dump. In the tests I did, it did generate a core dump, but the only thing I saw in the core dump was the code that my handler executed (i.e. the calls to signal and kill), so this did not help me. Perhaps on other platforms this trick works better.

Conclusions

There are many actions you need to do in order to enable core dumps in Embedded Linux system. Most of them can be automated in your Makefiles and configuration system when a Debug-Mode option is enabled. When a process crashes and a Core dump was generated successfully, the following message will appear:

Segmentation fault (core dumped)

In the next step, download the generated core dump to the host machine and run gdb in order to analyze it. Use the core command to load it, and file command to load the symbols of your application. Use the bt full command to get a complete back trace.

Resources

http://linux.die.net/man/2/prctl
http://linux.die.net/man/7/signal
http://linux.die.net/man/2/setrlimit
http://linux.die.net/man/5/core
http://www.gentoo.org/proj/en/qa/backtraces.xml
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka12961.html

Coretrace debugger tool: http://www.arbetsmyra.dyndns.org/coretrace/