How to enable core dumps for Apache and trace Apache segmentation fault on a Linux server

1. Connect to a Linux server via SSH.

2. Find a partition that is having a free space (use 'df -h') enough to store memory dumps. In this example, we use /var directory on the root '/' partition.

   Note: On CentOS 7, due to the PrivateTmp=true parameter in a systemd unit, it is not possible to use some directories inside /tmp/ as a directory for core dumps.

3. Modify Apache startup script:

   • For systemd systems, create a new unit file using the command:

     • on CentOS/RHEL-based distributions

       # systemctl edit httpd

     • on Debian/Ubuntu-based distributions

       # systemctl edit apache2

     and add the following lines:

     CONFIG_TEXT: [Service]
     LimitCORE=infinity

   • For non-systemd CentOS/RHEL-based systems, add the DAEMON_COREFILE_LIMIT=unlimited line to the /etc/sysconfig/httpd file.

   • For non-systemd Debian/Ubuntu-based systems, open the /etc/init.d/apache2 file and locate the do_start() section. Add the following line in this section:

     CONFIG_TEXT: ulimit -c unlimited

4. Create a folder where core dumps will be stored:

   # mkdir -p /var/coredumps
   # chmod a+w /var/coredumps

5. Specify the path and pattern for core dump files in the /proc/sys/kernel/core_pattern file:

   # echo /var/coredumps/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

   If for some reason the core_pattern file cannot be modified, specify a CoreDumpDirectory location in /etc/httpd/conf/httpd.conf (CentOS/RHEL) or /etc/apache2/apache2.conf (Debian/Ubuntu):

   # grep "CoreDumpDirectory" /etc/httpd/conf/httpd.conf
   CoreDumpDirectory /var/coredumps

6. Restart Apache service:

   • on CentOS/RHEL-based distributions

     # service httpd restart

   • on Debian/Ubuntu-based distributions

     # service apache2 restart

7. To be sure core dumps have been set up, perform the following test:

   7.1. List Apache processes and their PIDs:

   • on CentOS/RHEL-based distributions

     # ps auxf | grep httpd | grep -v grep
     root 15654 1.4 0.7 372448 13840 ? Ssl 13:25 0:00 /usr/sbin/httpd -DFOREGROUND
     apache 15680 0.0 0.3 370944 6164 ? S 13:25 0:00 \_ /usr/sbin/httpd -DFOREGROUND

   • on Debian/Ubuntu-based distributions

     # ps auxf | grep apache2 | grep -v grep
     root 2410 0.2 1.5 292588 33036 ? Ssl 13:32 0:00 /usr/sbin/apache2 -k start
     www-data 2413 0.0 0.3 237808 6504 ? S 13:32 0:00 \_ /usr/sbin/apache2 -k start

   7.2. Kill a main Apache process with SIGSEGV signal:

   # kill -SIGSEGV <PID>

   7.3. Check the folder with core dumps. The files will be named like:

   • on CentOS/RHEL-based distributions

     # ls /var/coredumps
     core-httpd-11-0-0-14750-1556864320

   • on Debian/Ubuntu-based distributions

     # ls /var/coredumps
     core-!usr!sbin!apach-11-0-0-2410-1556865180

   Note: If the abrtd daemon is running, it will prevent creation of core dump files.

8. Reproduce the issue.

   Once you finish with tracing, remove the changes in Apache startup script made on step 3 and restart Apache.

Note: On ubuntu 22, apache2-dbg package has been dropped. For other debug packages, check this Ubuntu documentation page.

1. Install the gdb utility:

   • on CentOS/RHEL-based distributions

     # yum install gdb
     # debuginfo-install httpd

   • on Debian/Ubuntu-based distributions

     # apt-get install gdb
     # apt-get install apache2-dbg

2. Replace the path to a core dump file in the command below and run it:

   • on CentOS/RHEL-based distributions

     # gdb /usr/sbin/httpd /var/coredumps/core-httpd-11-0-0-14750-1556864320

   • on Debian/Ubuntu-based distributions

     If the core dump file contains '!' symbols in its name, put a backslash '\' before each '!' symbol like this:

     # gdb /usr/sbin/apache2 /var/coredumps/core-\!usr\!sbin\!apach-11-0-0-2410-1556865180

3. Run the bt command in the gdb prompt to get a backtrace and analyse output:

   CONFIG_TEXT: (gdb) bt full