DebuggingProgramCrash

Differences between revisions 1 and 114 (spanning 113 versions)
Revision 1 as of 2005-05-28 20:36:44
Size: 1040
Editor: adsl-213-190-44-43
Comment: imported from the old wiki
Revision 114 as of 2012-08-26 18:48:15
Size: 8904
Editor: jarl-dk
Comment: updated link to list-symbols-packages-v2.sh
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= DebuggingProgramCrash = Available languages: [[https://wiki.ubuntu.com/DebuggingProgramCrash_it| Italiano]],
<<Include(Debugging/Header)>>
||<tablestyle="float:right; font-size: 0.9em; width:40%; background:#F1F1ED; margin: 0 0 1em 1em;" style="padding:0.5em;"><<TableOfContents>>||
Line 3: Line 5:
 1. Check if the package has a debugging version available. In general, debugging packages will be named with a `-dbg` suffix
 1. If not, you can generally* build one this way:
  a. Install the build-time dependencies for the package: {{{
sudo apt-get build-dep <package>
This document describes how to debug Ubuntu package crashes and install debug packages on Ubuntu, which will aid in providing information for bugs.

== Using apport-retrace ==

If you want to debug a crash in packaged Ubuntu software, [[https://wiki.ubuntu.com/Apport|Apport]] will usually pick it up, create a `.crash` report in `/var/crash/` and report the crashed program. From there it is easiest to use [[http://manpages.ubuntu.com/apport-retrace|apport-retrace]].

`apport-retrace` regenerates the stack traces (both the simple and the threaded one) from an apport crash report from the included core dump, or gives you a Terminal window with gdb in an environment with debugging symbols, reproducing the situation of the crash through the core dump. For this it figures out the set of necessary packages and their accompanying debug symbol packages, so that the regenerated stack trace will be fully symbolic and thus become much more useful for developers to fix the problem.

`apport-retrace` has two modes: By default it will just regenerate traces based on the packages which are currently installed in the system, i. e. it assumes that all necessary debug symbols for the report are installed (see the next paragraph for this). When specifying the `-S` option, it creates a temporary "sandbox" and downloads and installs all necessary packages and debug symbols there. It will not do any changes to your system and does not require any special privileges.

Please see the [[http://manpages.ubuntu.com/apport-retrace|apport-retrace manpage]] for details and some examples how to run it.

You can also invoke this directly from the Apport crash notification, by clicking the "Examine locally" button:

 {{attachment:apport-examine-locally.png}}

This will collect some package information and then ask you in which mode you want to run `apport-retrace`:

 {{attachment:apport-retrace-gui.png}}

Normally you want to keep the first option, which will get you in a gdb session
in the sandbox:

 {{attachment:apport-retrace-gdb.png}}

If you do not want an interactive gdb session but just want it to update the already existing `.crash` file with fully symbolic stack traces, you can select the third option.

== Debug Symbol Packages ==
If you want to debug a crash in a project you are developing yourself or from a third-party package, or need the debug symbols for particular libraries very often, it is helpful to install those permanently into your system.

First, check if there is a package with a -dbg suffix in the main Ubuntu repositories. These are the debug symbol packages, and are equivalent to '-dbgsym' described below. You can safely use either one, but not both at once. Try installing the package, adding -dbg to its name, for example:
{{{
sudo apt-get install xserver-xorg-core-dbg
Line 8: Line 39:
  a. Build .debs for debugging: {{{
DEB_BUILD_OPTIONS="nostrip noopt" sudo apt-get source -b <package>
if the package in question is the ''xserver-xorg-core'' package.

If there is no -dbg package:

 1. Create an `/etc/apt/sources.list.d/ddebs.list` by running the following line at a terminal:
 {{{
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list
Line 11: Line 48:
  a. Install the needed .debs (they will be in the current working directory if the build succeeded): {{{
sudo dpkg --install <package.deb>
 1. Stable releases (not alphas and betas) require three more lines adding to the same file, which is done by the following terminal command:
 {{{
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list
Line 14: Line 55:
 1. Start the program under control of `gdb`: {{{
gdb <program>
(gdb) run <arguments, if any>

 You may also add these lines using the Synaptic Package Manager:
  * Choose ''Synaptic'' via the ''System > Administration'' menu.
  * Choose ''Software Sources'' or ''Repositories'' via the ''Settings'' menu, and click on the ''Third-Party Software'' tab.
  * Click the ''Add'' button and enter each ''deb ...'' line as above one by one and click the ''Add Source'' button (you will have to add these lines one at a time).

 1. Import the debug symbol archive signing key:
 {{{
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
Line 18: Line 65:
 1. The program will start. Perform any actions necessary to reproduce the crash
 1. Retrieve a backtrace of the crash: {{{
(gdb) thread apply all bt full }}}
Line 22: Line 66:
 1. Include the complete output from gdb in your bug report  1. Then run
 {{{
sudo apt-get update
}}}
 to update your package list or click the ''Reload'' button if you used the Synaptic Package Manager.
Line 24: Line 72:
* most of the packages support the build of debugging version in this way. If that doesn't work for a package you can open a bug against it.  1. The debug symbol packages have the '-dbgsym' suffix attached, so to install the debug symbols for the package 'yelp', you first run:
 {{{
apt-cache policy yelp
}}}
 This will show you the version number currently installed (we'll use 2.22.1-0ubuntu2.8.04.1 in this example).
 '''NOTE:''' 'yelp' [which is the package name of "Help and Support" located under the 'System' menu] is ''just an example'' and '''not''' a part of the command: 'yelp' is only used to demonstrate the procedure. You must replace 'yelp' with the name of the package you want to debug.

 1. Install the debug symbols:
 {{{
sudo apt-get install yelp-dbgsym=2.22.1-0ubuntu2.8.04.1
}}}
You can also use the Synaptic Package Manager to search for 'yelp-dbgsym' and install it from there.

The above procedure will install the debug symbol package for yelp only. Chances are that yelp uses shared libraries in other packages and debug symbols for them might be required in order to obtain a readable stack trace. You can download the [[attachment:list-symbols-packages-v2.sh]] shell script and run it (after a chmod a+x list-symbols-packages-v2.sh) this way:

/!\ we updated [[attachment:list-dbgsym-packages-v2.sh]] to [[attachment:list-symbols-packages-v2.sh]]. This is the first change to make it compatible with newer GDB (which does not, by default, load all libraries at start anymore). -v2 is currently only working if you run it against a currently-executing binary (i.e., with '-p $(pidof <whatever>').

 {{{
bash list-symbols-packages-v2.sh yelp
}}}
in order to obtain a more complete list of symbol packages to install. The script may also attach to a running process and in this case it gives an even more complete list of symbol packages to install, and this is recommended if the problem you are having does not prevent you from running the program. You can usually get the process ID of a running process using <code>pidof</code>, so while you have yelp running you might run:
 {{{
bash list-symbols-packages-v2.sh -p $(pidof -s yelp)
}}}
to get a list of packages that should be installed for best debug information. If the running process is a server (daemon), you might have to run 'list-symbols-packages.sh' with sudo.

 a. Now you make a [[Backtrace]].
 a. You can also run [[Valgrind]], if the program crashes with a "Segmentation fault" or "Bus error".
 a. Optionally, you may be asked to produce an [[Strace]].
 a. You can also provide this file : '''~/.xsession-errors'''

=== How do we remove all this stuff after getting the trace and get back to a normal system? ===

You can remove all debug symbol packages and the ddebs repositories with the following commands:

 {{{
sudo apt-get remove \.*-dbgsym
sudo rm /etc/apt/sources.list.d/ddebs.list
sudo apt-get update
}}}


=== References ===

 * Announcement: https://lists.ubuntu.com/archives/ubuntu-devel-announce/2006-September/000195.html

== The Xorg server ==
The X server will by default trap its own crashes and dump a stack trace in /var/log/Xorg.0.log. However, this stack trace is modified by the signal handler itself. To get a "normal" crash, which will trigger a core dump (and ''apport'' reporting), add this to your /etc/X11/xorg.conf:
{{{
Section "ServerFlags"
        Option "NoTrapSignals" "true"
EndSection
}}}
Please see DebuggingXorg for how to debug Xorg server crashes.

== Info for the BugSquad ==

If you're trying to `apport-retrace` a crash report from a bug that didn't happen on the same Ubuntu release as the one you're running, do the following:

Say that you're running `Gutsy` and the crash happened on `Feisty`:

 0. This will create a minimal `feisty` system. {{{
sudo mkdir -p /chroots/feisty
sudo debootstrap feisty /chroots/feisty/}}}
 0. Now you change into this minimal `feisty` system. {{{
sudo chroot /chroots/feisty}}}
 0. edit `/etc/apt/sources/list` and all the repositories you need, especially Martin's ddeb repository.
 0. {{{
apt-get update; apt-get install gdb apport}}}
 0. use `apport-retrace` as you're used to.

For more information on dealing with bug reports [[Apport]] see [[Bugs/ApportRetraces]].

----
CategoryBugSquad CategoryDebugging

Available languages: Italiano,

Debugging Central

This page is part of the debugging series — pages with debugging details for a variety of Ubuntu packages.

This document describes how to debug Ubuntu package crashes and install debug packages on Ubuntu, which will aid in providing information for bugs.

Using apport-retrace

If you want to debug a crash in packaged Ubuntu software, Apport will usually pick it up, create a .crash report in /var/crash/ and report the crashed program. From there it is easiest to use apport-retrace.

apport-retrace regenerates the stack traces (both the simple and the threaded one) from an apport crash report from the included core dump, or gives you a Terminal window with gdb in an environment with debugging symbols, reproducing the situation of the crash through the core dump. For this it figures out the set of necessary packages and their accompanying debug symbol packages, so that the regenerated stack trace will be fully symbolic and thus become much more useful for developers to fix the problem.

apport-retrace has two modes: By default it will just regenerate traces based on the packages which are currently installed in the system, i. e. it assumes that all necessary debug symbols for the report are installed (see the next paragraph for this). When specifying the -S option, it creates a temporary "sandbox" and downloads and installs all necessary packages and debug symbols there. It will not do any changes to your system and does not require any special privileges.

Please see the apport-retrace manpage for details and some examples how to run it.

You can also invoke this directly from the Apport crash notification, by clicking the "Examine locally" button:

  • apport-examine-locally.png

This will collect some package information and then ask you in which mode you want to run apport-retrace:

  • apport-retrace-gui.png

Normally you want to keep the first option, which will get you in a gdb session in the sandbox:

  • apport-retrace-gdb.png

If you do not want an interactive gdb session but just want it to update the already existing .crash file with fully symbolic stack traces, you can select the third option.

Debug Symbol Packages

If you want to debug a crash in a project you are developing yourself or from a third-party package, or need the debug symbols for particular libraries very often, it is helpful to install those permanently into your system.

First, check if there is a package with a -dbg suffix in the main Ubuntu repositories. These are the debug symbol packages, and are equivalent to '-dbgsym' described below. You can safely use either one, but not both at once. Try installing the package, adding -dbg to its name, for example:

sudo apt-get install xserver-xorg-core-dbg

if the package in question is the xserver-xorg-core package.

If there is no -dbg package:

  1. Create an /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:

    echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
    sudo tee -a /etc/apt/sources.list.d/ddebs.list
  2. Stable releases (not alphas and betas) require three more lines adding to the same file, which is done by the following terminal command:
    echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
    deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
    deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
    sudo tee -a /etc/apt/sources.list.d/ddebs.list
    You may also add these lines using the Synaptic Package Manager:
    • Choose Synaptic via the System > Administration menu.

    • Choose Software Sources or Repositories via the Settings menu, and click on the Third-Party Software tab.

    • Click the Add button and enter each deb ... line as above one by one and click the Add Source button (you will have to add these lines one at a time).

  3. Import the debug symbol archive signing key:
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
  4. Then run
    sudo apt-get update

    to update your package list or click the Reload button if you used the Synaptic Package Manager.

  5. The debug symbol packages have the '-dbgsym' suffix attached, so to install the debug symbols for the package 'yelp', you first run:
    apt-cache policy yelp
    This will show you the version number currently installed (we'll use 2.22.1-0ubuntu2.8.04.1 in this example).

    NOTE: 'yelp' [which is the package name of "Help and Support" located under the 'System' menu] is just an example and not a part of the command: 'yelp' is only used to demonstrate the procedure. You must replace 'yelp' with the name of the package you want to debug.

  6. Install the debug symbols:
    sudo apt-get install yelp-dbgsym=2.22.1-0ubuntu2.8.04.1

You can also use the Synaptic Package Manager to search for 'yelp-dbgsym' and install it from there.

The above procedure will install the debug symbol package for yelp only. Chances are that yelp uses shared libraries in other packages and debug symbols for them might be required in order to obtain a readable stack trace. You can download the list-symbols-packages-v2.sh shell script and run it (after a chmod a+x list-symbols-packages-v2.sh) this way:

Warning /!\ we updated list-dbgsym-packages-v2.sh to list-symbols-packages-v2.sh. This is the first change to make it compatible with newer GDB (which does not, by default, load all libraries at start anymore). -v2 is currently only working if you run it against a currently-executing binary (i.e., with '-p $(pidof <whatever>').

  • bash list-symbols-packages-v2.sh yelp

in order to obtain a more complete list of symbol packages to install. The script may also attach to a running process and in this case it gives an even more complete list of symbol packages to install, and this is recommended if the problem you are having does not prevent you from running the program. You can usually get the process ID of a running process using <code>pidof</code>, so while you have yelp running you might run:

  • bash list-symbols-packages-v2.sh -p $(pidof -s yelp)

to get a list of packages that should be installed for best debug information. If the running process is a server (daemon), you might have to run 'list-symbols-packages.sh' with sudo.

  1. Now you make a Backtrace.

  2. You can also run Valgrind, if the program crashes with a "Segmentation fault" or "Bus error".

  3. Optionally, you may be asked to produce an Strace.

  4. You can also provide this file : ~/.xsession-errors

How do we remove all this stuff after getting the trace and get back to a normal system?

You can remove all debug symbol packages and the ddebs repositories with the following commands:

  • sudo apt-get remove \.*-dbgsym
    sudo rm /etc/apt/sources.list.d/ddebs.list
    sudo apt-get update

References

The Xorg server

The X server will by default trap its own crashes and dump a stack trace in /var/log/Xorg.0.log. However, this stack trace is modified by the signal handler itself. To get a "normal" crash, which will trigger a core dump (and apport reporting), add this to your /etc/X11/xorg.conf:

Section "ServerFlags"
        Option "NoTrapSignals" "true"
EndSection

Please see DebuggingXorg for how to debug Xorg server crashes.

Info for the BugSquad

If you're trying to apport-retrace a crash report from a bug that didn't happen on the same Ubuntu release as the one you're running, do the following:

Say that you're running Gutsy and the crash happened on Feisty:

  1. This will create a minimal feisty system.

    sudo mkdir -p /chroots/feisty
    sudo debootstrap feisty /chroots/feisty/
  2. Now you change into this minimal feisty system.

    sudo chroot /chroots/feisty
  3. edit /etc/apt/sources/list and all the repositories you need, especially Martin's ddeb repository.

  4. apt-get update; apt-get install gdb apport
  5. use apport-retrace as you're used to.

For more information on dealing with bug reports Apport see Bugs/ApportRetraces.


CategoryBugSquad CategoryDebugging

DebuggingProgramCrash (last edited 2018-06-19 20:59:53 by cjwatson)