Debugging

Loops with 100% CPU usage

If callweaver stop responding because it enter an infinite loop that brings it CPU usage up around 100% those are the steps to do to provide usefull debugging information before filing a new ticket:

• use “top” or “ps” to know wich is the PID of the callweaver proces acting crazy, for example: 2345 ;
• use “gdb” to connect to it while it is still running:

  gdb /usr/sbin/callweaver 2345

• lots of messages will appear, when they stop turn on logging to the “gdb.txt” file with the command:

set logging on

• give more commands:

bt
bt full
info thread
thread apply all bt
quit

• now gather the file “gdb.txt” and attach it when submitting a new ticket, don’t forge t you can compress it with “gzip” or “bzip2” if it is too big;
• if you wish to generate a “snapshot” of the actual state of the process for later inspecting just use the command

  genereate-core-file

  it will dump a “core.PID” file like when processes crashes;

Crashes

If callweaver crash, you need to start it with the -g command line parameters to permit gather “core” files. If callweaver was running with PID number 2345 then a crash will create file called “core.2345”.

• to extract debugging info from this file execute the command:

  gdb /usr/sbin/callweaver core.2345

• like seen before to the follow gdb commands when all the info stop loading onto the screen:

set logging on
bt
bt full
info thread
thread apply all bt
quit

• now gather the file “gdb.txt” and attach it when submitting a new ticket, don’t forge t to compress it;

The Easy Way

If you’re connected to a server just edit the variable ADMINEMAIL inside contrib/scripts/callweaver_grab_core then launch this script to receive on your email all the needed infoes.

From where to start to find/fix the bugs?

You have a core and can inspect manually using gdb:

• if you are connected to a server via SSH i suggest you to use cgdb a very nice frontend to gdb (it split the screen into two parts: upper source, lower gdb command line);
• tutorial debug with gdb
• tutorial gdb plus

I found a couple bug myself porting my stuff from Asterisk to CallWeaver:

• for OGI: ./res/res_ogi.c
• for the Manager network interface: ./corelib/manager.c
• music on hold problems: ./res/res_musiconhold.c
• briding and channel management: ./corelib/channel.c

Dead Locks

Since CallWeaver uses threads a lot it can happen some deadlock situation expecially when your developing some new code inside the core. At the moment i can only provide generic info about this:

• http://www.llnl.gov/computing/tutorials/pthreads/