SIGSEGV (Segmentation fault)
One customer of ours is using ShoutCast to stream content to our platform. Due to this fact he uses two Icecast source servers. Last Friday both the source servers where killed due to an unknown reason.
After restarting the servers they kept crashing at a random order. The logfiles (loglevel 4) just stopped without any form of error. Additional debugging was required, so i started source01 in a strace and source02 in gdb. They have crashed multiple times now, so additional debug information can be supplied at request.
Addon: After investigating with the customer it seems that there is something wrong with the connection from the customer to us. Both connections seems to be flapping randomly. I think due to this fact, Icecast killes itself somehow. (Please note, there are multiple customers on both sources, but this customer is using both sources. So I suspect that he is the bad-guy ;-)
Version information: Icecast 2.3.1 on Debian 3.1
Please note: I've used a self compiled version of Icecast. To narrow it down I've reinstalled the application (make clean;make;make install) to make sure it's compiled against the newest library's. This still makes Icecast crash. Secondly I've used the backports.org package to install icecast (icecast2_2.3.1-1bpo1_i386.deb) with "apt-get install icecast2". This should be compiled and tested against correct library's. Still no effect.
We've been using Icecast for some years now and are really happy with the product. So lets make it a even better product ;-)
Debug output
Last output on source01 (strace)
recv(9, "GET /status.xsl HTTP/1.1\r\nHost: "..., 4095, 0) = 81
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}], 2, 100) = 0
time(NULL) = 1162946180
recv(8, 0x807fe40, 4095, 0) = -1 EAGAIN (Resource temporarily unavailable)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Last output on source02 (strace):
fcntl64(11, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
setsockopt(11, SOL_TCP, TCP_NODELAY, [1], 4) = 0
time(NULL) = 1162774575
recv(11, "GET /status.xsl HTTP/1.1\r\nHost: "..., 4095, 0) = 81
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}], 2, 100) = 0
time(NULL) = 1162774575
recv(15, 0x80ba608, 4095, 0) = -1 EAGAIN (Resource temporarily unavailable)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Last output on source02 (gdb)
(gdb) run /usr/bin/icecast2 -c /etc/icecast_source02.xml
Starting program: /usr/bin/icecast2 /usr/bin/icecast2 -c /etc/icecast_source02.xml
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread -1213922208 (LWP 22162)]
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
Changed groupid to 65534.
Changed userid to 65534.
[New Thread -1214133328 (LWP 22163)]
[New Thread -1214657616 (LWP 22164)]
[New Thread -1215181904 (LWP 22165)]
[New Thread -1215706192 (LWP 22166)]
[New Thread -1216230480 (LWP 22167)]
[New Thread -1216754768 (LWP 22168)]
[New Thread -1217279056 (LWP 22172)]
[New Thread -1217803344 (LWP 22188)]
[New Thread -1218327632 (LWP 22205)]
[Thread -1218327632 (LWP 22205) exited]
[New Thread -1218327632 (LWP 22306)]
[Thread -1218327632 (LWP 22306) exited]
[New Thread -1218327632 (LWP 22367)]
[Thread -1218327632 (LWP 22367) exited]
[New Thread -1218327632 (LWP 22446)]
[Thread -1218327632 (LWP 22446) exited]
[New Thread -1218327632 (LWP 22552)]
[Thread -1218327632 (LWP 22552) exited]
[New Thread -1218327632 (LWP 22669)]
[Thread -1218327632 (LWP 22669) exited]
[New Thread -1218327632 (LWP 22802)]
[Thread -1218327632 (LWP 22802) exited]
[New Thread -1218327632 (LWP 22866)]
[Thread -1218327632 (LWP 22866) exited]
[New Thread -1218327632 (LWP 22927)]
[Thread -1218327632 (LWP 22927) exited]
[New Thread -1218327632 (LWP 22999)]
[Thread -1218327632 (LWP 22999) exited]
[New Thread -1218327632 (LWP 23083)]
[Thread -1218327632 (LWP 23083) exited]
[cut]
[New Thread -1218327632 (LWP 25122)]
[Thread -1218327632 (LWP 25122) exited]
[New Thread -1218327632 (LWP 25226)]
[Thread -1218327632 (LWP 25226) exited]
[New Thread -1218327632 (LWP 25302)]
[Thread -1218327632 (LWP 25302) exited]
[New Thread -1218327632 (LWP 25482)]
[Thread -1218327632 (LWP 25482) exited]
[New Thread -1218327632 (LWP 25576)]
[Thread -1217279056 (LWP 22172) exited]
[Thread -1218327632 (LWP 25576) exited]
[New Thread -1218327632 (LWP 25784)]
[Thread -1218327632 (LWP 25784) exited]
[New Thread -1218327632 (LWP 25926)]
[Thread -1218327632 (LWP 25926) exited]
[New Thread -1218327632 (LWP 26151)]
[Thread -1218327632 (LWP 26151) exited]
[New Thread -1218327632 (LWP 26242)]
[Thread -1218327632 (LWP 26242) exited]
[New Thread -1218327632 (LWP 26329)]
[Thread -1218327632 (LWP 26329) exited]
[New Thread -1218327632 (LWP 26616)]
[Thread -1218327632 (LWP 26616) exited]
[New Thread -1218327632 (LWP 27193)]
[Thread -1218327632 (LWP 27193) exited]
[New Thread -1218327632 (LWP 27389)]
[Thread -1218327632 (LWP 27389) exited]
[New Thread -1218327632 (LWP 27508)]
[Thread -1218327632 (LWP 27508) exited]
[New Thread -1218327632 (LWP 27624)]
[Thread -1218327632 (LWP 27624) exited]
[New Thread -1218327632 (LWP 27777)]
[Thread -1218327632 (LWP 27777) exited]
[New Thread -1218327632 (LWP 27968)]
[Thread -1218327632 (LWP 27968) exited]
[New Thread -1218327632 (LWP 28249)]
[Thread -1218327632 (LWP 28249) exited]
[New Thread -1218327632 (LWP 28407)]
[Thread -1218327632 (LWP 28407) exited]
[New Thread -1218327632 (LWP 28539)]
[Thread -1218327632 (LWP 28539) exited]
[New Thread -1218327632 (LWP 28630)]
[Thread -1218327632 (LWP 28630) exited]
[New Thread -1218327632 (LWP 28719)]
[Thread -1218327632 (LWP 28719) exited]
[New Thread -1218327632 (LWP 28799)]
[Thread -1218327632 (LWP 28799) exited]
[New Thread -1218327632 (LWP 28885)]
[Thread -1218327632 (LWP 28885) exited]
[New Thread -1218327632 (LWP 28999)]
[Thread -1218327632 (LWP 28999) exited]
[New Thread -1218327632 (LWP 29427)]
[Thread -1218327632 (LWP 29427) exited]
[New Thread -1218327632 (LWP 29614)]
[Thread -1218327632 (LWP 29614) exited]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1213922208 (LWP 22162)]
0x0804faaf in ?? ()
(gdb) bt
#0 0x0804faaf in ?? ()
#1 0x00000064 in ?? ()
#2 0x0806ac4a in _IO_stdin_used ()
#3 0x00000fff in ?? ()
#4 0x080a7858 in ?? ()
#5 0x31333032 in ?? ()
#6 0x0000000f in ?? ()
#7 0x08070d34 in xmlFree ()
#8 0x00000002 in ?? ()
#9 0x00000000 in ?? ()
#10 0x0806c99e in _IO_stdin_used ()
#11 0xbfb194b8 in ?? ()
#12 0x0804fe94 in ?? ()
#13 0x0809b368 in ?? ()
#14 0x0806c99e in _IO_stdin_used ()
#15 0x00000000 in ?? ()
#16 0x00000000 in ?? ()
#17 0x000001a8 in ?? ()
#18 0x0806ac03 in _IO_stdin_used ()
#19 0x08095a00 in ?? ()
#20 0x00000001 in ?? ()
#21 0x00000003 in ?? ()
#22 0x0806a5fb in _IO_stdin_used ()
#23 0xbfb194c8 in ?? ()
#24 0x0804e9f6 in ?? ()
#25 0x0000024d in ?? ()
#26 0x00000001 in ?? ()
#27 0xbfb19b18 in ?? ()
#28 0x0804edf0 in ?? ()
#29 0x00000000 in ?? ()
#30 0x00000003 in ?? ()
#31 0x0806a5fb in _IO_stdin_used ()
#32 0x0806bfd9 in _IO_stdin_used ()
#33 0x0806a5de in _IO_stdin_used ()
#34 0x00000000 in ?? ()
#35 0xb7f1dca0 in ?? () from /lib/ld-linux.so.2
#36 0x0804973f in ?? ()
#37 0x0804973f in ?? ()
#38 0xbfb195b8 in ?? ()
#39 0xbfb194c0 in ?? ()
#40 0x00000000 in ?? ()
#41 0x00000000 in ?? ()
#42 0xb7f1dca0 in ?? () from /lib/ld-linux.so.2
#43 0x0804960c in ?? ()
#44 0x0804960c in ?? ()
#45 0xbfb195d4 in ?? ()
#46 0xbfb194dc in ?? ()
#47 0xb7f12ff6 in _dl_catch_error () from /lib/ld-linux.so.2
Previous frame inner to this frame (corrupt stack?)
(gdb)