Ices leaks memory after losing connection(s) to icecast
We run ices2 with three seperate streams - one unchanged, one a quality 0
circa 64kbps stream, and the other a quality -1 22khz mono stream. It
seems that when cron.daily runs, one or more of these streams will lose
connectivity with the icecast2 server (which is running on another box).
The number of streams dropped varies from day to day. After this point,
it appears that ices leaks memory until it eventually runs out of memory
and is killed by the kernel.
To illustrate this, here's a ps -aux from when ices was started yesterday
afternoon (US time):
kirk 8589 0.1 1.5 11376 3876 tty3 S 16:23 0:00 ../ices
egoplay.xml
kirk 8591 0.0 1.5 11376 3876 tty3 S 16:23 0:00 ../ices
egoplay.xml
kirk 8592 13.6 1.5 11376 3876 tty3 S 16:23 0:04 ../ices
egoplay.xml
kirk 8593 0.0 1.5 11376 3876 tty3 S 16:23 0:00 ../ices
egoplay.xml
kirk 8594 4.8 1.5 11376 3876 tty3 S 16:23 0:01 ../ices
egoplay.xml
These are pretty typical of what it looked like prior to the streams being
disconnected at 01:01 this morning. Here's what it looked like at about
03:42:
kirk 8589 0.0 69.2 185692 177352 ? S Oct31 0:01 ../ices
egoplay.xml
kirk 8591 0.0 69.2 185692 177352 ? S Oct31 0:00 ../ices
egoplay.xml
kirk 8592 16.8 69.2 185692 177352 ? R Oct31 114:23 ../ices
egoplay.xml
kirk 8593 6.1 69.2 185692 177352 ? R Oct31 41:58 ../ices
egoplay.xml
kirk 8594 9.6 69.2 185692 177352 ? R Oct31 65:16 ../ices
egoplay.xml
And here's what it looked like just before Kirk killed it off just before
6Am:
kirk 8589 0.0 80.1 320268 205368 ? S Oct31 0:03 ../ices
egoplay.xml
kirk 8591 0.0 80.1 320268 205368 ? S Oct31 0:00 ../ices
egoplay.xml
kirk 8592 18.3 80.1 320268 205368 ? R Oct31 149:13 ../ices
egoplay.xml
kirk 8593 9.4 80.1 320268 205368 ? R Oct31 76:49 ../ices
egoplay.xml
kirk 8594 12.3 80.1 320268 205368 ? R Oct31 100:08 ../ices
egoplay.xml
Left to its own devices, it would have run out of memory.
The logs don't say a lot. The ices log says nothing, really. The only
pointer that something's gone wrong is the fact that the encoder and
resample initialisations stop appearing in the log with each song. The
songs continue to be printed, however.
the icecast error log seems to indicate that the source just died at the
other end.
[2002-11-01 01:01:40] WARN source/source_main Disconnecting source:
socket timeout (10 s) expired
[2002-11-01 01:01:40] INFO source/source_main Removing source following
disconnection
[2002-11-01 01:01:40] DBUG source/source_main Source exiting
[2002-11-01 01:01:40] WARN source/source_main Disconnecting source:
socket timeout (10 s) expired
[2002-11-01 01:01:40] INFO source/source_main Removing source following
disconnection
[2002-11-01 01:01:40] DBUG source/source_main Source exiting
[2002-11-01 01:01:40] WARN source/source_main Disconnecting source:
socket timeout (10 s) expired
[2002-11-01 01:01:40] INFO source/source_main Removing source following
disconnection
[2002-11-01 01:01:40] DBUG source/source_main Source exiting
This has been happening consistantly since I first wrote about it on the
9th of October.
http://www.xiph.org/archives/icecast/3367.html
http://www.xiph.org/archives/icecast/3401.html
This is with current CVS ices and libshout2.
What I speculate is happening is that, for whatever reason, probably due
to system load with updatedb or sometihng, one or more streams lose their
connection with icecast. Despite the settings in the icex XML config, no
attempts are made to reconnect, acording to all of the logs. So some part
of ices seems to think that they're still connected.
