Indefinite 100% CPU when disconnecting from SSL port when Icecast is running inside a Podman container
UPDATE: This bug seems hard to track down and the solution as described below of spawning the Podman container as root rather than rootlessly doesn't actually work. It definitely seemed to be the fix last night, but I've now encountered the same high CPU usage problem that spawning as rootless was producing, but as root. Noteworthy is that I tested the same build process on the host system without containers, and the bug was no more, but judging how the bug has seemed to have has disappeared completely and then returned a few times, I'm not certain of the results. Leaving this report up as it could still lead to an answer.
The easiest way for me to communicate the issue is through these steps:
- Create a Let's Encrypt cert.
- Create
icecast.pem
using what Let's Encrypt produced:fullchain.pem
+privkey.pem
. - Place
icecast.pem
into same dir asContainerfile
andentrypoint.sh
(attached files). - Install Podman.
- As an unprivileged user, run:
podman build --force-rm -f Containerfile -t icecast
. This builds Icecast 2.4.4 from source with--with-curl
and--with-openssl
flags (OpenSSL/1.1.1g
is what Alpine currently serves). - Open ports
8000/tcp
and8001/tcp
on the host if needed. - Spawn an Icecast container (replace
<hostname>
):
podman run --rm --interactive --tty --publish 8000:8000 --publish 8001:8001 \
--env IC_HOSTNAME="<hostname>" \
--env IC_HTTP_PORT="8001" \
--env IC_HTTPS_PORT="8000" \
--env IC_LOG_LEVEL="4" \
--name icecast \
icecast
- To monitor Icecast's logs via the host, you can use
podman exec icecast tail /usr/local/var/log/icecast/access.log -f
andpodman exec icecast tail /usr/local/var/log/icecast/error.log -f
. - Stream to the server's HTTPS port
8000
using a source client (in my case, Butt on Windows) with the default credentials. - After connection, disconnect.
htop
on the host should now report 100% CPU for the container process, and reconnection via the source client should no longer be possible. - Pressing
s
inhtop
on one of the twoicecast -c /usr/local/etc/icecast.xml
commands listed running at ~100% CPU,strace
shows intense polling spam:
...
poll([{fd=7, events=POLLIN}], 1, 250) = 1 ([{fd=7, revents=POLLIN}])
read(7, read(7, read(7, read(7, ) = 1 ([{fd=7, revents=POLLIN}])
poll([{fd=7, events=POLLIN}], 1, 250) = 1 ([{fd=7, revents=POLLIN}])
read(7, "", 5) = 0
read(7, read(7, read(7, poll([{fd=7, events=POLLIN}], 1, 250) = 1 ([{fd=7, revents=POLLIN}])
...
For the other high CPU Icecast command, strace
shows less spam:
...
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}], 2, 300) = 0 (Timeout)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}], 2, 300) = 0 (Timeout)
...
- With the container still running, running
podman exec icecast netstat -t
on the host shows Alpine waiting:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 <hostname>:46372 <hostname>:8000 FIN_WAIT2
tcp 0 0 <hostname>:8000 <hostname>:46372 CLOSE_WAIT
netstat: /proc/net/tcp6: No such file or directory
It looks like the connection is never closed properly, as visiting https://<hostname>:8000/admin/
and pressing Kill Source
brings things back to normal. This is specifically confined to SSL as connecting to the HTTP port 8001
and then disconnecting has no adverse effects. Only the HTTPS port is affected.
The solution for me has been to spawn the Podman container as root rather than rootlessly. When doing so, Icecast doesn't get stuck in a loop waiting for the connection to drop after disconnection from HTTPS port (or whatever the exact issue is). Ideally, Icecast could function in a container run rootlessly as this is a major advantage of Podman's approach to containers over Docker's.
I am unsure if there is a bug on Podman's side (concerning container networking, or possible misconfiguration on my part), but I don't believe Icecast should be able to get stuck in this scenario producing 100% CPU indefinitely. Therefore, I have reported this here rather than to Podman.
Thanks.
Infos:
Icecast 2.4.4 (from source) running on Alpine inside the container
Podman 1.9.3
Fedora release 32 (Thirty Two)
Linux 5.6.19-300.fc32.x86_64