Streaming over SSL disconnects
I reported this via IRC but wanted to make sure a proper bug report gets filed before moving on.
I'm building a streaming radio service for Newgrounds.com. More than one application has an embedded radio player and these applications are all served over SSL.
In order to avoid mixed content warnings, I built Icecast 2.4.3 with SSL support so I could get status info and serve the stream over SSL as well.
(None of the Debian or Ubuntu packages have been built with SSL yet for some reason.)
I'm using Debian 8 x64. libssl is 1.0.1t (which is the latest 1.0.1 LTS version). liquidsoap 1.1.1 is the source and connects to Icecast via port 80 on localhost (both liquidsoap and icecast are on the same VM).
I have six mounts configured all the same. liquidsoap is re-encoding the mp3s at a constant bitrate of 128k. I've uploaded a stripped down Icecast config.
Here is what happens: normally, streaming over SSL works perfectly fine. But sometimes, either right at stream start or at some seemingly random amount of time later, one of the mounts becomes "corrupted." The web player (using a simple tag) quickly buffers and disconnects. VLC reconnects and gets disconnected over and over.
I've tested this on my local network over wifi using the web player, VLC, and iTunes, and with ServeStream on Android over cellular. All same result.
If I connect to the stream via port 80, I do not get disconnected. I get disconnected from the stream ONLY when streaming over SSL and ONLY when the mount becomes corrupted.
Observations:
-
This happens on every mount seemingly at random, but it seems that only ONE mount becomes corrupted at a time.
-
Doesn't seem to be triggered by any one song, a track change, or anything else station related that I can find.
-
Happens if I use ogg or mp3.
-
Shutting down the liquidsoap instance for the corrupted mount, waiting a minute or so (a quick restart doesn't seem to fix it) and then restarting liquidsoap seems to resolve the issue (at least temporarily).
-
Nothing in the logs that I can see, even with debug logging on:
[2016-08-21 20:30:08] DBUG stats/modify_node_event update "/electronic" listeners (1)
[2016-08-21 20:30:08] DBUG format/format_check_http_buffer processing pending client headers
[2016-08-21 20:30:08] DBUG client/client_send_bytes Client connection died
[2016-08-21 20:30:08] DBUG source/source_main Client removed
[2016-08-21 20:30:08] INFO source/source_main listener count on /electronic.mp3 now 0
These cycle over and over again as the client disconnects and reconnects. I can find nothing else in any of the logs on the system.
- This happens on both staging and production which is on two separate servers.
I was asked to run Icecast using valgrind. We saw lots of SSL errors, so one possible issue could be a buggy system libssl (which I do not think is the case at this point). I even got the latest OpenSSL 1.0.2h built and then built Icecast against that to rule out 1.0.1t causing a bunch of errors.
I will upload the Valgrind output for both 1.0.1t and 1.0.2h.
I simply started Icecast and then shut it down via ctrl-c. This was NOT running while experiencing a corrupted mount.
That's about as much information as I think I have at this point. I have decided to abandon using SSL for the streams as it simply does not seem to be production ready at this point. This will cause mixed content errors in our applications, but streaming over http seems much more stable.
I'd be happy to help provide more information if requested.