220.127.116.11 leaks filedescriptors?
We've upgraded some of our production icecast servers to 18.104.22.168 this morning (from 2.4.2, because of the better TLS support). 1 one the 8 upgraded servers is expierencing problems which seem to be known as a "filedescriptor leak". The server does not respond to new connections anymore, but seems to handle a lot of existing connections still fine. With netstat and lsof a lot of CLOSE_WAIT connections show up:
$ sudo lsof -np 28035 | wc -l 12895
$ sudo lsof -np 28035 | grep CLOSE_WAIT | wc -l 5973
The other 7 servers do not experience this problem (yet). The maximum amount of listeners today was about 6500 per server (on a number of channels). Could it be that icecast is leaking filedescriptors somehow in situations that are not always triggered (since not all servers encounter this problem)? All servers are running the same kernel; 4.1.39 and the installation is identical on all systems.
Any suggestions what could be done to debug this problem? It would be nice if we could reproduce this, but I have no clue what could cause this.