We
encountered a problem earlier today where pound had too many open
files. This had never happened before (in 3 years of running pound),
and happened right after we lost one of our web servers unexpectedly.
We have a HAPort setup for all of our backends (port other than 80),
and its checked every 15 sec. At the time when pound started
complaining about too many open files, it hadn't removed the dead
backend from available list.
I believe I have tracked the problem to the fact that "shutdown" is not
being called prior to "close(socket)" on line 774 of http.c V2.4.2.
Looking at the rest of the code, every time a socket is about to be
closed, shutdown is called. I don't know if this is the problem we
encountered with too many file handles being opened, but I was
wondering if shutdown() needs to be called before close(). Here's the
snippet of pound errors when this occurred:
Sep 10 09:45:36 p2 pound: (b4429ba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:37 p2 pound: (b3b05ba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:37 p2 pound: (b3b05ba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:37 p2 pound: (b487aba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:37 p2 pound: (b487aba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:38 p2 pound: HTTP accept: Too many open files
Sep 10 09:45:38 p2 pound: (b52e3ba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:38 p2 pound: (b52e3ba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:34 p2 pound: (b4e51ba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:34 p2 pound: (b4e51ba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:34 p2 pound: (b697cba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:34 p2 pound: (b697cba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:35 p2 pound: (b509aba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:35 p2 pound: (b509aba0) backend 192.168.111.23:80 connect:
Connection timed out
Sep 10 09:45:35 p2 pound: (b70d9ba0) connect_nb: error after
getsockopt: Connection timed out
Sep 10 09:45:35 p2 pound: (b70d9ba0) backend 192.168.111.23:80 connect:
Connection timed out