Hi,

When I hit the "pound: HTTP accept: Too many open files"-Problem I ran out of 
fds. That's clear to me.

My question is. How many fds does pound use per request?

During a peak we had 8.000.000 "Too many open files" Errors in the pound log 
out of 10.000.000 requests served during that time.

Does this mean that 80% (8.000.000 "Too may open files" Errors) of all request 
failed? Or does pound use more than 1 filehandle per request?

We use cookie based sessions and have no URL based redirecting onto different 
backends.


thanks,


Tom

Hi again,


On Thursday, 24. April 2008, Tom Fernandes wrote:
> Hi,
>
> When I hit the "pound: HTTP accept: Too many open files"-Problem I ran out
> of fds. That's clear to me.
>
> My question is. How many fds does pound use per request?

in the README I read:

      on most System V derived Unices (of which Linux up to 2.4 is one),
      a thread is a process. This means that when doing a 'ps' you will see
      as many processes with the name 'pound' as there are active threads.
      Each such process uses only two file descriptors, but the system needs
      to support the required number of processes, both in total and per
      user (possibly also per process group). In bash, this is 'ulimit -u',
      in csh this is 'limit maxproc'.


that would mean 2 fds / request - is that still true? Isn't there e.g. a fds 
needed for logging, one for the certificate, one for session tracking when 
using cookie based sessions, ... ? Or are some the once I mentioned only used 
once - the time pound starts?



thanks,


Tom

>
> During a peak we had 8.000.000 "Too many open files" Errors in the pound
> log out of 10.000.000 requests served during that time.
>
> Does this mean that 80% (8.000.000 "Too may open files" Errors) of all
> request failed? Or does pound use more than 1 filehandle per request?
>
> We use cookie based sessions and have no URL based redirecting onto
> different backends.
>
>
> thanks,
>
>
> Tom
>
> --
> To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.

I am under the same problem, but I don´t have enough hardware to
server  10.000.000.

Could you send me your hardware configuration?

Thnaks!

On Thu, Apr 24, 2008 at 10:24 AM, Tom Fernandes <anyaddress(at)gmx.net> wrote:
> Hi again,
>
>
>
>  On Thursday, 24. April 2008, Tom Fernandes wrote:
>  > Hi,
>  >
>  > When I hit the "pound: HTTP accept: Too many open files"-Problem I ran out
>  > of fds. That's clear to me.
>  >
>  > My question is. How many fds does pound use per request?
>
>  in the README I read:
>
>       on most System V derived Unices (of which Linux up to 2.4 is one),
>       a thread is a process. This means that when doing a 'ps' you will see
>       as many processes with the name 'pound' as there are active threads.
>       Each such process uses only two file descriptors, but the system needs
>       to support the required number of processes, both in total and per
>       user (possibly also per process group). In bash, this is 'ulimit -u',
>       in csh this is 'limit maxproc'.
>
>
>  that would mean 2 fds / request - is that still true? Isn't there e.g. a fds
>  needed for logging, one for the certificate, one for session tracking when
>  using cookie based sessions, ... ? Or are some the once I mentioned only
used
>  once - the time pound starts?
>
>
>
>  thanks,
>
>
>  Tom
>
>
>
>  >
>  > During a peak we had 8.000.000 "Too many open files" Errors in the pound
>  > log out of 10.000.000 requests served during that time.
>  >
>  > Does this mean that 80% (8.000.000 "Too may open files" Errors) of all
>  > request failed? Or does pound use more than 1 filehandle per request?
>  >
>  > We use cookie based sessions and have no URL based redirecting onto
>  > different backends.
>  >
>  >
>  > thanks,
>  >
>  >
>  > Tom
>  >
>  > --
>  > To unsubscribe send an email with subject unsubscribe to
pound(at)apsis.ch.
>  > Please contact roseg(at)apsis.ch for questions.
>
>
>
>  --
>  To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
>  Please contact roseg(at)apsis.ch for questions.
>



-- 
Hamilton Vera
int Administrator (char Network[],char ComputationalSystems[]);
Seven Internet http://lib.seven.com.br
"Google is my shepherd, no want shall I know"

On Thu, 2008-04-24 at 15:24 +0200, Tom Fernandes wrote:
> that would mean 2 fds / request - is that still true? Isn't there e.g. a fds 
> needed for logging, one for the certificate, one for session tracking when 
> using cookie based sessions, ... ? Or are some the once I mentioned only used

> once - the time pound starts?

Pound uses 2 fds per active request. The syslog works over UDP, so all
it needs is one socket (regardless of the number of threads). The
certificates are read at start-up, and sessions are tracked in memory.

This really means you need to estimate the number of concurrently active
requests and multiply that by two. Please note that this is ACTIVE
requests - you need to think how many requests per second you get, and
how long it takes to complete a request.

Example: assume you are serving some very large files, and the average
request takes about 15 seconds to complete. From the logs you see an
average of 1000 requests per second. This means you need at least 30000
file descriptors.

As you rightly noted, this holds true on systems where the threading
library works as a single process. If a thread is an independent process
then you need 3 fds per process, but allow a limit of 15000 processes.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

Hello,

On Friday, 25. April 2008, Hamilton Vera wrote:
> I am under the same problem, but I dont have enough hardware to
> server  10.000.000.
>
> Could you send me your hardware configuration?

we have a dual-quadcore system with 16GB RAM - but this doesn't mean anything. 
The system holds pound _and_ one of the webservers.

We run debian etch on amd64 and a backported version of pound (2.4-2). We had 
regular segfaults (in kern.log) with the standard version 2.0-1.2 in debian 
etch although pound never actually crashed.

Pound only uses upto 1.7GB (VIRT) and 400MB (RSS) once it's pretty busy 
(serving roughly 1300 req/s - at lot of them HTTPS). At that point pound uses 
100% of one of the CPUs (one processor has 2327.507 Mhz).

So - AFAICS having a SMP system for pound doesn't help you much but rather 
some number crunching singel-processor machine. I might be wrong but this is 
the impression I got from what I observed as pound never used more than 100% 
when looking at the output of top.

You might be better of using a different OS - then a SMP machine might help.

Regarding file descriptors I observed the following:

 * it is correct, that pound uses 2/request (based on testing with ab (apache 
benchmark) and wget)
 * if you use FF and surf a website pound will open 4 fds for all the 
    HTTP requests you make if you surf different websites on the same host.
    IE (6 and 7) will open 8 instead.
 * the time pound waits for closing the connection is (at least) based on the
    "Client" value in the pound configuration. This seems to be equivalent 
    (or similar) to Keep-Alive in HTTP. We have a value of 300 which helps not
    having to do the TCP handshake again and again for new connections (which
    saves a bit of bandwidth for the server and the client). You might live 
    very well with a lower value. But - with this high value the FDS for a 
    connections remain open upto 300s after the browsers last request.
 * when pound runs out of open FDS I find two different types of errors in the 
    logfile:

	pound: HTTP accept: Too many open files

and

	pound: (2aaacc189960) e503 backend 10.0.1.11:80 socket create: Too
	many open files

From what I observed with ab tests only the second message indicates a failed 
HTTP request where the client received a 503 error page / status.


What I did to get an estimate of the FDS actually used by pound is run:

while true; do ls -1 /proc/$PID_OF_POUND/fd/ | wc -l; sleep 1s;  done

for 5m and calculate the number of unique IPs during that time.



I also came up with the following calculation:

	2000 x 6.5 = 13.000

where 1800 was the number of unique IPs during the 5m (remember we have 
a "Client" value in the pound config of 300) and 6.5 is the estimate number 
of FDS (based on a relation of 2/3 for FF/IE - FF uses 4 FDS and IE 8)

This is how I found out how we crossed our limit of 12.000 filehandles while 
only having 1300 HTTP requests/s :-) .


Of course - please correct me if / where I am wrong...



hope this helps,


Tom

>
> Thnaks!
>
> On Thu, Apr 24, 2008 at 10:24 AM, Tom Fernandes <anyaddress(at)gmx.net>
wrote:
> > Hi again,
> >
> >  On Thursday, 24. April 2008, Tom Fernandes wrote:
> >  > Hi,
> >  >
> >  > When I hit the "pound: HTTP accept: Too many open files"-Problem I ran
> >  > out of fds. That's clear to me.
> >  >
> >  > My question is. How many fds does pound use per request?
> >
> >  in the README I read:
> >
> >       on most System V derived Unices (of which Linux up to 2.4 is one),
> >       a thread is a process. This means that when doing a 'ps' you will
> > see as many processes with the name 'pound' as there are active threads.
> > Each such process uses only two file descriptors, but the system needs to
> > support the required number of processes, both in total and per user
> > (possibly also per process group). In bash, this is 'ulimit -u', in csh
> > this is 'limit maxproc'.
> >
> >
> >  that would mean 2 fds / request - is that still true? Isn't there e.g. a
> > fds needed for logging, one for the certificate, one for session tracking
> > when using cookie based sessions, ... ? Or are some the once I mentioned
> > only used once - the time pound starts?
> >
> >
> >
> >  thanks,
> >
> >
> >  Tom
> >
> >  > During a peak we had 8.000.000 "Too many open files" Errors in the
> >  > pound log out of 10.000.000 requests served during that time.
> >  >
> >  > Does this mean that 80% (8.000.000 "Too may open files" Errors) of all
> >  > request failed? Or does pound use more than 1 filehandle per request?
> >  >
> >  > We use cookie based sessions and have no URL based redirecting onto
> >  > different backends.
> >  >
> >  >
> >  > thanks,
> >  >
> >  >
> >  > Tom
> >  >
> >  > --
> >  > To unsubscribe send an email with subject unsubscribe to
> >  > pound(at)apsis.ch. Please contact roseg(at)apsis.ch for questions.
> >
> >  --
> >  To unsubscribe send an email with subject unsubscribe to
pound(at)apsis.ch.
> >  Please contact roseg(at)apsis.ch for questions.

Am 25.04.2008 um 21:03 schrieb Tom Fernandes:

> We run debian etch on amd64 and a backported version of pound (2.4-2).

Can you please tell me a source where to get this backported packages?

Falk

Hi,

On Monday, 28. April 2008, smartTERRA NOC wrote:
> > We run debian etch on amd64 and a backported version of pound (2.4-2).
>
> Can you please tell me a source where to get this backported packages?

I backported it myself.
Here's the diff between my package and the debian unstable one:


tom.fernandes(at)kronos:~/building/pound$ debdiff pound_2.4-2.dsc 
pound_2.4-2bm.dsc
diff -u pound-2.4/debian/compat pound-2.4/debian/compat
--- pound-2.4/debian/compat
+++ pound-2.4/debian/compat
(at)(at) -1 +1 (at)(at)
-6
+5
diff -u pound-2.4/debian/changelog pound-2.4/debian/changelog
--- pound-2.4/debian/changelog
+++ pound-2.4/debian/changelog
(at)(at) -1,3 +1,15 (at)(at)
+pound (2.4-2bm) unstable; urgency=low
+
+  * backport for etch
+  * debian/control:
+    + change debhelper version from 6 to 5
+  * debian/compat:
+    + change version from 6 to 5
+  * debian/rules:
+    + remove -fstack-protector from CFLAGS to make pound compile under etch
+
+ -- Tom Fernandes <fernandes(at)xyz>  Tue, 22 Apr 2008 20:43:11 +0000
+
 pound (2.4-2) unstable; urgency=medium

   [Michael Mende]
diff -u pound-2.4/debian/control pound-2.4/debian/control
--- pound-2.4/debian/control
+++ pound-2.4/debian/control
(at)(at) -3,7 +3,7 (at)(at)
 Priority: extra
 Maintainer: Michael Mende <debian(at)menole.net>
 Uploaders: Michael Piefel <piefel(at)debian.org>
-Build-Depends: debhelper (>= 6), libssl-dev (>= 0.9.7), autotools-dev, 
libpcre3-dev
+Build-Depends: debhelper (>= 5), libssl-dev (>= 0.9.7), autotools-dev, 
libpcre3-dev
 Standards-Version: 3.7.3
 Homepage: http://www.apsis.ch/pound/

diff -u pound-2.4/debian/rules pound-2.4/debian/rules
--- pound-2.4/debian/rules
+++ pound-2.4/debian/rules
(at)(at) -27,7 +27,7 (at)(at)
        INSTALL_PROGRAM += -s
 endif
 ifneq ($(DEB_HOST_GNU_CPU),arm)
-       CFLAGS += -fstack-protector
+       #CFLAGS += -fstack-protector
 endif
 CFLAGS += -fPIE
 LDFLAGS += -Wl,-z,relro,-z,noexecstack -pie



hope this helps,


Tom