/ Zope / Apsis / Pound Mailing List / Archive / 2007 / 2007-05 / Re: [Pound Mailing List] Re: excess CPU usage?

[ << ] [ >> ]

[ AUTO: Tim Siney is out of the office. / ... ] [ host dead/resurrect on static content. / Albert ... ]

Re: [Pound Mailing List] Re: excess CPU usage?
Robert Segall <roseg(at)apsis.ch>
2007-05-04 16:28:00 [ FULL ]
On Thu, 2007-05-03 at 15:30 -0400, David Barr wrote:[...]

Try "configure --with-t_rsa=21600" and Pound will recompute the
ephemeral keys only once every 6 hours.[...]

Re: [Pound Mailing List] blocked requests with increased concurrency
Robert Segall <roseg(at)apsis.ch>
2007-05-04 16:36:38 [ FULL ]
On Fri, 2007-05-04 at 13:13 +0200, RedShift wrote:[...]

Pound does no storage - once the request headers are read in and
processed they are sent to the back-end, following which the request
body is read from the client (possibly in packet-sized pieces) and sent
on. Same mechanism for the response.

I have no idea where the delays come from - first suspect is the
threading library (bad scheduling), second is the back-end (delays in
reading the request or generating the response), third is the networking
layer in your OS (it chokes if too many connections are open
simultaneously, or too many fragments are pending) and we can't really
exclude hardware (we have seen nasty performance problems in the past
due to lousy network cards). The delays seem to occur in very few
requests (usually under 1%) and not on all systems (RedHat may be
particularly prone to this, though that could be only a coincidence).

I would certainly appreciate more data on this - for the moment I have
no solution.[...]

Re: [Pound Mailing List] blocked requests with increased concurrency
Robert Segall <roseg(at)apsis.ch>
2007-05-04 16:38:08 [ FULL ]
On Wed, 2007-05-02 at 13:00 -0400, Albert wrote:[...]

Older RedHat had a peculiar implementation of the threading library. I
am not familiar enough with the newer versions to answer that, but 2.6.9
is not exactly the newest kernel.[...]

Re: [Pound Mailing List] Documentaion woes
Robert Segall <roseg(at)apsis.ch>
2007-05-04 16:54:06 [ FULL ]
On Thu, 2007-05-03 at 15:20 -0400, Wesley Moxam wrote:[...]

Thanks for pointing that out - we'll fix it.[...]

Re: [Pound Mailing List] blocked requests with increased concurrency
Khaled Hassounah <khaled.hassounah(at)medhelp.org>
2007-05-04 19:08:59 [ FULL ]
Robert,
[...]

Is there any different kind of info that you think might be helpful? or 
did you mean the same kind of data from different systems to draw a pattern?

Khaled

Re: [Pound Mailing List] blocked requests with increased concurrency
Albert <pound(at)alacra.com>
2007-05-04 19:26:51 [ FULL ]
I don't know if it helps, but when I link, here are my flags:
"gcc"  -pthread -o pound pound.o http.o config.o svc.o -lpcreposix -lssl 
-lcrypt o -lresolv -ldl   -lm -ltcmalloc

When running configure, its not able to get find libphtreads  (with an 
's' in the end) libarary.  Is it available for RedHat 2.6.9 kernel?



Khaled Hassounah wrote:[...][...][...]
Attachments:  
text.html text/html 1119 Bytes

Re: [Pound Mailing List] blocked requests with increased concurrency
Khaled Hassounah <khaled.hassounah(at)medhelp.org>
2007-05-06 23:27:39 [ FULL ]
My config.log says that -libpthreads was not found on RedHat 2.6.9 
kernel as well. But it resorts to using -pthread later. I believe those 
to be equivalent as -lpthread set the compiler and linkers flags to 
include pthreads.

Khaled

Albert wrote:[...]

Re: [Pound Mailing List] blocked requests with increased concurrency
Stefan Lambrev <stefan.lambrev(at)sun-fish.com>
2007-05-08 08:38:54 [ FULL ]
Robert Segall wrote:[...][...][...]
Red Hate have their own version numbering that is not the same as 
vanilla kernel.
So 2.6.9 actually is the latest kernel for red hat :)
2.6.9-42.0.10 is not old kernel.
I'm not sure about the threading library in this red hat, but I think it 
is merged from 2.6.18/19 vanilla kernel.
[...]

Re: [Pound Mailing List] blocked requests with increased concurrency
Stefan Lambrev <stefan.lambrev(at)sun-fish.com>
2007-05-08 09:01:13 [ FULL ]
Hi,

Khaled Hassounah wrote:[...]

On this results is shown very slow performance :
Transfer rate:          1093.62 [Kbytes/sec] received

So you are hitting very low bottle neck and this is not a pound problem 
for sure.[...]

Pointless to make performance tests with ab and concurrency lower then 
~15 as it will show very bad results.
[...]
As I said you hit bottle neck and it is not pound - if you want to find 
the problem in pound try testing in environment
that will put the load  on the pound itself not app web server or DB.

But I can confirm that I notice something similar using ab for 
benchmarking. Now I'm trying to setup
good environment where I can test/benchmark pound.

Also note that you do not use -k (keep-alive) and most browsers do! :)[...]
[...]

Re: [Pound Mailing List] blocked requests with increased concurrency
Khaled Hassounah <khaled.hassounah(at)medhelp.org>
2007-05-08 09:31:30 [ FULL ]
Stefan,
As I
said you hit bottle neck and it is not pound - if you want to find the
problem in pound try testing in environment
  
that will put the load  on the pound itself not app web server or DB.
  

Not sure if you went through my test numbers. I was consistently able
to run the same test against the web server directly with no noticeable
issues. I also ran the test from the pound machine against the web
server to verify that there are no issues in the connection between the
pound box and the web server and there were no issues.

But I can confirm that I notice something similar using ab for
benchmarking. Now I'm trying to setup
  
good environment where I can test/benchmark pound.
  

would be interested to see what your results look like.

Also note that you do not use -k (keep-alive) and most browsers do! :)
  

good point. I'll re-run with that to see how it impacts the results.

Khaled
[...]
Attachments:  
text.html text/html 1608 Bytes

Re: [Pound Mailing List] blocked requests with increased concurrency
RedShift <redshift(at)pandora.be>
2007-05-08 14:43:06 [ FULL ]
Robert Segall wrote:[...][...][...]


I do this on a dual pentium 3 1 Ghz with 1 GB of RAM, running the latest 
kernel (2.6.21.1), latest glibc (2.5). The backends reside on the same 
host: the backend address is always 127.0.0.1. The server is doing atm 
almost nothing, serving request every now and then.

Maybe I should try that connect test with a standalone apache instance.

Re: [Pound Mailing List] intermittent 404s
Robert Segall <roseg(at)apsis.ch>
2007-05-29 18:29:22 [ FULL ]
On Fri, 2007-05-25 at 09:37 -0700, Mike Ellery wrote:[...]

Your belief is unfounded: Pound NEVER generates a 404 - they are
definitely coming from one of your back-ends. I suggest checking on
192.168.0.54 (if you have more than one back-end) to see why the page is
missing.[...]

Re: Pound Mailing List] intermittent 404s
Mike Ellery <mikee(at)s2technologies.com>
2007-05-30 15:25:41 [ FULL ]
Robert Segall wrote:
[...]

192.168.0.54 is the client IP.  The backends consist of three procs on 
the local machine:

ListenHTTP
   Address   0.0.0.0
   Port         3001
   LogLevel  2
   Service
     BackEnd
       Address  127.0.0.1
       Port          8009
       TimeOut        600
     End
     BackEnd
       Address  127.0.0.1
       Port          8010
       TimeOut        600
     End
     BackEnd
       Address  127.0.0.1
       Port          8011
       TimeOut        600
     End
   End
End

The request is part of a rails application, so it won't simply go 
missing - the application would have to fail to function at all for some 
period of time.

from my latest logs (at level 2 now), it's interesting that the failures 
spread across all backends:

May 29 00:10:31 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8009) 
0.025 sec
May 29 00:11:33 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8010) 
0.012 sec
May 29 00:12:57 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8011) 
0.013 sec
May 29 00:13:57 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8011) 
0.013 sec
May 29 00:14:59 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8010) 
0.013 sec
May 29 00:15:59 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8010) 
0.013 sec

..but the failures are very tightly clustered in time (in this case they 
happened for a span of 5 minutes).  And, the very same backends are 
successfully handling the same request in the same time period (so 
clearly the page is not missing):

May 29 00:11:58 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8010) 10.302 sec
May 29 00:12:46 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8011) 18.277 sec
May 29 00:12:57 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8010) 10.676 sec
May 29 00:13:51 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8009) 10.301 sec
May 29 00:15:39 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8010) 10.250 sec

MailBoxer