/ Zope / Apsis / Pound Mailing List / Archive / 2007 / 2007-03 / Expectations, performance, and pound.

[ << ] [ >> ]

[ Failover with sessions? / "Charlie ... ] [ Log disabling with latest Pound / "John ... ]

Expectations, performance, and pound.
"Daniel M. Lipton" <photo(at)sequential.org>
2007-03-10 18:22:49 [ FULL ]
I have pound sitting in front of a couple of web servers.  To date, I've 
had very little actual traffic through the set up, the product is in 
private beta, but I've managed to run hundreds of thousands of load 
testing requests through ApacheBench.  What I've discovered has puzzled me.

With 10 concurrent requests per second, the load on the machine pound 
runs on is at or near 0.00.  The back end application servers gets load 
equivalent to 0.25 to 1.00.  Operating under the assumption that the 
machines should produce equivalent numbers of requests per second, I was 
puzzled to see one machine handling 20 requests per second and the other 
handling 30 requests per second.  (As an aside, I discovered that the 
machines are, in fact, not identical.)  Even more puzzling were the 
results run through pound: 22 - 25 requests per second.  Furthermore, 
most of the load was being distributed to the faster machine.

I upgraded to pound 2.2.6, which immediately yielded better benchmarks 
through pound, on the order of 30 requests per second and the load 
distributed more evenly.  I switched hardware for the next set of tests 
to two machines with identical benchmarks from ab, and moved my 
benchmark tests to a remote location.  Both servers handled a sustained 
30 requests per second, but I only saw a total throughput increase to 
about 40 requests per second through pound.  Puzzled, I ran a benchmark 
with 100 concurrent requests per second and, just as I had expected all 
along, the benchmarks approached 60 requests per second.  Is this the 
expected behavior of pound at low and moderate numbers of requests per 
second?

In the process of debugging this, I noticed something else: pound 2.2.6, 
over the course of hours of continuous uptime and tens of thousands of 
requests, significantly favors one server.  In this case, pound is 
compiled with dynamic scaling enabled.  I took some time away from the 
computer and came back and checked the sites.  Interestingly, all 
requests were hitting pound and the same back end server.  So I whipped 
up a little test script to see how long it would take for me to hit the 
second back end server.  The results varied considerably, but the range 
was on the order of once every 50 requests to once every 900 requests.  
Out of curiousity, I restarted pound and now the range is between 1 and 
6 requests.  I'll continue to monitor this to see if it crops up again, 
but it seems to me that at least with dynamic scaling on, pound still 
develops a favorite back end.  If I notice the behavior again, I'll use 
the binary with dynamic scaling disabled.  One other symptom of the 
problem is that pound seems to stop responding to /etc/init.d/pound 
commands.  Start, stop, and force-reload all produce nothing to either 
STDOUT or /var/log/syslog.  To restart, I am forced to send a KILL 
signal to the processes.

Lastly, and please excuse me if this is answered clearly elsewhere, how 
do I turn session handling off?  I've hacked something together that 
seems to work, but it's not clear to me from the documentation how it 
should be turned off.  Our application does not tie sessions to servers, 
so I'd prefer load to always been distributed evenly without looking at 
session data.

Thanks.

--
Daniel M. Lipton

Re: [Pound Mailing List] Expectations, performance, and pound.
Kenneth Burgener <kenneth(at)mail1.ttak.org>
2007-03-11 05:30:14 [ FULL ]
Daniel M. Lipton wrote:[...]


I have noticed this exact behavior as well.

I eventually just disabled dynamic scaling, and I got even distribution.
 I like the idea of dynamic scaling, but when distribution becomes so
overwhelmingly unbalanced, it because ineffective.


[...]


Session handling is only enabled if you have a SESSION section.  If you
remove the SESSION section, sessions will no longer be tracked.


Kenneth

Re: [Pound Mailing List] Expectations, performance, and pound.
Albert <pound(at)alacra.com>
2007-03-11 06:05:34 [ FULL ]
[...]
I concur, I see similar behavior.  Whats more, this sort of behavior 
could be blamed on a slow backend, but I had stats thats show 
otherwise.  I have 2 servers running pound, and in one case pound like a 
particular server, and in another it absolutely "hates" it.  I'll share 
the stats later in the week to support my case.[...]
I think dynamic scaling should be a config option.  Whats more, I'd like 
to be able to have the flag on the Service level (just like LogLevel).

MailBoxer