I will be putting pound through some serious hurt on Friday, Saturday 
and Sunday.

We have live coverage of three international cricket matches, and if 
Wednesday is anything to go by, I can expect our image caches to 
generate about 1500 hits EVERY second between 9.30 and 16.30 GMT. For 
tomorrow, I'm proposing to put a single pound (running on an otherwise 
idle quad 2.8GB Xeon) in front of them.

This is basically stress-testing to see if pound will do what we need 
it to do, before we deploy it over the next couple of months all over 
our cluster. We're big fans of it for its simplicity and usefulness.

Is there anything the developers would like measured, monitored or 
otherwise inspected while this is happening, or any patches they want 
testing? Currently we're using 1.7 (from Debian testing) but I can 
compile up pound-current if you like.
-- 
Mike Whitaker      | Phone: +44 1733 327544
System Architect   | Mobile: +44 7971 977375
WisdenCricinfo Ltd | Email: mike(at)wisdengroup.com

On 2 Sep 2004, at 17:09, Mike Whitaker wrote:

> I will be putting pound through some serious hurt on Friday, Saturday 
> and Sunday.
>
> We have live coverage of three international cricket matches, and if 
> Wednesday is anything to go by, I can expect our image caches to 
> generate about 1500 hits EVERY second between 9.30 and 16.30 GMT. For 
> tomorrow, I'm proposing to put a single pound (running on an otherwise 
> idle quad 2.8GB Xeon) in front of them.

Up at 300+ hits/s, 2000+ threads, load average below 1, 20% cpu.

Had to restart because it ran out of open files (ulimit -n was down at 
1024, which was STUPID!)

Should this... (after a restart)

Sep  3 09:21:48 new-master pound: starting...
Sep  3 09:21:51 new-master pound: MONITOR: worker exited on signal 11, 
restarting...

or this...

Sep  3 09:22:41 new-master pound: copy_bin error writing: Broken pipe
Sep  3 09:22:41 new-master pound: error copy server cont: Broken pipe

concern me?

Appear to be running into another problem:

Running just one backend, which is QUITE capable of coping on its own, 
without the pound, at 1400+ hits a second, pound keeps losing its 
backend connection.

With a bit of juggling, I've started up a second backend for pound to 
share between. I'm now seeing a LOT of these:

Sep  3 10:54:38 new-master kernel: printk: 7457 messages suppressed.

which concern me a little.

Currently up around 2500 threads, splitting 1500+ hits a sec.

And it's just lost BOTH backends. *hrm*

On 3 Sep 2004, at 12:01, Mike Whitaker wrote:
>  I'm now seeing a LOT of these:
>
> Sep  3 10:54:38 new-master kernel: printk: 7457 messages suppressed.
>
> which concern me a little.

These may be a red herring. But the fact remains that I'm losing the 
backend regularly at this level of traffic, with which it is able to 
cope on its own.

I've had to turn pound off and let the squid do the work for now, which 
is a shame.

On Friday 03 September 2004 13.10, Mike Whitaker wrote:
> On 3 Sep 2004, at 12:01, Mike Whitaker wrote:
> >  I'm now seeing a LOT of these:
> >
> > Sep  3 10:54:38 new-master kernel: printk: 7457 messages suppressed.
> >
> > which concern me a little.
>
> These may be a red herring. But the fact remains that I'm losing the
> backend regularly at this level of traffic, with which it is able to
> cope on its own.
>
> I've had to turn pound off and let the squid do the work for now, which
> is a shame.

Mike please use -current, 1.7 had several known problems.

For your level of load you may want to look at some custom system settings to 
ensure you do not run out of sockets (a lot of clients do not close 
connections properly, causing the FIN_WAIT/2 problem).

Don't worry about the "broken pipe" messages, they just indicate a prematurely 
closed connection.

I have no idea where the "printk" comes from - this is very much a kernel 
issue, I doubt it has anything to do with Pound.

Good work and keep us posted.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904

Hello!

Am Freitag, 3. September 2004 13:01 schrieb Mike Whitaker:
> Sep  3 10:54:38 new-master kernel: printk: 7457 messages suppressed.
>
> which concern me a little.

This is a useful feature of the newer kernels. You can configure its behaviour 
with sysctl.

printk_ratelimit:

Some warning messages are rate limited. printk_ratelimit specifies
the minimum length of time between these messages (in jiffies), by
default we allow one every 5 seconds.
A value of 0 will disable rate limiting.

printk_ratelimit_burst:

While long term we enforce one message per printk_ratelimit
seconds, we do allow a burst of messages to pass through.
printk_ratelimit_burst specifies the number of messages we can
send before ratelimiting kicks in.


Andi
-- 
-> Andreas Roedl            -> Senior IT Manager
-> NATIVE INSTRUMENTS GmbH  -> andreas.roedl(at)native-instruments.de
-> Schlesische Strasse 28   -> http://www.native-instruments.de/
-> D-10997 Berlin           -> Tel. +49-30-61 10 35-430
-> Germany                  -> Fax  +49-30-61 10 35-35