On Thu, 2007-05-03 at 15:30 -0400, David Barr wrote:
> On 5/3/07, David Barr wrote:
> 
> > I'm using "top" to monitor the processes on the system, and about
> > every five minutes I see "pound" jump up to the top of the process
> > list.  It will typically be using about 25% of the CPU, and it will
> > stay like that for about 30 seconds.  I'm logging all messages of
> 
> I think I've figured out the problem.  I ran gprof and dbx to help me find
it.
> 
> There's a comment in the code that says "Periodically regenerate
> ephemeral RSA keys runs every T_RSA_KEYS seconds" which corresponds to
> what I see happening.
> 
> I don't even want to use SSL encryption right now.  Maybe the best
> thing to do is set N_RSA_KEYS to something low (like 0 or 1), and set
> T_RSA_KEYS to something really big.

Try "configure --with-t_rsa=21600" and Pound will recompute the
ephemeral keys only once every 6 hours.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

On Fri, 2007-05-04 at 13:13 +0200, RedShift wrote:
> I have a monitoring service which also reports this, it polls every hour 
> to see if http is alive. This happens on any load:
> 
> Worst result recorded:
> 
>       gemiddelde reactietijd : 0,519 seconde (average response time)
>       kleinste reactietijd : 0,041 seconde (smallest response time)
>       grootste reactietijd : 14,176 seconden (greatest response time)
> 
> Last result:
> 
>       gemiddelde reactietijd : 0,187 seconde (average)
>       kleinste reactietijd : 0,038 seconde (smallest)
>       grootste reactietijd : 4,172 seconden (greatest)
> 
> (I think 4 seconds is still alot, internet users already give a weird 
> look if stuff doesn't come rolling in immediatly).
> 
> One question: does pound start forwarding the http response once the 
> full response from the backend is received (store-and-forward) or does 
> it start forwarding the response immediatly (cut-through)?

Pound does no storage - once the request headers are read in and
processed they are sent to the back-end, following which the request
body is read from the client (possibly in packet-sized pieces) and sent
on. Same mechanism for the response.

I have no idea where the delays come from - first suspect is the
threading library (bad scheduling), second is the back-end (delays in
reading the request or generating the response), third is the networking
layer in your OS (it chokes if too many connections are open
simultaneously, or too many fragments are pending) and we can't really
exclude hardware (we have seen nasty performance problems in the past
due to lousy network cards). The delays seem to occur in very few
requests (usually under 1%) and not on all systems (RedHat may be
particularly prone to this, though that could be only a coincidence).

I would certainly appreciate more data on this - for the moment I have
no solution.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

On Wed, 2007-05-02 at 13:00 -0400, Albert wrote:
> Robert,
> 
> what would we need to upgrade?  I'm guessing the thread library, what is 
> your recommendation for Redhat linux?

Older RedHat had a peculiar implementation of the threading library. I
am not familiar enough with the newer versions to answer that, but 2.6.9
is not exactly the newest kernel.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

On Thu, 2007-05-03 at 15:20 -0400, Wesley Moxam wrote:
> Hello,
> 
> I just had a hell of a time configuring mongrel to play nicely with  
> WebDAV. The README file contains the following:
> 
> --enable-msdav -- enable or disable support for the Microsoft WebDAV  
> extensions (default: disabled).
> 
> However the CHANGELOG says:
> 
> Cumulative changes since 2.1:
> ...
> - removed msdav compile-time configuration flag and MSDAV  
> configuration flag, extended xHTTP to allow for
>      WebDAV, MS-DAV and MS-RPC
> ...
> 
> Also the README says:
> 
> In addition to this flag you must specify 'WebDAV 1' in the config file.
> If you also use the SSL wrapper feature in front of a Microsoft server
> you should probably also add 'AddHeader "Front-End-Https: on"'.
> 
> These changes are also required to access a Subversion server via Pound.
> 
> (Note: the Pound homepage (http://www.apsis.ch/pound/) contains this  
> info as well).
> 
> This is incorrect. Putting 'WebDAV 1' in the config file will result  
> in Pound complaining about a configuration error. After much googling  
> I found (http://www.slashdotdash.net/articles/2007/01/23/deploying- 
> rails-with-pound-rest-headache) that the solution was to use 'xHTTP 2' .
> 
> I realize now that the man page is in fact up to date, but since I  
> consulted the Pound homepage first, I wasn't looking for the xHTTP  
> keyword and missed it.
> 
> Could the documentation be updated to save others from having to  
> search around?

Thanks for pointing that out - we'll fix it.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

Robert,

> I would certainly appreciate more data on this - for the moment I have
> no solution.

Is there any different kind of info that you think might be helpful? or 
did you mean the same kind of data from different systems to draw a pattern?

Khaled

I don't know if it helps, but when I link, here are my flags:
"gcc"  -pthread -o pound pound.o http.o config.o svc.o -lpcreposix -lssl 
-lcrypt o -lresolv -ldl   -lm -ltcmalloc

When running configure, its not able to get find libphtreads  (with an 
's' in the end) libarary.  Is it available for RedHat 2.6.9 kernel?



Khaled Hassounah wrote:
> Robert,
>
>> I would certainly appreciate more data on this - for the moment I have
>> no solution.
>
> Is there any different kind of info that you think might be helpful? 
> or did you mean the same kind of data from different systems to draw a 
> pattern?
>
> Khaled
>

My config.log says that -libpthreads was not found on RedHat 2.6.9 
kernel as well. But it resorts to using -pthread later. I believe those 
to be equivalent as -lpthread set the compiler and linkers flags to 
include pthreads.

Khaled

Albert wrote:
> I don't know if it helps, but when I link, here are my flags:
> "gcc"  -pthread -o pound pound.o http.o config.o svc.o -lpcreposix -lssl 
> -lcrypt o -lresolv -ldl   -lm -ltcmalloc
> 
> When running configure, its not able to get find libphtreads  (with an 
> 's' in the end) libarary.  Is it available for RedHat 2.6.9 kernel?

Robert Segall wrote:
> On Wed, 2007-05-02 at 13:00 -0400, Albert wrote:
>   
>> Robert,
>>
>> what would we need to upgrade?  I'm guessing the thread library, what is 
>> your recommendation for Redhat linux?
>>     
>
> Older RedHat had a peculiar implementation of the threading library. I
> am not familiar enough with the newer versions to answer that, but 2.6.9
> is not exactly the newest kernel.
>   
Red Hate have their own version numbering that is not the same as 
vanilla kernel.
So 2.6.9 actually is the latest kernel for red hat :)
2.6.9-42.0.10 is not old kernel.
I'm not sure about the threading library in this red hat, but I think it 
is merged from 2.6.18/19 vanilla kernel.

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177

Hi,

Khaled Hassounah wrote:
> Hello all,
>
> I setup a pound server in front of our .NET server (which we're
> transitioning from) and ran some tests to see whether our QPS and
> responsiveness were affected. The results showed a weird behavior at
> high concurrency.
>
> Here are the results: http://208.109.51.74/share/performance_tests.html
>   

On this results is shown very slow performance :
Transfer rate:          1093.62 [Kbytes/sec] received

So you are hitting very low bottle neck and this is not a pound problem 
for sure.
> Three test sets. One at low concurrency (3 threads), one at medium (10
> threads) and one at high (30 threads). Each test set included a run
> directly against the web server, and one through pound.
>   

Pointless to make performance tests with ab and concurrency lower then 
~15 as it will show very bad results.

> At low concurrency, requests per seconds, and time per request were very
> similar between direct and pound. When I ran with 10 threads, I noticed
> that the numbers starting differing, but that while 98% of the requests
> had a similar time per request, about 2% took more than 5 seconds!
> Things started getting worse with the higher concurrency test. At 30
> threads, At least 5% of the requests were taking longer than 5 seconds
> to complete.
>
> I am using ab for the tests, and all commands looked like: ab -c 30 -n
> 3000 http://pound/
>   
As I said you hit bottle neck and it is not pound - if you want to find 
the problem in pound try testing in environment
that will put the load  on the pound itself not app web server or DB.

But I can confirm that I notice something similar using ab for 
benchmarking. Now I'm trying to setup
good environment where I can test/benchmark pound.

Also note that you do not use -k (keep-alive) and most browsers do! :)
> It seems to me like some responses are being held hostage while others
> are being served. Is that expected? This is somewhat concerned, as we do
> see averages exceeding 20 requests for parts of the day.
>
> Any insight would be appreciated. I like pound a lot, but this is very
> concerning.
>
> Khaled
>
>   

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177

Stefan,
As I
said you hit bottle neck and it is not pound - if you want to find the
problem in pound try testing in environment
  
that will put the load  on the pound itself not app web server or DB.
  

Not sure if you went through my test numbers. I was consistently able
to run the same test against the web server directly with no noticeable
issues. I also ran the test from the pound machine against the web
server to verify that there are no issues in the connection between the
pound box and the web server and there were no issues.

But I can confirm that I notice something similar using ab for
benchmarking. Now I'm trying to setup
  
good environment where I can test/benchmark pound.
  

would be interested to see what your results look like.

Also note that you do not use -k (keep-alive) and most browsers do! :)
  

good point. I'll re-run with that to see how it impacts the results.

Khaled

-- 
Khaled Hassounah |
MedHelp

Robert Segall wrote:
> On Fri, 2007-05-04 at 13:13 +0200, RedShift wrote:
>> I have a monitoring service which also reports this, it polls every hour 
>> to see if http is alive. This happens on any load:
>>
>> Worst result recorded:
>>
>>       gemiddelde reactietijd : 0,519 seconde (average response time)
>>       kleinste reactietijd : 0,041 seconde (smallest response time)
>>       grootste reactietijd : 14,176 seconden (greatest response time)
>>
>> Last result:
>>
>>       gemiddelde reactietijd : 0,187 seconde (average)
>>       kleinste reactietijd : 0,038 seconde (smallest)
>>       grootste reactietijd : 4,172 seconden (greatest)
>>
>> (I think 4 seconds is still alot, internet users already give a weird 
>> look if stuff doesn't come rolling in immediatly).
>>
>> One question: does pound start forwarding the http response once the 
>> full response from the backend is received (store-and-forward) or does 
>> it start forwarding the response immediatly (cut-through)?
> 
> Pound does no storage - once the request headers are read in and
> processed they are sent to the back-end, following which the request
> body is read from the client (possibly in packet-sized pieces) and sent
> on. Same mechanism for the response.
> 
> I have no idea where the delays come from - first suspect is the
> threading library (bad scheduling), second is the back-end (delays in
> reading the request or generating the response), third is the networking
> layer in your OS (it chokes if too many connections are open
> simultaneously, or too many fragments are pending) and we can't really
> exclude hardware (we have seen nasty performance problems in the past
> due to lousy network cards). The delays seem to occur in very few
> requests (usually under 1%) and not on all systems (RedHat may be
> particularly prone to this, though that could be only a coincidence).
> 
> I would certainly appreciate more data on this - for the moment I have
> no solution.


I do this on a dual pentium 3 1 Ghz with 1 GB of RAM, running the latest 
kernel (2.6.21.1), latest glibc (2.5). The backends reside on the same 
host: the backend address is always 127.0.0.1. The server is doing atm 
almost nothing, serving request every now and then.

Maybe I should try that connect test with a standalone apache instance.

On Fri, 2007-05-25 at 09:37 -0700, Mike Ellery wrote:
> I'm running pound 2.3.1 on FC7.  My configuration seems to working 
> nicely except for some very intermittent 404 I'm getting from pound:
> 
> May 24 15:02:05 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 24 15:03:08 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 24 15:14:53 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 24 15:15:54 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 24 15:16:55 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 25 04:33:26 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 25 04:34:50 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 25 04:36:02 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> May 25 04:37:05 localhost pound: 192.168.0.54 POST /loaders/create 
> HTTP/1.1 - HTTP/1.1 404 Not Found
> 
>   (there are LOTS of other successful POSTs to /loaders/create 
> before/after/during the times when these 404s happen)
> 
> I believe these requests are not making it to my backend (based on logs 
> from my backend service).  Under what circumstance does pound issue a 
> 404 for a request?  Is there some way I can ask pound to put more 
> information in the log about why it returned 404? If pound thought my 
> backends were dead, it would issue a 503, right?  Any help tracking this 
> down is appreciated.

Your belief is unfounded: Pound NEVER generates a 404 - they are
definitely coming from one of your back-ends. I suggest checking on
192.168.0.54 (if you have more than one back-end) to see why the page is
missing.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

Robert Segall wrote:

> 
> Your belief is unfounded: Pound NEVER generates a 404 - they are
> definitely coming from one of your back-ends. I suggest checking on
> 192.168.0.54 (if you have more than one back-end) to see why the page is
> missing.

192.168.0.54 is the client IP.  The backends consist of three procs on 
the local machine:

ListenHTTP
   Address   0.0.0.0
   Port         3001
   LogLevel  2
   Service
     BackEnd
       Address  127.0.0.1
       Port          8009
       TimeOut        600
     End
     BackEnd
       Address  127.0.0.1
       Port          8010
       TimeOut        600
     End
     BackEnd
       Address  127.0.0.1
       Port          8011
       TimeOut        600
     End
   End
End

The request is part of a rails application, so it won't simply go 
missing - the application would have to fail to function at all for some 
period of time.

from my latest logs (at level 2 now), it's interesting that the failures 
spread across all backends:

May 29 00:10:31 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8009) 
0.025 sec
May 29 00:11:33 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8010) 
0.012 sec
May 29 00:12:57 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8011) 
0.013 sec
May 29 00:13:57 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8011) 
0.013 sec
May 29 00:14:59 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8010) 
0.013 sec
May 29 00:15:59 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 404 Not Found (s2server5:3001/- -> 127.0.0.1:8010) 
0.013 sec

..but the failures are very tightly clustered in time (in this case they 
happened for a span of 5 minutes).  And, the very same backends are 
successfully handling the same request in the same time period (so 
clearly the page is not missing):

May 29 00:11:58 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8010) 10.302 sec
May 29 00:12:46 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8011) 18.277 sec
May 29 00:12:57 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8010) 10.676 sec
May 29 00:13:51 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8009) 10.301 sec
May 29 00:15:39 localhost pound: 192.168.0.54 POST /loaders/create 
HTTP/1.1 - HTTP/1.1 200 OK (s2server5:3001/- -> 127.0.0.1:8010) 10.250 sec