Hi guys,

I'm experiencing some interesting problems with Pound.

BACKGROUND: We use Pound to load balance between multiple app  
servers.  Our desktop client application talks to these servers  
through Pound.

PROBLEM: For some longer requests that take around 5 seconds to  
execute an "Internal Server Error" is returned.  There are plenty of  
5 or 6 or 7 second requests that go through okay.

TIMEOUTS:  My timeout is the default for Pound (I have not specified  
a value, and received an error when I tried).  But I don' think this  
is a timeout problem: if it was the problem would be repeatable based  
on how long the request took to execute.  This is influenced by it,  
but it seems there are other factors.

LOGS: I've reviewed the syslog, and there are no "Internal Server  
Error" references there.  In the past when the application server did  
legitimately return 500 error's, it showed up in this log.

TEST 1: I've switched to having pound only "load balance" to one  
server, eliminating the possibility that there was any confusion  
there.  The problem persisted.

TEST 2: I started connecting the desktop client to the application  
server directly.  This removed pound from the equation, and there was  
no incidents of the problem.  Based on these two tests, I'm thinking  
there may be an issue with Pound.


I'm under a crunch to fix this quickly, so any advice or fresh looks  
on the problem will be greatly appreciated.

Thanks for your help,
Pete DeLaurentis

> PROBLEM: For some longer requests that take around 5 seconds to execute
[...]

For the sake of curiosity, what happens if you increase the timeout to 
say, 45 seconds?  It sure sounds like a timeout problem...

Regards,[...]

>> PROBLEM: For some longer requests that take around 5 seconds to 
[...][...]

Thanks David.

I just set the timeout to 45 seconds, but the problem persists.   
There doesn't seem to be a hard cut-off since some long requests  
work.  Any other ideas?

-Pete

Just one thing to confirm... I'm assuming Pound's TimeOut just  
measures latency before the server starts returning it's response.   
Once the response starts streaming, I'd assume it can't timeout  
anymore.  Does this sound right to everyone?

I also saw there was a Client Timeout, and have started playing with  
this.  I read the description in the man page (below), but still  
don't understand how this condition could happen.  HTTP is a  
stateless protocol.  Is this referring to if the client takes a long  
time to transmit the request data, or ACK back after the request is  
received?

Except from Man Page

Client value - Specify for how long Pound will wait for a client  
request (default: 10 seconds). After this long has passed without the  
client sending any data Pound will close the connection. Set it  
higher if your clients time-out on a slow network or over-loaded  
server, lower if you start getting DOS attacks or run into problems  
with IE clients. This value can be overridden for specific listeners.


Thanks for the help,
Pete



On Dec 15, 2007, at 3:43 PM, Pete DeLaurentis wrote:

>>> PROBLEM: For some longer requests that take around 5 seconds to  
>>> execute an "Internal Server Error" is returned.  There are plenty 

>>> of 5 or 6 or 7 second requests that go through okay.
>>> TIMEOUTS:  My timeout is the default for Pound (I have not  
>>> specified a value, and received an error when I tried).  But I  
>>> don' think this is a timeout problem: if it was the problem would 

>>> be repeatable based on how long the request took to execute.   
>>> This is influenced by it, but it seems there are other
factors.[...][...]

Pete DeLaurentis wrote:[...]

That's my understanding but I'll defer to Robert or others.  It may be 
an upper limit for the request - there are some other prominent pieces 
of software that do that (rsync, for one).
[...]

Its the time between the client connecting to the server and sending a 
request.  You can illustrate a contrived case by telnet'ing to your 
pound server and letting it just sit there - the server should close the 
connection after the client timeout period has passed.

Regards,[...]

I had the same problem recently where I was getting an internal server error
from pound.

I set my TimeOut on the HTTPS listener (I used it for HTTPS termination) to
60 seconds. Then I also set the timeout on connections in the application
server (tomcat) to also match 60 seconds.

The defaults for pound and tomcat are pretty short - 15 seconds and 20
seconds, respectively.

You may need to address the real cause of your timeout, though, and that is
latency in your application. Look for slow running queries in your databse
and set the number of processes in your app server to a reasonable level
that reflects your traffic load. If you set that too low, then your clients
are getting queued up and will eventually timeout.

To verify that pound is delivering the timeout message, try using a
different output message than the default message. The generic pound
internal server message can be misleading.

The TimeOut in pound is just a bandaid. Don't rely upon it as your long-term
fix.
[...]

On Sat, 2007-12-15 at 16:56 -0800, Pete DeLaurentis wrote:[...]

Not really. The timeout value shows the maximum time Pound will wait for
input (start, middle or end of the request or response). In other words,
if the first part of the response arrives and then there is a delay
longer than timeout seconds before the next part (packet) is received,
the transaction is failed. Same applies to opening a connection or
sending a request to a back-end.
[...]

Imagine a client uploads a file or does a POST request (parameters are
part of the request body). If during transmission a long pause occurs,
the transaction fails.[...]

Thanks for the explanation on these timeouts, Robert.

I think I found the source of the problem.  I had my client + server  
timeouts set to 120 (I'm assuming it's in seconds), and I setup a  
custom 500 message for Pound.  This custom message showed up,  
confirming that Pound was timing out requests after 5-6 seconds.

But why?  Well, here's my setup: Pound was sending requests to  
Lighttpd, which proxies them (with crucial X-Sendfile support) to  
several Mongrel ruby app servers.

Lighttpd was the problem.  The proxying inside Lighttpd was messing  
up and if the request took longer than 5 seconds, it would try to  
send it again to another app server.  When the cloned requests  
completed, Lighty would get confused and send two responses back to  
Pound.  I believe this was causing Pound to timeout early.

Now I think I might need a new webserver.

Thanks for the help,
Pete



On Dec 17, 2007, at 9:37 AM, Robert Segall wrote:
[...][...][...][...][...]