Adrian Bucur napsal(a):[...]

Hello Adrian,

I seriously doubt this has anything to do with pound itself. It looks to 
me you have some issues with your network. Are the pound server and 
backends on the same network or is there any router between them? Do you 
use any dynamic routing like OSPF or RIP? It looks like your pound 
looses the route to backends for a while. Or maybe some firewall 
reinitialization?

I suggest you run ping on the pound server to the backend and look for 
anything suspicpious around the time this error occurs.
[...]

    Well I thought of that in the first place but since the backends
are in the same lan with the server then I cannot see how it can be a
routing issue. The proxy server has an internal interface with an
internal ip address and external interface on which the request are
made. It is as simple as that. The firewall is pf from opnebsd and it
does the nat and redirection perfectly. 
    I tried the ping and nothe suspicious. 
    Maybe this is a more general error and it is something from pound.









 
Adrian Bucur 
Senior System
Administrator
NOBEL Ltd.
 [...]

Adrian Bucur wrote:[...]
Can you install pftop (or just use pfctl) and see how many active state 
have in your firewall?
If pound is serving busy site, and you use keep alive (I think pf 4.1 
use them by default)
then most probably you reach the max limit of states.
I suggest increasing "set limit states" (default value is 10,000)

IF your firewall is too restrictive it can happen that you block packets 
that close connection,
and you will reach max limit states very very fast on loaded site.

Also if the connection between pound and backends is done using internal 
lan yous can
config your firewall to "set skip on $INT_IF", and keep firewalling only 
on external interface(s)[...]
[...]

Hi Stefan,

       Thanks for the advice. I set the pf state number to 256k. I had
around 6-7k everytime I checked but it didn't solve my issue. Is there
any way I can see in the logs a more detailed error?

Thank you,









 
Adrian Bucur 
Senior System
Administrator
NOBEL Ltd.
 [...]

Dean Maunder wrote:[...]

You don't need to redirect. You can include a 1x1 pixel image generated 
by a php script in the Err500 HTML. Something like:
<img src='http://failsafe.domain.com/img.php?e=It+is+broken'
/>. Or some 
fancy javascript/AJAX stuff.

But! Are you sure you really want that? I mean, if your web serves some 
interesting amount of traffic (which is likely, if you need pound), 
then, in case things go south, you'll be receiving few hundred e-mails a 
second. So you'll have broken web AND flooded e-mail server.
[...]

Why don't you just monitor the logs for errors?
There are many tools to do such a thing, monit
<www.tildeslash.com/monit/>for example.
[...]

Oops. I meant I wonder if pound has the optional ability to establish
constant connections with a backend, then reuse those (i.e. its own
keep-alive's with an http proxy--establish a single connection, client A
connects to pound, pound uses it, then after that B connects to pound, pound
reuses A's connection for B's transfer).  That might be useful. Thanks!
-Roger

On Mon, Sep 17, 2007 at 01:10:17PM -0600, Roger Pack wrote:[...]
I was also considering this feature, as it would be rather useful on
heavily loaded environments. Since the available sources ports are
very limited, both in numbers and both in time since the FINWAIT state
has to be waited even with lowered TTLs.

When i was testing pound with various floods (=heavy traffic), one of
the biggest problems was this, i always ran out of available source ports,
and because of this, pound rendered the backends DEAD for a time.


Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu
[...]

>[...]




Ahh you mean testing it on one machine used up all available TCP ports on
that machine?
  That would be problematic to testing :)

On Tue, Sep 18, 2007 at 01:22:25PM -0600, Roger Pack wrote:[...]
nope. it's problematic for the business critical application, not for
the testing. i had to highly lower tcp timeouts in the firewall to make
pound able to operate under such a traffic.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu
[...]

Wondering if you could help me understand this--so the problem is that when
you retain X open connections with a backend, there is a limit to the total
number of TCP socket numbers open for pound to connect on, leaving you
TotalAvailable - X sockets that user's can connect to? I'm having trouble
understanding why one would run out of ports, exactly...thank you


On 9/18/07, Gergely CZUCZY <phoemix(at)harmless.hu> wrote:[...]

[...]

On Wed, Sep 19, 2007 at 01:03:16PM -0600, Roger Pack wrote:[...]
Read the RFC that explains TCP, and wonder about post-connection lingering
packets and the idea if FIN_WAIT, FIN_WAIT2 states. Also take into account
the 16 bits in which the tcp and udp numbers are represented. Notice what
data pairs identifies a connection. Just read,read,read,read and read a
bit more about how tcp/ip works. That's all.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu
[...]

Ahh so the problem is that when you flood a host pound runs out of handles
and/or ports (I've seen this before with TIME_WAIT's preventing servers from
serving.
If this is the case then shouldn't this problem exist whenever you test
pound?  And actually be lessened by reusing connections from pound to its
backends?
Thank you!
-Roger

On 9/19/07, Gergely CZUCZY <phoemix(at)harmless.hu> wrote:[...]


Geniuses find out if they are geniuses by testing their answers. :)

On Mon, 2007-09-17 at 13:10 -0600, Roger Pack wrote:[...]

That's not a very good idea - you would be creating a bottle-neck for no
good reason. Basically you are serialising the requests from all clients
over a single connection (don't forget that you need to wait until the
response comes back BEFORE you can send the next request).[...]

On Thu, Sep 20, 2007 at 06:28:37PM +0200, Robert Segall wrote:[...]
I think it would be rather a feature then a bug. Though, it really
adds to the complexity factor.

An algorithm looks simple at first for this problem. Pound uses
keepalives to the backends as far as a connection is available
to recieve a new request (that means, the previous one has been
finished and the connection is still open), or if there are no
such "slots" available pound could open a new keepalive connection
to a backend.

This would reduce the availabe sourceport problem, and also
reduce the network overhead because it wouldn't bee need to
establish such many new TCP sessions. It would also be better
for the stateful firewalls, because it could be managed by
less processing power.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu
[...]

A variation on this is common used by hardware balancers.  They will reuse
any idle connection.  This means that you wind up with as many connections
as there are active worker threads on the server.  More importantly, you
don't wind up with lots of slowing expiring sockets in TIME_WAIT.


On 9/20/07 9:28 AM, "Robert Segall" <roseg(at)apsis.ch> wrote:
[...][...][...]

>[...]


True that it 'stops' all connections at one point (at pound), putting them
in a queue--this might not always be bad, though.
The instance when it would help would be if it is farming out requests to
mongrel (RoR) instances, which can only handle one request at a time,
anyway.  It might be useful for instances when the back-end is a little bit
handicap and might get tied up with one request.  It also avoids TCP slow
starts within localhost, though that doesn't take much time, and, as you
noted, would not allow back ends to do any concurrent processing on requests
(like parsing the header while also doing its previous request).

The extension of this type of paradigm is having a port to which back-ends
can arbitrarily connect to (i.e. any number), and then load balance among
them, and this allows the user to fire up more back ends and have them
connect, should the load grow (without having to restart).  I wouldn't
presume to request that but it's where some proxies are headed :)  Dunno.

I'd say it's probably an extension only useful to mongrel users.  Just
thinking out loud.
Thanks
 -Roger