Adrian Bucur napsal(a):
>   Sep 10 11:02:47 pound: backend xx.xx.xx.xx:80 connect: No route to
> host 

Hello Adrian,

I seriously doubt this has anything to do with pound itself. It looks to 
me you have some issues with your network. Are the pound server and 
backends on the same network or is there any router between them? Do you 
use any dynamic routing like OSPF or RIP? It looks like your pound 
looses the route to backends for a while. Or maybe some firewall 
reinitialization?

I suggest you run ping on the pound server to the backend and look for 
anything suspicpious around the time this error occurs.

-- 
Michal Táborský
chief systems architect
Internet Mall, a.s.
<http://www.MALL.cz>

    Well I thought of that in the first place but since the backends
are in the same lan with the server then I cannot see how it can be a
routing issue. The proxy server has an internal interface with an
internal ip address and external interface on which the request are
made. It is as simple as that. The firewall is pf from opnebsd and it
does the nat and redirection perfectly. 
    I tried the ping and nothe suspicious. 
    Maybe this is a more general error and it is something from pound.









 
Adrian Bucur 
Senior System
Administrator
NOBEL Ltd.
 
___________________
US:         
 
+1 866 776 6235 ext 6464
RO:         
 
+40 21 211 01 85 ext 6464
Fax:        
  +40 21 211 04 85
E-mail:     
  adrian.bucur(at)nobelglobe.com
IM
MSN:    
bucur_adrian_ciprian(at)hotmail.com
Web:      
  
www.nobelglobe.com
 
This
e-mail and attachments, if any, may contain confidential and/or
proprietary
information. Please be advised that the unauthorized use or disclosure
of the
information is strictly prohibited. The information herein is intended
only for
use by the intended recipient(s) named above. If you have received this
transmission in error, please notify the sender immediately and
permanently
delete the e-mail and any copies, printouts or attachments thereof.
 



Michal Taborsky - Internet Mall wrote:
Adrian Bucur napsal(a):
  
    Sep 10 11:02:47 pound: backend
xx.xx.xx.xx:80 connect: No route to
    
host 
  
Hello Adrian,
  
  
I seriously doubt this has anything to do with pound itself. It looks
to me you have some issues with your network. Are the pound server and
backends on the same network or is there any router between them? Do
you use any dynamic routing like OSPF or RIP? It looks like your pound
looses the route to backends for a while. Or maybe some firewall
reinitialization?
  
  
I suggest you run ping on the pound server to the backend and look for
anything suspicpious around the time this error occurs.

Adrian Bucur wrote:
>     Well I thought of that in the first place but since the backends
> are in the same lan with the server then I cannot see how it can be a
> routing issue. The proxy server has an internal interface with an
> internal ip address and external interface on which the request are
> made. It is as simple as that. The firewall is pf from opnebsd and it
> does the nat and redirection perfectly. 
>     I tried the ping and nothe suspicious. 
>     Maybe this is a more general error and it is something from pound.
>   
Can you install pftop (or just use pfctl) and see how many active state 
have in your firewall?
If pound is serving busy site, and you use keep alive (I think pf 4.1 
use them by default)
then most probably you reach the max limit of states.
I suggest increasing "set limit states" (default value is 10,000)

IF your firewall is too restrictive it can happen that you block packets 
that close connection,
and you will reach max limit states very very fast on loaded site.

Also if the connection between pound and backends is done using internal 
lan yous can
config your firewall to "set skip on $INT_IF", and keep firewalling only 
on external interface(s)
>
>
>
>
>
>
>
>
>  
> Adrian Bucur 
> Senior System
> Administrator
> NOBEL Ltd.
>  
> ___________________
> US:         
>  
> +1 866 776 6235 ext 6464
> RO:         
>  
> +40 21 211 01 85 ext 6464
> Fax:        
>   +40 21 211 04 85
> E-mail:     
>   adrian.bucur(at)nobelglobe.com
> IM
> MSN:    
> bucur_adrian_ciprian(at)hotmail.com
> Web:      
>   
> www.nobelglobe.com
>  
> This
> e-mail and attachments, if any, may contain confidential and/or
> proprietary
> information. Please be advised that the unauthorized use or disclosure
> of the
> information is strictly prohibited. The information herein is intended
> only for
> use by the intended recipient(s) named above. If you have received this
> transmission in error, please notify the sender immediately and
> permanently
> delete the e-mail and any copies, printouts or attachments thereof.
>  
>
>
>
> Michal Taborsky - Internet Mall wrote:
> Adrian Bucur napsal(a):
>   
>     Sep 10 11:02:47 pound: backend
> xx.xx.xx.xx:80 connect: No route to
>     
> host 
>   
> Hello Adrian,
>   
>   
> I seriously doubt this has anything to do with pound itself. It looks
> to me you have some issues with your network. Are the pound server and
> backends on the same network or is there any router between them? Do
> you use any dynamic routing like OSPF or RIP? It looks like your pound
> looses the route to backends for a while. Or maybe some firewall
> reinitialization?
>   
>   
> I suggest you run ping on the pound server to the backend and look for
> anything suspicpious around the time this error occurs.
> --
> To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
>   

-- 

Best Wishes,
Stefan Lambrev
ICQ# 24134177

Hi Stefan,

       Thanks for the advice. I set the pf state number to 256k. I had
around 6-7k everytime I checked but it didn't solve my issue. Is there
any way I can see in the logs a more detailed error?

Thank you,









 
Adrian Bucur 
Senior System
Administrator
NOBEL Ltd.
 
___________________
US:         
 
+1 866 776 6235 ext 6464
RO:         
 
+40 21 211 01 85 ext 6464
Fax:        
  +40 21 211 04 85
E-mail:     
  adrian.bucur(at)nobelglobe.com
IM
MSN:    
bucur_adrian_ciprian(at)hotmail.com
Web:      
  
www.nobelglobe.com
 
This
e-mail and attachments, if any, may contain confidential and/or
proprietary
information. Please be advised that the unauthorized use or disclosure
of the
information is strictly prohibited. The information herein is intended
only for
use by the intended recipient(s) named above. If you have received this
transmission in error, please notify the sender immediately and
permanently
delete the e-mail and any copies, printouts or attachments thereof.
 



Stefan Lambrev wrote:

  
Adrian Bucur wrote:
  
      Well I thought of that in the first place
but since the backends
    
are in the same lan with the server then I cannot see how it can be a
    
routing issue. The proxy server has an internal interface with an
    
internal ip address and external interface on which the request are
    
made. It is as simple as that. The firewall is pf from opnebsd and it
    
does the nat and redirection perfectly.     I tried the ping and nothe
suspicious.     Maybe this is a more general error and it is something
from pound.
    
  
Can you install pftop (or just use pfctl) and see how many active state
have in your firewall?
  
If pound is serving busy site, and you use keep alive (I think pf 4.1
use them by default)
  
then most probably you reach the max limit of states.
  
I suggest increasing "set limit states" (default value is 10,000)
  
  
IF your firewall is too restrictive it can happen that you block
packets that close connection,
  
and you will reach max limit states very very fast on loaded site.
  
  
Also if the connection between pound and backends is done using
internal lan yous can
  
config your firewall to "set skip on $INT_IF", and keep firewalling
only on external interface(s)
  
  
    
    
    
    
    
    
    
 
    
Adrian Bucur Senior System
    
Administrator
    
NOBEL Ltd.
    
 
    
___________________
    
US:          
    
+1 866 776 6235 ext 6464
    
RO:          
    
+40 21 211 01 85 ext 6464
    
Fax:          +40 21 211 04 85
    
E-mail:       adrian.bucur(at)nobelglobe.com
    
IM
    
MSN:    bucur_adrian_ciprian(at)hotmail.com
    
Web:        www.nobelglobe.com
    
 
    
This
    
e-mail and attachments, if any, may contain confidential and/or
    
proprietary
    
information. Please be advised that the unauthorized use or disclosure
    
of the
    
information is strictly prohibited. The information herein is intended
    
only for
    
use by the intended recipient(s) named above. If you have received this
    
transmission in error, please notify the sender immediately and
    
permanently
    
delete the e-mail and any copies, printouts or attachments thereof.
    
 
    
    
    
    
Michal Taborsky - Internet Mall wrote:
    
Adrian Bucur napsal(a):
    
      Sep 10 11:02:47 pound: backend
    
xx.xx.xx.xx:80 connect: No route to
    
    host   Hello Adrian,
    
    I seriously doubt this has anything to do with pound itself. It
looks
    
to me you have some issues with your network. Are the pound server and
    
backends on the same network or is there any router between them? Do
    
you use any dynamic routing like OSPF or RIP? It looks like your pound
    
looses the route to backends for a while. Or maybe some firewall
    
reinitialization?
    
    I suggest you run ping on the pound server to the backend and look
for
    
anything suspicpious around the time this error occurs.
    
--
    
To unsubscribe send an email with subject unsubscribe to
pound(at)apsis.ch.
    
Please contact roseg(at)apsis.ch for questions.
    
 

Dean Maunder wrote:
> I was wondering if anyone had tried to do anything dynamic with the
> Err500 etc returns.  What we would like to do is be able to send an
> email to support if one of the errors occurs, eg via a php script.  It
> seems that pound just sends back the text, so then we would have to use
> a redirect to send the user to a php page that would handle the mailing
> etc.  Is it possible to do this any other way?

You don't need to redirect. You can include a 1x1 pixel image generated 
by a php script in the Err500 HTML. Something like:
<img src='http://failsafe.domain.com/img.php?e=It+is+broken' />. Or some 
fancy javascript/AJAX stuff.

But! Are you sure you really want that? I mean, if your web serves some 
interesting amount of traffic (which is likely, if you need pound), 
then, in case things go south, you'll be receiving few hundred e-mails a 
second. So you'll have broken web AND flooded e-mail server.

-- 
Michal Táborský
chief systems architect
Internet Mall, a.s.
<http://www.MALL.cz>

Why don't you just monitor the logs for errors?
There are many tools to do such a thing, monit
<www.tildeslash.com/monit/>for example.

-- 
François Rejeté
www.macbidouille.com

On 9/14/07, Michal Taborsky - Internet Mall <michal.taborsky(at)mall.cz> wrote:
>
> Dean Maunder wrote:
> > I was wondering if anyone had tried to do anything dynamic with the
> > Err500 etc returns.  What we would like to do is be able to send an
> > email to support if one of the errors occurs, eg via a php script.  It
> > seems that pound just sends back the text, so then we would have to use
> > a redirect to send the user to a php page that would handle the mailing
> > etc.  Is it possible to do this any other way?
>
> You don't need to redirect. You can include a 1x1 pixel image generated
> by a php script in the Err500 HTML. Something like:
> <img src='http://failsafe.domain.com/img.php?e=It+is+broken' />. Or some
> fancy javascript/AJAX stuff.
>
> But! Are you sure you really want that? I mean, if your web serves some
> interesting amount of traffic (which is likely, if you need pound),
> then, in case things go south, you'll be receiving few hundred e-mails a
> second. So you'll have broken web AND flooded e-mail server.
>
> --
> Michal Táborský
> chief systems architect
> Internet Mall, a.s.
> <http://www.MALL.cz>
>
> --
> To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
>

 Oops. I meant I wonder if pound has the optional ability to establish
constant connections with a backend, then reuse those (i.e. its own
keep-alive's with an http proxy--establish a single connection, client A
connects to pound, pound uses it, then after that B connects to pound, pound
reuses A's connection for B's transfer).  That might be useful. Thanks!
-Roger

On Mon, Sep 17, 2007 at 01:10:17PM -0600, Roger Pack wrote:
>  Oops. I meant I wonder if pound has the optional ability to establish
> constant connections with a backend, then reuse those (i.e. its own
> keep-alive's with an http proxy--establish a single connection, client A
> connects to pound, pound uses it, then after that B connects to pound, pound
> reuses A's connection for B's transfer).  That might be useful. Thanks!
> -Roger
I was also considering this feature, as it would be rather useful on
heavily loaded environments. Since the available sources ports are
very limited, both in numbers and both in time since the FINWAIT state
has to be waited even with lowered TTLs.

When i was testing pound with various floods (=heavy traffic), one of
the biggest problems was this, i always ran out of available source ports,
and because of this, pound rendered the backends DEAD for a time.


Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu

-- 
Weenies test. Geniuses solve problems that arise.

>
> When i was testing pound with various floods (=heavy traffic), one of
> the biggest problems was this, i always ran out of available source ports,
> and because of this, pound rendered the backends DEAD for a time.




Ahh you mean testing it on one machine used up all available TCP ports on
that machine?
  That would be problematic to testing :)

On Tue, Sep 18, 2007 at 01:22:25PM -0600, Roger Pack wrote:
> >
> > When i was testing pound with various floods (=heavy traffic), one of
> > the biggest problems was this, i always ran out of available source ports,
> > and because of this, pound rendered the backends DEAD for a time.
> Ahh you mean testing it on one machine used up all available TCP ports on
> that machine?
>   That would be problematic to testing :)
nope. it's problematic for the business critical application, not for
the testing. i had to highly lower tcp timeouts in the firewall to make
pound able to operate under such a traffic.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu

-- 
Weenies test. Geniuses solve problems that arise.

Wondering if you could help me understand this--so the problem is that when
you retain X open connections with a backend, there is a limit to the total
number of TCP socket numbers open for pound to connect on, leaving you
TotalAvailable - X sockets that user's can connect to? I'm having trouble
understanding why one would run out of ports, exactly...thank you


On 9/18/07, Gergely CZUCZY <phoemix(at)harmless.hu> wrote:
>
> On Tue, Sep 18, 2007 at 01:22:25PM -0600, Roger Pack wrote:
> > >
> > > When i was testing pound with various floods (=heavy traffic), one of
> > > the biggest problems was this, i always ran out of available source
> ports,
> > > and because of this, pound rendered the backends DEAD for a time.
> > Ahh you mean testing it on one machine used up all available TCP ports
> on
> > that machine?
> >   That would be problematic to testing :)
> nope. it's problematic for the business critical application, not for
> the testing. i had to highly lower tcp timeouts in the firewall to make
> pound able to operate under such a traffic.
>
> Sincerely,
>
> Gergely Czuczy
> mailto: gergely.czuczy(at)harmless.hu
>
> --
> Weenies test. Geniuses solve problems that arise.
>
>
> --
> To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
>



-- 
-Roger Pack
I like belief. http://www.google.com/search?q=free+bible

On Wed, Sep 19, 2007 at 01:03:16PM -0600, Roger Pack wrote:
> Wondering if you could help me understand this--so the problem is that when
> you retain X open connections with a backend, there is a limit to the total
> number of TCP socket numbers open for pound to connect on, leaving you
> TotalAvailable - X sockets that user's can connect to? I'm having trouble
> understanding why one would run out of ports, exactly...thank you
Read the RFC that explains TCP, and wonder about post-connection lingering
packets and the idea if FIN_WAIT, FIN_WAIT2 states. Also take into account
the 16 bits in which the tcp and udp numbers are represented. Notice what
data pairs identifies a connection. Just read,read,read,read and read a
bit more about how tcp/ip works. That's all.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu

-- 
Weenies test. Geniuses solve problems that arise.

Ahh so the problem is that when you flood a host pound runs out of handles
and/or ports (I've seen this before with TIME_WAIT's preventing servers from
serving.
If this is the case then shouldn't this problem exist whenever you test
pound?  And actually be lessened by reusing connections from pound to its
backends?
Thank you!
-Roger

On 9/19/07, Gergely CZUCZY <phoemix(at)harmless.hu> wrote:
>
> On Wed, Sep 19, 2007 at 01:03:16PM -0600, Roger Pack wrote:
> > Wondering if you could help me understand this--so the problem is that
> when
> > you retain X open connections with a backend, there is a limit to the
> total
> > number of TCP socket numbers open for pound to connect on, leaving you
> > TotalAvailable - X sockets that user's can connect to? I'm having
> trouble
> > understanding why one would run out of ports, exactly...thank you
> Read the RFC that explains TCP, and wonder about post-connection lingering
> packets and the idea if FIN_WAIT, FIN_WAIT2 states. Also take into account
> the 16 bits in which the tcp and udp numbers are represented. Notice what
> data pairs identifies a connection. Just read,read,read,read and read a
> bit more about how tcp/ip works. That's all.
>
> Sincerely,
>
> Gergely Czuczy
> mailto: gergely.czuczy(at)harmless.hu
>
> --
> Weenies test. Geniuses solve problems that arise.


Geniuses find out if they are geniuses by testing their answers. :)

On Mon, 2007-09-17 at 13:10 -0600, Roger Pack wrote:
>  Oops. I meant I wonder if pound has the optional ability to establish
> constant connections with a backend, then reuse those (i.e. its own
> keep-alive's with an http proxy--establish a single connection, client A
> connects to pound, pound uses it, then after that B connects to pound, pound
> reuses A's connection for B's transfer).  That might be useful. Thanks!

That's not a very good idea - you would be creating a bottle-neck for no
good reason. Basically you are serialising the requests from all clients
over a single connection (don't forget that you need to wait until the
response comes back BEFORE you can send the next request).
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

On Thu, Sep 20, 2007 at 06:28:37PM +0200, Robert Segall wrote:
> On Mon, 2007-09-17 at 13:10 -0600, Roger Pack wrote:
> >  Oops. I meant I wonder if pound has the optional ability to establish
> > constant connections with a backend, then reuse those (i.e. its own
> > keep-alive's with an http proxy--establish a single connection, client A
> > connects to pound, pound uses it, then after that B connects to pound,
pound
> > reuses A's connection for B's transfer).  That might be useful. Thanks!
> 
> That's not a very good idea - you would be creating a bottle-neck for no
> good reason. Basically you are serialising the requests from all clients
> over a single connection (don't forget that you need to wait until the
> response comes back BEFORE you can send the next request).
I think it would be rather a feature then a bug. Though, it really
adds to the complexity factor.

An algorithm looks simple at first for this problem. Pound uses
keepalives to the backends as far as a connection is available
to recieve a new request (that means, the previous one has been
finished and the connection is still open), or if there are no
such "slots" available pound could open a new keepalive connection
to a backend.

This would reduce the availabe sourceport problem, and also
reduce the network overhead because it wouldn't bee need to
establish such many new TCP sessions. It would also be better
for the stateful firewalls, because it could be managed by
less processing power.

Sincerely,

Gergely Czuczy
mailto: gergely.czuczy(at)harmless.hu

-- 
Weenies test. Geniuses solve problems that arise.

A variation on this is common used by hardware balancers.  They will reuse
any idle connection.  This means that you wind up with as many connections
as there are active worker threads on the server.  More importantly, you
don't wind up with lots of slowing expiring sockets in TIME_WAIT.


On 9/20/07 9:28 AM, "Robert Segall" <roseg(at)apsis.ch> wrote:

> On Mon, 2007-09-17 at 13:10 -0600, Roger Pack wrote:
>>  Oops. I meant I wonder if pound has the optional ability to establish
>> constant connections with a backend, then reuse those (i.e. its own
>> keep-alive's with an http proxy--establish a single connection, client A
>> connects to pound, pound uses it, then after that B connects to pound, pound
>> reuses A's connection for B's transfer).  That might be useful. Thanks!
> 
> That's not a very good idea - you would be creating a bottle-neck for no
> good reason. Basically you are serialising the requests from all clients
> over a single connection (don't forget that you need to wait until the
> response comes back BEFORE you can send the next request).

>
>
> That's not a very good idea - you would be creating a bottle-neck for no
> good reason. Basically you are serialising the requests from all clients
> over a single connection (don't forget that you need to wait until the
> response comes back BEFORE you can send the next request).


True that it 'stops' all connections at one point (at pound), putting them
in a queue--this might not always be bad, though.
The instance when it would help would be if it is farming out requests to
mongrel (RoR) instances, which can only handle one request at a time,
anyway.  It might be useful for instances when the back-end is a little bit
handicap and might get tied up with one request.  It also avoids TCP slow
starts within localhost, though that doesn't take much time, and, as you
noted, would not allow back ends to do any concurrent processing on requests
(like parsing the header while also doing its previous request).

The extension of this type of paradigm is having a port to which back-ends
can arbitrarily connect to (i.e. any number), and then load balance among
them, and this allows the user to fire up more back ends and have them
connect, should the load grow (without having to restart).  I wouldn't
presume to request that but it's where some proxies are headed :)  Dunno.

I'd say it's probably an extension only useful to mongrel users.  Just
thinking out loud.
Thanks
 -Roger