/ Zope / Apsis / Pound Mailing List / Archive / 2005 / 2005-10 / When a backend server hangs up...

[ << ] [ >> ]

[ pound solution to CAN-2005-2088 - HTTP Response ... ] [ ANNOUNCE: Pound - reverse proxy and load balancer ... ]

When a backend server hangs up...
Martin PAPY <papy(at)imageforce.com>
2005-10-19 12:10:15 [ FULL ]
Hello everybody,

We are using Pound for months now to load balance the traffic between 
two back end servers, and we have always the same problem and I can't 
find a good solution.

If one of the backend server (running Apache / Coldfusion ) hangs up (ie 
: Apache take the connection, but the Application server hangs) Pound 
don't detect that there is a problem.

Is there a way to configure Pound to consider a backend server as "dead" 
if the connection reaches a time out ? Or maybe test the result of a 
specific URL avery "Alive" seconds ?.

I saw the ha_port diective but it is useless in our case...

Best regards,
[...]
Attachments:  
text.html text/html 1439 Bytes

Re: [Pound Mailing List] When a backend server hangs up...
Robert Segall <roseg(at)apsis.ch>
2005-10-19 13:21:37 [ FULL ]
On Wed, 2005-10-19 at 12:10 +0200, Martin PAPY wrote:[...]

No, there isn't. A timeout is not indicative of a "dead" back-end (it
could just be heavily loaded). For URL testing see below.
[...]

That's a puzzling statement. HAport gives you a completely generic
method for checking on back-end health.

There is no requirement that the HAport be opened by the web server. Any
socket will be fine, opened by whatever program you want.

In your case what you want is a small program that opens a receiving
socket and periodically accepts connections on it (this will be defined
as the HAport). In the meantime it also checks that your back-end is
healthy (whatever that may mean - you are free to make it as simple or
as fancy as you wish), and closes the socket if it gets the "wrong"
response, or no response, or anything you define. The socket is
re-opened once the back-end is healthy again.

Given the countless possibilities involved I think this is a rather
flexible approach, allowing you to tailor your health checks to your
needs. Putting such checks in Pound itself would create needless
complexity.[...]

Re: [Pound Mailing List] When a backend server hangs up...
Martin PAPY <papy(at)imageforce.com>
2005-10-19 13:58:51 [ FULL ]
Hello Robert,

Thank you for this answer. Indeed, I didn't realize that we could use 
the /haport/ this way :)

Is there any existing program or scripts exemple somewhere that we could 
use for our purpose ?

In the team we only have web developpers, so I think nobody is capable 
of doing this...

Best regards,

Martin

Robert Segall a écrit :
[...][...]
Attachments:  
text.html text/html 1805 Bytes

Re: [Pound Mailing List] When a backend server hangs up...
Guillaume VIEL <guillaume.viel(at)openfarm.fr>
2005-10-19 15:08:51 [ FULL ]
Martin PAPY wrote:
[...]

You can use Mon to monitor the application server failure. Mon checks 
the app server and in case of failure (time out) Mon triggers the 
shutdown of the apache server which will lead pound to consider the 
backend server as "dead".
Any comments?
[...]

[...]
Attachments:  
text.html text/html 2615 Bytes

Re: [Pound Mailing List] When a backend server hangs up...
Robert Segall <roseg(at)apsis.ch>
2005-10-19 18:24:22 [ FULL ]
On Wed, 2005-10-19 at 13:58 +0200, Martin PAPY wrote:[...]

Have a look at

http://www.apsis.ch/pound/pound_list/archive/2005/2005-06/1118772641000/index_html?fullMode=1#1120147797000

for an example that was posted here, or google.
[...]

So I guess you'll need to get somebody who can. You'll need someone in
any case to deal with all the infrastructure issues.[...]

Re: [Pound Mailing List] When a backend server hangs up...
Sascha Ottolski <sascha.ottolski(at)gallileus.de>
2005-10-20 12:47:24 [ FULL ]
Am Mittwoch, 19. Oktober 2005 13:58 schrieb Martin PAPY:[...]
Hi Martin,

I just hacked a little script for that purpose, extending the idea I posted 
some weeks ago. It's not thouroghly tested, and not very robust in some 
aspects (it doesn't prevent you from starting multiple instances, for 
example, and the socket should be secured by a chmod), but should be good 
enough as a starting point.

Obviously, you would have to adjust the addr and socketname attributes. In 
most real life cases, you'd use the servers real address instead of 127.0.0.1 
if you want an external pound to be able to connect.

example:

in one console:

	$ python zope_monitor.py
	monitor started

and in another:

	$ python
	Python 2.4.1 (#1, May 16 2005, 15:15:14)
	[GCC 4.0.0 20050512 (Red Hat 4.0.0-5)] on linux2
	Type "help", "copyright", "credits" or "license" for more information.
	>>> from zope_monitor import *
	>>> check_monitor()
	>>> open_monitor()
	>>> close_monitor()
	>>> check_monitor()
	Traceback (most recent call last):
	  File "<stdin>", line 1, in ?
	  File "zope_monitor.py", line 78, in check_monitor
	    s.connect(MonitorServer.addr)
	  File "<string>", line 1, in connect
	socket.error: (111, 'Connection refused')
	>>> open_monitor()
	>>> check_monitor()
	>>>

as you can see, if check_console() returns without an error, the monitor is 
up; otherwise an exception is raised. not very fancy, but should be goog 
enough for basic testing.

in the first console, you should see something like:

	$ python zope_monitor.py
	monitor started
	keeping status: open
	monitor stopped
	monitor started


Hope it helps. If anyone has ideas for improvements, please let me know. Oh, 
may be I should mention that I didn't manage to test script against a pound 
server, but I can't imagine why it shouldn't work as expected. Don't forget 
to configure your pound accordingly:

	BackEnd 127.0.0.1,80,9,55555


Cheers, Sascha
[...]
Attachments:  
zope_monitor.py text/x-java 2556 Bytes

Re: [Pound Mailing List] When a backend server hangs up...
Sascha Ottolski <sascha.ottolski(at)gallileus.de>
2005-10-20 13:20:40 [ FULL ]
Am Donnerstag, 20. Oktober 2005 12:47 schrieb Sascha Ottolski:[...]

just to confirm: it works :-)

Cheers, Sascha
[...]

Re: [Pound Mailing List] When a backend server hangs up...
Martin PAPY <papy(at)imageforce.com>
2005-10-20 14:01:30 [ FULL ]
Hello everybody !

Hum did I forget to tell you that the OS is Windows 2000 ? (Yes I know 
it sucks... but my boss was working before at Microsoft France :'( ... 
totaly brain washed ! )

Anyway, I will try to develop with a friend a little prog in Java for 
checking URL because my boss don't want to see anything else than Java 
or .Net on the web servers... (not event php ! )

I'll send the prog when it's finished if someone is interessed in.

Thanks to all for your help !

Martin

Sascha Ottolski a écrit :
[...][...][...]
Attachments:  
text.html text/html 1365 Bytes

MailBoxer