/ Zope / Apsis / Pound Mailing List / Archive / 2005 / 2005-07 / [patch] support for external health checks

[ << ] [ >> ]

[ detecting state of backends, priority of new ... ] [ HTTPS/SSL and Certificates / Gary Affonso ... ]

[patch] support for external health checks
Joerg Wendland <wendland(at)scan-plus.de>
2005-07-04 16:53:09 [ FULL ]
Hi,
We've implemented a simple interface for pound supporting health checks
for backends external to the pound process. We needed this because we
are load balancing tomcat servers with pound. Under certain circumstances 
tomcat accepts connections but does not answer them causing the load
balancer and its clients to stuck. Since pound justs checks for open
ports to decide whether a backend is alive we chose to build some
scripts that check for these failures and report to pound.

The interface is based on the existance of files. Therefor pound.cfg was
extended to recognize the directive

AliveDir /some/path/to/a/directory

This directory is checked every 'Alive' seconds for the existance of
files named like <backend>:<port>.dead or
<backend>:<port>.down. So if
you had for example a UrlGroup like

UrlGroup ".*"
  BackEnd 1.2.3.4,80,1
  BackEnd 1.2.3.5,80,1
EndGroup

you could simply touch /some/path/to/a/directory/1.2.3.4:80.dead to
cause pound to kill this backend. Removal of this file causes pound to
resurrect the respective backend, but still subject its own health
checking by trying to connect to the backend.

The reason we use this simple interface is that we did not want to have
pound start external scripts or applications that could pound to hang
due to whatever error. pound always checks for the two files mentioned
above (.dead and .down) so we can use .dead files for the healt checking
scripts and facilitate .down files to take backend administratively down
without having to reconfigure and restart pound.

I hope this is useful to others,
I'd even like to see this included upstream if there are no objections
(it's all GPL, eh?).

Thanks for your work on pound,
best regards,
  Jörg
[...]
Attachments:  
pound-extchecks.diff.gz application/octet-stream 2437 Bytes
signature.asc application/pgp-signature 190 Bytes

Re: [Pound Mailing List] [patch] support for external health checks
Robert Segall <roseg(at)apsis.ch>
2005-07-05 16:59:19 [ FULL ]
On Mon, 4 Jul 2005 16:53:09 +0200 Joerg Wendland
<wendland(at)scan-plus.de>
wrote:
[...]
checks[...]
circumstances [...]
was[...]
have[...]
checking[...]
down[...]

Thanks for your patch Jörg. We'll have a look and see what we can do
about them. In any case I'd like to propose a vote on this feature - for
or against, let's hear your opinions.[...]

Re: [Pound Mailing List] [patch] support for external health checks
Chris Wilson <chris(at)aidworld.org>
2005-07-05 17:10:15 [ FULL ]
Hi all,

On Tue, 2005-07-05 at 15:59, Robert Segall wrote:[...]

We are doing the same.
[...]

I would definitely like to see this kind of facility in Pound. Until a
better option is available, I would like to see this patch included.

"Better" in our case would be the ability to see and modify Pound's
opinion about the liveness its backends in real time, and add and remove
backends at will. I understand this is a lot of work, but Balance
already does this via a shared memory interface and command line tool.

Cheers, Chris.[...]

Re: [Pound Mailing List] [patch] support for external health checks
Chris Wilson <chris(at)aidworld.org>
2005-07-05 17:12:11 [ FULL ]
Hi Robert,

Your mailing list script seems to be getting stuck in a loop again. I'm
getting lots of copies of this message.

Cheers, Chris.

On Tue, 2005-07-05 at 15:59, Robert Segall wrote:[...]
[...][...]

RE: [Pound Mailing List] [patch] support for external health checks
<F.Alcala-Soler(at)iaea.org>
2005-07-05 17:13:06 [ FULL ]
I haven't tested it, but it sounds like something I'd like to use very soon. My
backend servers keep running while their databases are being updated, which
substantially decreases their performance. I'd like to have a mechanism to
signal the updates, so that the backend servers are easily taken out of the
load balancing. I could shut down the web server, but I also like this option.

A vote FOR this feature.

Thanks,

 Curro
[...]
05-07/1120488789000/1120575559000

This email message is intended only for the use of the named recipient.
Information contained in this email message and its attachments may be
privileged, confidential and protected from disclosure. If you are not the
intended recipient, please do not read, copy, use or disclose this
communication to others. Also please notify the sender by replying to this
message and then delete it from your system.

Re: [Pound Mailing List] [patch] support for external health checks
Robert Segall <roseg(at)apsis.ch>
2005-07-05 17:58:19 [ FULL ]
On Tue, 05 Jul 2005 16:12:11 +0100 Chris Wilson <chris(at)aidworld.org>
wrote:
[...]
I'm[...]

Thanks Chris. I just replaced MailHost with MaildropHost - hopefully
this will fix it.[...]

RE: [Pound Mailing List] [patch] support for external health chec ks
Victor Aker <Victor.aker(at)startec.com>
2005-07-05 19:02:23 [ FULL ]
While the idea listed below sounds good, why not just have a 1 byte file
hosted on each backend server, and have pound try to retrieve it every
'alive' seconds. Failure to retrieve this file would indicate a down or hung
server, and it would negate the need for external scripts or existance of
files.

Just my 2 cents worth.

--victor

-----Original Message-----
From: F.Alcala-Soler(at)iaea.org [mailto:F.Alcala-Soler(at)iaea.org]
Sent: Tuesday, July 05, 2005 8:13 AM
To: pound(at)apsis.ch
Subject: RE: [Pound Mailing List] [patch] support for external health
checks


I haven't tested it, but it sounds like something I'd like to use very soon.
My backend servers keep running while their databases are being updated,
which substantially decreases their performance. I'd like to have a
mechanism to signal the updates, so that the backend servers are easily
taken out of the load balancing. I could shut down the web server, but I
also like this option.

A vote FOR this feature.

Thanks,

 Curro
[...]
05-07/1120488789000/1120575559000

This email message is intended only for the use of the named recipient.
Information contained in this email message and its attachments may be
privileged, confidential and protected from disclosure. If you are not the
intended recipient, please do not read, copy, use or disclose this
communication to others. Also please notify the sender by replying to this
message and then delete it from your system.

[...]

Re: [Pound Mailing List] [patch] support for external health checks
Robert Segall <roseg(at)apsis.ch>
2005-07-06 12:20:17 [ FULL ]
On Tue, 5 Jul 2005 17:13:06 +0200 <F.Alcala-Soler(at)iaea.org> wrote:
[...]
soon. My backend servers keep running while their databases are being
updated, which substantially decreases their performance. I'd like to
have a mechanism to signal the updates, so that the backend servers are
easily taken out of the load balancing. I could shut down the web
server, but I also like this option.[...]

I should have been more clear: I meant a vote for controlling back-end
availability, as opposed to the HA_port method. What you want to do is
easily achieved with HA_port.[...]

RE: [Pound Mailing List] [patch] support for external health checks
<F.Alcala-Soler(at)iaea.org>
2005-07-06 12:48:09 [ FULL ]
Hi Robert,
[...]

That's clear, thanks. I just gave a bad example. Sorry about that.

Nonetheless, I can still see the use of such a feature, if a procedure running
outside the back-end machine needs to disable a particular back-end machine
without being able to shutdown/restart the HA_port running in the back-end
itself. Of course, I could add remote commands to such a procedure, run them
over SSH and actually bring the HA_port up or down as needed, but this feature
makes it all that much simpler and one would not have to worry about opening
extra ports or installing extra services on the back-ends for the sake of
HA_port management.

I am sorry, if I still got the point wrong.

Thanks,

 Curro

This email message is intended only for the use of the named recipient.
Information contained in this email message and its attachments may be
privileged, confidential and protected from disclosure. If you are not the
intended recipient, please do not read, copy, use or disclose this
communication to others. Also please notify the sender by replying to this
message and then delete it from your system.

Re: [Pound Mailing List] [patch] support for external health checks
Will Tatam <wtatam(at)premierit.com>
2005-07-14 13:21:20 [ FULL ]
I am for this patch

On Tue, 2005-07-05 at 16:59 +0200, Robert Segall wrote:
[...]

Will Tatam
Internal Systems Manager

Tel +44 (0) 20 7837 2690
Fax +44 (0) 20 7278 3113

JID wtatam(at)jabber.premierit.com

Premier IT http://www.premierit.com/ 

New Premier House | Southampton Row
London | WC1B 5AL
Attachments:  
text.html text/html 4577 Bytes

RE: [Pound Mailing List] [patch] support for external health chec ks
Will Tatam <wtatam(at)premierit.com>
2005-07-14 13:27:30 [ FULL ]
But surely the point of this patch is that it's to deal with the backend
failing to be running correctly, but yet still responding in some form.

For example, you might be trying to upgrade a version of PHP on a
server, but something goes horribly wrong and you just want to disable
for a short while. It seams a neater way to create a dot-down file than
to modify you pound config and restart pound.

I think it's far more common for people to be wanting to take a back
offline for a short while than people wanting to add backend servers on
the fly

On Tue, 2005-07-05 at 13:02 -0400, Victor Aker wrote:
[...]

Will Tatam
Internal Systems Manager

Tel +44 (0) 20 7837 2690
Fax +44 (0) 20 7278 3113

JID wtatam(at)jabber.premierit.com

Premier IT http://www.premierit.com/ 

New Premier House | Southampton Row
London | WC1B 5AL
Attachments:  
text.html text/html 9760 Bytes

Re: [Pound Mailing List] [patch] support for external health checks
Nicholas Avenell <nicholas(at)evolvingmedia.co.uk>
2005-07-14 13:50:22 [ FULL ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm also voting for this patch

Will Tatam wrote:[...][...]
*snip*
>>> extended to recognize the directive
>>> 
>>> AliveDir /some/path/to/a/directory
*snip*

- --
Nicholas Avenell
Technical Developer[...]

MailBoxer