I'd like to make a case for the addition of this functionality...

On occasion there is a need to remove one or more of our webservers from within
the balanced mix, usually for testing purposes. When we do this we have to
rewrite the pound config (no problem there!) and restart pound. Upon restart,
pound terminates all in-process page-pulls causing a moment of mayhem, which
then causes our phone bank to light-up like a Christmas tree! It would be
fantastic to be able to send a HUP signal to pound which would force it to
reload its config files without severing pre-existing connections. All new
connections would operate on the new pound ruleset. Everyone is happy. No bumps
in the road.

Have I sold you on this feature yet?




		
__________________________________ 
Discover Yahoo! 
Stay in touch with email, IM, photo sharing and more. Check it out! 
http://discover.yahoo.com/stayintouch.html

Am Dienstag, 14. Juni 2005 20:10 schrieb CG:[...]

Hi "CG",

sorry if I'm totally mistaken about what's your point, but isn't that what 
pound is designed for? that if you have several backends in your config, and 
take some of them out of service, pound would notice that and stop sending 
requests to them, thus keeping the service up and running by using only the 
remaining backends?

of course that could break active sessions if you use pounds session feature, 
but that would happen as well if you restart (or reconfigure) your pound, i 
guess.

again, sorry if i missed your point, just what came into my mind...


cheers,

sascha

Hi Sascha,

On Wed, 2005-06-15 at 10:50, Sascha Ottolski wrote:
[...]

Not exactly. I would like the same functionality as CG, both for testing
servers and to automatically repair the cluster if something is wrong
with a server. If you're testing a server, it may well still be
responding to HTTP requests, in which case Pound will assume that it's
still up and continue to send requests to it. Not good for testing!

I need to be able to add and remove hosts from the cluster whenever I
want for any reason, not just when they stop accepting connections
(which I believe is how Pound detects a dead backend). Currently we have
to run another load balancer (Balance) behind Pound to achieve this,
which means that I can't use Pound's session tracking. 

It might be worth examining how Balance does its command-line interface
using shared memory to talk to the running process and add or remove
hosts dynamically. Probably this mechanism could be ported to Pound
without too much difficulty (both are written in C) and this would
simplify our architecture and enable us to use Pound's session tracking.
[...]

When you restart pound it breaks active connections, as well as active
sessions. Breaking an active connection midway is much worse (IMHO). You
can automatically recover from a broken session on the backend (just ask
user to log in again) but a broken download is much more difficult (half
a page is not much use to anyone!)

Cheers, Chris.[...]

On Tue, 14 Jun 2005 11:10:41 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:
[...]
from within[...]
have to[...]
restart,[...]
which[...]
be[...]
it to[...]
new[...]
No bumps[...]

This subject was discussed repeatedly on this list and the simple answer
is that it doesn't work: if you run in a root jail the config file is
not accessible at all, not to mention the issue with existing sessions.[...]

On Wed, 15 Jun 2005 11:38:19 +0100 Chris Wilson <chris(at)aidworld.org>
wrote:
[...]
that what [...]
config, and [...]
sending [...]
only the [...]
testing[...]
have[...]
interface[...]
tracking.[...]
feature, [...]
pound, i [...]
You[...]
ask[...]
(half[...]

You can always use the HA feature in Pound. Use another port for
checking the alive/dead status of your back-end. Open/close this port to
control which back-ends Pound considers available.

See the High-Availability section in the man page for details.[...]

[...]

I mean this in the nicest way possible: Repeated discussion might be a sign
that this feature may be useful to a wide group of pound users, and it could be
worthwile to spend some time in sincere exploration.

I can think of two solutions for the chroot problem. Why not make the config
file a cellmate for pound to talk to? Or, Why not just run pound without chroot
if you want this feature?

A quick thought... If jailed, Apache keeps its config files in its jail. If
compromised, Apache has equal or more potential for destruction than pound. I
understand that some may see keeping the config file in the chroot jail as
being dangerous, but I feel that I can mitigate the danger by effectively
isolating the server as I would any other server running services accessable to
the outside world.

We've thought about using the HA port to "down" a balanced webserver, but it
puts the function of removal onto the webserver itself, and not on pound. It
also doesn't solve the issue of migrating to a new config file during a period
of uptime.


[...]

On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:
[...]
answer[...]
is[...]
sessions.[...]
sign[...]
could be[...]

That's the way I took it. I fully understand it is a desirable feature -
I just don't know how it can be done...
[...]
config[...]
without chroot[...]
jail. If[...]
pound. I[...]
jail as[...]
effectively[...]
accessable to[...]

By changing the config file an attacker can gain access to servers that
should be private. We feel that is a serious problem.
[...]
but it[...]
pound. It[...]
a period[...]

It doesn't have to be the server itself (even though some servers
support it). It can just as easily be a small monitor program that you
can start and stop at will. Would adding such a program to the Pound
distribution be useful?

The second issue you did not address is the problem of sessions: by
changing the config in a running program you create mayhem for the
existing sessions. How should that be dealt with?[...]

--- Robert Segall <roseg(at)apsis.ch> wrote:
[...]

If pound is running setuid/gid then one's config file can be owned by root or
by some other non-pound user, and readable to pound but not writable. If pound
is compromised, the compromised pound would only be able to read the config
file, but not change it. However, a true administrator could modify the config
and tell pound to reload.

As far as sessions go, here are some brainstorming ideas: 

Assign a checksum to each URLGroup. Compare old configfile urlgroups to the new
configfile's URLGroups, and if they match 100%, keep those sessions active. If
they don't match, let the current inquiries finish and terminate the sessions.

One may also argue that when the config file changes, sessions become
completely meaningless. This is a model used by classic Active Server Pages...
You may just want to let all the transactions-in-process finish, and start new
sessions for all new connections. 

Ideally, when pound receives a signal to reload, pound would have a split
personality for a small amount of time, letting old connections finish, and
forcing new connections to use the new ruleset. Can't this be done by starting
a new group of threads and then retiring the old threads when they become idle?
The memory structures pertaining to session information which should survive
the transition can persist.

If none of this sounds good, and we finally abandon the idea of reloading
config files, I would settle for a "graceful shutdown" option ... A "graceful
shutdown" consists of letting all in-process connections complete for a maximum
amount of time (say, 30 seconds, or even n seconds specified in the config
file... :) and deny all new incoming requests with a 503 until all transactions
complete or are finally terminated, at which time a new pound would load up.


		
__________________________________ 
Discover Yahoo! 
Stay in touch with email, IM, photo sharing and more. Check it out! 
http://discover.yahoo.com/stayintouch.html

Really, couldn't this be "approximated" by a graceful shutdown of the 
backend webservers *first*, assuming they're capable of doing so, then 
killing pound and restarting it?

CG wrote:[...][...]
>>>>This subject was discussed repeatedly on this list and the
simple[...]
>>>I can think of two solutions for the chroot problem. Why not make
the[...]
>>>
>>>A quick thought... If jailed, Apache keeps its config files in
its[...]
>>>We've thought about using the HA port to "down" a balanced
webserver,[...][...]

Hi Robert,
[...]

I think there are other ways to do that. If the attacker can either
modify the filesystem (in the jail) or inject code into Pound (e.g. via
a buffer overflow) then they don't need to modify the config file to
gain access to those servers.

Cheers, Chris.[...]

In that case the config files are the least of a sysadmin's worries.

--- Chris Wilson <chris(at)aidworld.org> wrote:
[...]



		
__________________________________ 
Discover Yahoo! 
Find restaurants, movies, travel and more fun for the weekend. Check it out! 
http://discover.yahoo.com/weekend.html

Except with a config "reload" new incoming requests would not be refused during
the "graceful shutdown" period. Incoming requests would instead be serviced by
the new config file rules.

--- Ed R Zahurak <ezahurak(at)atlanticbb.net> wrote:
[...]

[...]

I think there might still be a way to do it.  How many web servers do 
you have behind pound, in your instance?

Ed


CG wrote:[...][...]
>>>
>>>
>>>
>>>>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG
<cgg007(at)yahoo.com> wrote:
>>>>
>>>>
>>>>
>>>>>>This subject was discussed repeatedly on this list and
the simple
>>>>
>>>>answer
>>>>
>>>>
>>>>>>is that it doesn't work: if you run in a root jail the
config file
>>>>
>>>>is
>>>>
>>>>
>>>>>>not accessible at all, not to mention the issue with
existing
>>>>
>>>>sessions.
>>>>
>>>>
>>>>>I mean this in the nicest way possible: Repeated discussion
might be a
>>>>
>>>>sign
>>>>
>>>>
>>>>>that this feature may be useful to a wide group of pound
users, and it
>>>>
>>>>could be
>>>>
>>>>
>>>>>worthwile to spend some time in sincere exploration.
>>>>
>>>>That's the way I took it. I fully understand it is a desirable
feature -
>>>>I just don't know how it can be done...
>>>>
>>>>
>>>>
>>>>>I can think of two solutions for the chroot problem. Why
not make the
>>>>
>>>>config
>>>>
>>>>
>>>>>file a cellmate for pound to talk to? Or, Why not just run
pound
>>>>
>>>>without chroot
>>>>
>>>>
>>>>>if you want this feature?
>>>>>
>>>>>A quick thought... If jailed, Apache keeps its config files
in its
>>>>
>>>>jail. If
>>>>
>>>>
>>>>>compromised, Apache has equal or more potential for
destruction than
>>>>
>>>>pound. I
>>>>
>>>>
>>>>>understand that some may see keeping the config file in the
chroot
>>>>
>>>>jail as
>>>>
>>>>
>>>>>being dangerous, but I feel that I can mitigate the danger
by
>>>>
>>>>effectively
>>>>
>>>>
>>>>>isolating the server as I would any other server running
services
>>>>
>>>>accessable to
>>>>
>>>>
>>>>>the outside world.
>>>>
>>>>By changing the config file an attacker can gain access to
servers that
>>>>should be private. We feel that is a serious problem.
>>>>
>>>>
>>>>
>>>>>We've thought about using the HA port to "down" a balanced
webserver,
>>>>
>>>>but it
>>>>
>>>>
>>>>>puts the function of removal onto the webserver itself, and
not on
>>>>
>>>>pound. It
>>>>
>>>>
>>>>>also doesn't solve the issue of migrating to a new config
file during
>>>>
>>>>a period
>>>>
>>>>
>>>>>of uptime.
>>>>
>>>>It doesn't have to be the server itself (even though some
servers
>>>>support it). It can just as easily be a small monitor program
that you
>>>>can start and stop at will. Would adding such a program to the
Pound
>>>>distribution be useful?
>>>>
>>>>The second issue you did not address is the problem of
sessions: by
>>>>changing the config in a running program you create mayhem for
the
>>>>existing sessions. How should that be dealt with?
>>>
>>>
>>>If pound is running setuid/gid then one's config file can be owned
by root[...]
>>>file, but not change it. However, a true administrator could modify
the[...]
>>>
>>>As far as sessions go, here are some brainstorming ideas: 
>>>
>>>Assign a checksum to each URLGroup. Compare old configfile
urlgroups to the[...]
>>>completely meaningless. This is a model used by classic Active
Server[...]
>>>
>>>Ideally, when pound receives a signal to reload, pound would have a
split
>>>personality for a small amount of time, letting old connections
finish, and
>>>forcing new connections to use the new ruleset. Can't this be done
by[...]
>>>
>>>If none of this sounds good, and we finally abandon the idea of
reloading
>>>config files, I would settle for a "graceful shutdown" option ...
A[...]
>>>file... :) and deny all new incoming requests with a 503 until
all[...]
>>>		
>>>__________________________________ 
>>>Discover Yahoo! 
>>>Stay in touch with email, IM, photo sharing and more. Check it out!

>>>http://discover.yahoo.com/stayintouch.html
>>>[...][...]

Here's a few possible approaches

A. Re-read configuration file from one of the backend webservers
B. Start another pound instance, which listens on a backup socket
   Send HUP to initial pound instance, which forwards new sessions to
backup pound
   When existing sessions are all dead, backup pound takes over the
normal socket
 

-----Original Message-----
From: Ed R Zahurak [mailto:ezahurak(at)atlanticbb.net] 
Sent: Friday, 17 June 2005 9:18 AM
To: pound(at)apsis.ch
Subject: Re: [Pound Mailing List] Signal HUP to reload config files
without terminating connections

I think there might still be a way to do it.  How many web servers do
you have behind pound, in your instance?

Ed


CG wrote:[...]
[...][...]
[...]
>>>
>>>
>>>
>>>>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG
<cgg007(at)yahoo.com>
wrote:
>>>>
>>>>
>>>>
>>>>>>This subject was discussed repeatedly on this list and
the simple
>>>>
>>>>answer
>>>>
>>>>
>>>>>>is that it doesn't work: if you run in a root jail the
config file
>>>>
>>>>is
>>>>
>>>>
>>>>>>not accessible at all, not to mention the issue with
existing
>>>>
>>>>sessions.
>>>>
>>>>
>>>>>I mean this in the nicest way possible: Repeated discussion
might 
>>>>>be a
>>>>
>>>>sign
>>>>
>>>>
>>>>>that this feature may be useful to a wide group of pound
users, and

>>>>>it
>>>>
>>>>could be
>>>>
>>>>
>>>>>worthwile to spend some time in sincere exploration.
>>>>
>>>>That's the way I took it. I fully understand it is a desirable 
>>>>feature - I just don't know how it can be done...
>>>>
>>>>
>>>>
>>>>>I can think of two solutions for the chroot problem. Why
not make 
>>>>>the
>>>>
>>>>config
>>>>
>>>>
>>>>>file a cellmate for pound to talk to? Or, Why not just run
pound
>>>>
>>>>without chroot
>>>>
>>>>
>>>>>if you want this feature?
>>>>>
>>>>>A quick thought... If jailed, Apache keeps its config files
in its
>>>>
>>>>jail. If
>>>>
>>>>
>>>>>compromised, Apache has equal or more potential for
destruction 
>>>>>than
>>>>
>>>>pound. I
>>>>
>>>>
>>>>>understand that some may see keeping the config file in the
chroot
>>>>
>>>>jail as
>>>>
>>>>
>>>>>being dangerous, but I feel that I can mitigate the danger
by
>>>>
>>>>effectively
>>>>
>>>>
>>>>>isolating the server as I would any other server running
services
>>>>
>>>>accessable to
>>>>
>>>>
>>>>>the outside world.
>>>>
>>>>By changing the config file an attacker can gain access to
servers 
>>>>that should be private. We feel that is a serious problem.
>>>>
>>>>
>>>>
>>>>>We've thought about using the HA port to "down" a balanced 
>>>>>webserver,
>>>>
>>>>but it
>>>>
>>>>
>>>>>puts the function of removal onto the webserver itself, and
not on
>>>>
>>>>pound. It
>>>>
>>>>
>>>>>also doesn't solve the issue of migrating to a new config
file 
>>>>>during
>>>>
>>>>a period
>>>>
>>>>
>>>>>of uptime.
>>>>
>>>>It doesn't have to be the server itself (even though some
servers 
>>>>support it). It can just as easily be a small monitor program
that 
>>>>you can start and stop at will. Would adding such a program to
the 
>>>>Pound distribution be useful?
>>>>
>>>>The second issue you did not address is the problem of
sessions: by 
>>>>changing the config in a running program you create mayhem for
the 
>>>>existing sessions. How should that be dealt with?
>>>
>>>
>>>If pound is running setuid/gid then one's config file can be owned
by

>>>root[...]

>>>If[...]
>>>config file, but not change it. However, a true administrator could

>>>modify the[...]
>>>
>>>As far as sessions go, here are some brainstorming ideas: 
>>>
>>>Assign a checksum to each URLGroup. Compare old configfile
urlgroups 
>>>to the[...]
active.[...]

>>>completely meaningless. This is a model used by classic Active
Server[...]
>>>start[...]
>>>
>>>Ideally, when pound receives a signal to reload, pound would have a

>>>split personality for a small amount of time, letting old
connections

>>>finish, and forcing new connections to use the new ruleset. Can't 
>>>this be done by[...]
>>>become[...]
>>>
>>>If none of this sounds good, and we finally abandon the idea of 
>>>reloading config files, I would settle for a "graceful shutdown" 
>>>option ... A[...]

>>>a[...]
>>>config file... :) and deny all new incoming requests with a 503
until

>>>all[...]
>>>load[...]
>>>		
>>>__________________________________
>>>Discover Yahoo! 
>>>Stay in touch with email, IM, photo sharing and more. Check it out!

>>>http://discover.yahoo.com/stayintouch.html
>>>[...]
pound(at)apsis.ch.[...][...]

[...]

Approach B has some definite potential. The only issue I can see is that this
would kill all sessions which occur in SSL encrypted webspace. 

Client(SSL) --> Pound(Dying) --forwards without decryption--> Pound(Live)


Pound(Live) decodes the request and finds that it doesn't recognise the
session. 

Of couse Pound(Dying) could continue to run and service active http sessions
until they were over, assuming that the old config is still valid for those
session. That's a pretty big assumption. It might be better to let the
transactions terminate and then pass over instead of waiting for the sessions
to finish.

CG

--- "Chui G. Tey" <chui.tey(at)advdata.com.au> wrote:
[...]
=== message truncated ===



		
____________________________________________________ 
Yahoo! Sports 
Rekindle the Rivalries. Sign up for Fantasy Football 
http://football.fantasysports.yahoo.com

Hi Robert,

--On 15 June 2005 17:01 +0200 Robert Segall <roseg(at)apsis.ch> wrote:

<-- snip -->[...][...][...][...][...][...][...][...]
FWIW, I would find such a program very useful.  For example, in a Zope/Zeo 
setup, it would be handy to take the ZEO "backend" client down, update the 
code, bring it back up and test it again, without making it accessible via 
Pound. A standalone monitor program sounds like the easiest way to achieve 
this.

Many thanks,

Dominic

Am Donnerstag, 30. Juni 2005 16:41 schrieb Dominic Hiles:[...]

How about this script:

    $ cat backend_monitor.py
    #!/bin/env python
    from SocketServer import BaseRequestHandler, TCPServer
    
    class NoOpMonitor(BaseRequestHandler):
    
        def handle(self):
            pass
    
    server = TCPServer(('127.0.0.1', 55555), NoOpMonitor)
    server.serve_forever()


and a pound config like

   BackEnd 127.0.0.1,80,55555


run in background

    $ ./backend_monitor.py &
    [1] 26845


now pound can connect to the ha_port:

    $ telnet localhost 55555
    Trying 127.0.0.1...
    Connected to localhost.localdomain (127.0.0.1).
    Escape character is '^]'.
    Connection closed by foreign host.


if the monitor is gone...

    $ kill 26845


pound should assume the backends dead:

    $ telnet localhost 55555
    Trying 127.0.0.1...
    telnet: connect to address 127.0.0.1: Connection refused
    telnet: Unable to connect to remote host: Connection refused


I didn't test it with pound, and don't know how to make sure that no other 
program uses the same port while the monitor is down (maybe it would be best 
to listen on a low port to prevent this?), but it might help as a starting 
point.

I'd be very interesting if this works :-)


Cheers,

Sascha


[...]

Am Donnerstag, 30. Juni 2005 18:09 schrieb Sascha Ottolski:[...]

ups, I think it should read

    BackEnd 127.0.0.1,80,9,55555

I missed the priority.


Cheers, Sascha