I'd like to make a case for the addition of this functionality...

On occasion there is a need to remove one or more of our webservers from within
the balanced mix, usually for testing purposes. When we do this we have to
rewrite the pound config (no problem there!) and restart pound. Upon restart,
pound terminates all in-process page-pulls causing a moment of mayhem, which
then causes our phone bank to light-up like a Christmas tree! It would be
fantastic to be able to send a HUP signal to pound which would force it to
reload its config files without severing pre-existing connections. All new
connections would operate on the new pound ruleset. Everyone is happy. No bumps
in the road.

Have I sold you on this feature yet?




		
__________________________________ 
Discover Yahoo! 
Stay in touch with email, IM, photo sharing and more. Check it out! 
http://discover.yahoo.com/stayintouch.html

Am Dienstag, 14. Juni 2005 20:10 schrieb CG:
> I'd like to make a case for the addition of this functionality...
>
> On occasion there is a need to remove one or more of our webservers from
> within the balanced mix, usually for testing purposes. When we do this we
> have to rewrite the pound config (no problem there!) and restart pound.
> Upon restart, pound terminates all in-process page-pulls causing a moment
> of mayhem, which then causes our phone bank to light-up like a Christmas
> tree! It would be fantastic to be able to send a HUP signal to pound which
> would force it to reload its config files without severing pre-existing
> connections. All new connections would operate on the new pound ruleset.
> Everyone is happy. No bumps in the road.
>
> Have I sold you on this feature yet?

Hi "CG",

sorry if I'm totally mistaken about what's your point, but isn't that what 
pound is designed for? that if you have several backends in your config, and 
take some of them out of service, pound would notice that and stop sending 
requests to them, thus keeping the service up and running by using only the 
remaining backends?

of course that could break active sessions if you use pounds session feature, 
but that would happen as well if you restart (or reconfigure) your pound, i 
guess.

again, sorry if i missed your point, just what came into my mind...


cheers,

sascha

Hi Sascha,

On Wed, 2005-06-15 at 10:50, Sascha Ottolski wrote:

> sorry if I'm totally mistaken about what's your point, but isn't that what 
> pound is designed for? that if you have several backends in your config, and 
> take some of them out of service, pound would notice that and stop sending 
> requests to them, thus keeping the service up and running by using only the 
> remaining backends?

Not exactly. I would like the same functionality as CG, both for testing
servers and to automatically repair the cluster if something is wrong
with a server. If you're testing a server, it may well still be
responding to HTTP requests, in which case Pound will assume that it's
still up and continue to send requests to it. Not good for testing!

I need to be able to add and remove hosts from the cluster whenever I
want for any reason, not just when they stop accepting connections
(which I believe is how Pound detects a dead backend). Currently we have
to run another load balancer (Balance) behind Pound to achieve this,
which means that I can't use Pound's session tracking. 

It might be worth examining how Balance does its command-line interface
using shared memory to talk to the running process and add or remove
hosts dynamically. Probably this mechanism could be ported to Pound
without too much difficulty (both are written in C) and this would
simplify our architecture and enable us to use Pound's session tracking.

> of course that could break active sessions if you use pounds session feature,

> but that would happen as well if you restart (or reconfigure) your pound, i 
> guess.

When you restart pound it breaks active connections, as well as active
sessions. Breaking an active connection midway is much worse (IMHO). You
can automatically recover from a broken session on the backend (just ask
user to log in again) but a broken download is much more difficult (half
a page is not much use to anyone!)

Cheers, Chris.
-- 
(aidworld) chris wilson | chief engineer (chris(at)aidworld.org)

On Tue, 14 Jun 2005 11:10:41 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:

> I'd like to make a case for the addition of this functionality...
> 
> On occasion there is a need to remove one or more of our webservers
from within
> the balanced mix, usually for testing purposes. When we do this we
have to
> rewrite the pound config (no problem there!) and restart pound. Upon
restart,
> pound terminates all in-process page-pulls causing a moment of mayhem,
which
> then causes our phone bank to light-up like a Christmas tree! It would
be
> fantastic to be able to send a HUP signal to pound which would force
it to
> reload its config files without severing pre-existing connections. All
new
> connections would operate on the new pound ruleset. Everyone is happy.
No bumps
> in the road.
> 
> Have I sold you on this feature yet?

This subject was discussed repeatedly on this list and the simple answer
is that it doesn't work: if you run in a root jail the config file is
not accessible at all, not to mention the issue with existing sessions.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

On Wed, 15 Jun 2005 11:38:19 +0100 Chris Wilson <chris(at)aidworld.org>
wrote:

> Hi Sascha,
> 
> On Wed, 2005-06-15 at 10:50, Sascha Ottolski wrote:
> 
> > sorry if I'm totally mistaken about what's your point, but isn't
that what 
> > pound is designed for? that if you have several backends in your
config, and 
> > take some of them out of service, pound would notice that and stop
sending 
> > requests to them, thus keeping the service up and running by using
only the 
> > remaining backends?
> 
> Not exactly. I would like the same functionality as CG, both for
testing
> servers and to automatically repair the cluster if something is wrong
> with a server. If you're testing a server, it may well still be
> responding to HTTP requests, in which case Pound will assume that it's
> still up and continue to send requests to it. Not good for testing!
> 
> I need to be able to add and remove hosts from the cluster whenever I
> want for any reason, not just when they stop accepting connections
> (which I believe is how Pound detects a dead backend). Currently we
have
> to run another load balancer (Balance) behind Pound to achieve this,
> which means that I can't use Pound's session tracking. 
> 
> It might be worth examining how Balance does its command-line
interface
> using shared memory to talk to the running process and add or remove
> hosts dynamically. Probably this mechanism could be ported to Pound
> without too much difficulty (both are written in C) and this would
> simplify our architecture and enable us to use Pound's session
tracking.
> 
> > of course that could break active sessions if you use pounds session
feature, 
> > but that would happen as well if you restart (or reconfigure) your
pound, i 
> > guess.
> 
> When you restart pound it breaks active connections, as well as active
> sessions. Breaking an active connection midway is much worse (IMHO).
You
> can automatically recover from a broken session on the backend (just
ask
> user to log in again) but a broken download is much more difficult
(half
> a page is not much use to anyone!)
> 
> Cheers, Chris.

You can always use the HA feature in Pound. Use another port for
checking the alive/dead status of your back-end. Open/close this port to
control which back-ends Pound considers available.

See the High-Availability section in the man page for details.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

> This subject was discussed repeatedly on this list and the simple answer
> is that it doesn't work: if you run in a root jail the config file is
> not accessible at all, not to mention the issue with existing sessions.

I mean this in the nicest way possible: Repeated discussion might be a sign
that this feature may be useful to a wide group of pound users, and it could be
worthwile to spend some time in sincere exploration.

I can think of two solutions for the chroot problem. Why not make the config
file a cellmate for pound to talk to? Or, Why not just run pound without chroot
if you want this feature?

A quick thought... If jailed, Apache keeps its config files in its jail. If
compromised, Apache has equal or more potential for destruction than pound. I
understand that some may see keeping the config file in the chroot jail as
being dangerous, but I feel that I can mitigate the danger by effectively
isolating the server as I would any other server running services accessable to
the outside world.

We've thought about using the HA port to "down" a balanced webserver, but it
puts the function of removal onto the webserver itself, and not on pound. It
also doesn't solve the issue of migrating to a new config file during a period
of uptime.



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:

> 
> > This subject was discussed repeatedly on this list and the simple
answer
> > is that it doesn't work: if you run in a root jail the config file
is
> > not accessible at all, not to mention the issue with existing
sessions.
> 
> I mean this in the nicest way possible: Repeated discussion might be a
sign
> that this feature may be useful to a wide group of pound users, and it
could be
> worthwile to spend some time in sincere exploration.

That's the way I took it. I fully understand it is a desirable feature -
I just don't know how it can be done...

> I can think of two solutions for the chroot problem. Why not make the
config
> file a cellmate for pound to talk to? Or, Why not just run pound
without chroot
> if you want this feature?
> 
> A quick thought... If jailed, Apache keeps its config files in its
jail. If
> compromised, Apache has equal or more potential for destruction than
pound. I
> understand that some may see keeping the config file in the chroot
jail as
> being dangerous, but I feel that I can mitigate the danger by
effectively
> isolating the server as I would any other server running services
accessable to
> the outside world.

By changing the config file an attacker can gain access to servers that
should be private. We feel that is a serious problem.

> We've thought about using the HA port to "down" a balanced webserver,
but it
> puts the function of removal onto the webserver itself, and not on
pound. It
> also doesn't solve the issue of migrating to a new config file during
a period
> of uptime.

It doesn't have to be the server itself (even though some servers
support it). It can just as easily be a small monitor program that you
can start and stop at will. Would adding such a program to the Pound
distribution be useful?

The second issue you did not address is the problem of sessions: by
changing the config in a running program you create mayhem for the
existing sessions. How should that be dealt with?
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

--- Robert Segall <roseg(at)apsis.ch> wrote:

> On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:
> 
> > 
> > > This subject was discussed repeatedly on this list and the simple
> answer
> > > is that it doesn't work: if you run in a root jail the config file
> is
> > > not accessible at all, not to mention the issue with existing
> sessions.
> > 
> > I mean this in the nicest way possible: Repeated discussion might be a
> sign
> > that this feature may be useful to a wide group of pound users, and it
> could be
> > worthwile to spend some time in sincere exploration.
> 
> That's the way I took it. I fully understand it is a desirable feature -
> I just don't know how it can be done...
> 
> > I can think of two solutions for the chroot problem. Why not make the
> config
> > file a cellmate for pound to talk to? Or, Why not just run pound
> without chroot
> > if you want this feature?
> > 
> > A quick thought... If jailed, Apache keeps its config files in its
> jail. If
> > compromised, Apache has equal or more potential for destruction than
> pound. I
> > understand that some may see keeping the config file in the chroot
> jail as
> > being dangerous, but I feel that I can mitigate the danger by
> effectively
> > isolating the server as I would any other server running services
> accessable to
> > the outside world.
> 
> By changing the config file an attacker can gain access to servers that
> should be private. We feel that is a serious problem.
> 
> > We've thought about using the HA port to "down" a balanced webserver,
> but it
> > puts the function of removal onto the webserver itself, and not on
> pound. It
> > also doesn't solve the issue of migrating to a new config file during
> a period
> > of uptime.
> 
> It doesn't have to be the server itself (even though some servers
> support it). It can just as easily be a small monitor program that you
> can start and stop at will. Would adding such a program to the Pound
> distribution be useful?
> 
> The second issue you did not address is the problem of sessions: by
> changing the config in a running program you create mayhem for the
> existing sessions. How should that be dealt with?

If pound is running setuid/gid then one's config file can be owned by root or
by some other non-pound user, and readable to pound but not writable. If pound
is compromised, the compromised pound would only be able to read the config
file, but not change it. However, a true administrator could modify the config
and tell pound to reload.

As far as sessions go, here are some brainstorming ideas: 

Assign a checksum to each URLGroup. Compare old configfile urlgroups to the new
configfile's URLGroups, and if they match 100%, keep those sessions active. If
they don't match, let the current inquiries finish and terminate the sessions.

One may also argue that when the config file changes, sessions become
completely meaningless. This is a model used by classic Active Server Pages...
You may just want to let all the transactions-in-process finish, and start new
sessions for all new connections. 

Ideally, when pound receives a signal to reload, pound would have a split
personality for a small amount of time, letting old connections finish, and
forcing new connections to use the new ruleset. Can't this be done by starting
a new group of threads and then retiring the old threads when they become idle?
The memory structures pertaining to session information which should survive
the transition can persist.

If none of this sounds good, and we finally abandon the idea of reloading
config files, I would settle for a "graceful shutdown" option ... A "graceful
shutdown" consists of letting all in-process connections complete for a maximum
amount of time (say, 30 seconds, or even n seconds specified in the config
file... :) and deny all new incoming requests with a 503 until all transactions
complete or are finally terminated, at which time a new pound would load up.


		
__________________________________ 
Discover Yahoo! 
Stay in touch with email, IM, photo sharing and more. Check it out! 
http://discover.yahoo.com/stayintouch.html

Really, couldn't this be "approximated" by a graceful shutdown of the 
backend webservers *first*, assuming they're capable of doing so, then 
killing pound and restarting it?

CG wrote:
> 
> --- Robert Segall <roseg(at)apsis.ch> wrote:
> 
> 
>>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:
>>
>>
>>>>This subject was discussed repeatedly on this list and the simple
>>
>>answer
>>
>>>>is that it doesn't work: if you run in a root jail the config file
>>
>>is
>>
>>>>not accessible at all, not to mention the issue with existing
>>
>>sessions.
>>
>>>I mean this in the nicest way possible: Repeated discussion might be a
>>
>>sign
>>
>>>that this feature may be useful to a wide group of pound users, and it
>>
>>could be
>>
>>>worthwile to spend some time in sincere exploration.
>>
>>That's the way I took it. I fully understand it is a desirable feature -
>>I just don't know how it can be done...
>>
>>
>>>I can think of two solutions for the chroot problem. Why not make the
>>
>>config
>>
>>>file a cellmate for pound to talk to? Or, Why not just run pound
>>
>>without chroot
>>
>>>if you want this feature?
>>>
>>>A quick thought... If jailed, Apache keeps its config files in its
>>
>>jail. If
>>
>>>compromised, Apache has equal or more potential for destruction than
>>
>>pound. I
>>
>>>understand that some may see keeping the config file in the chroot
>>
>>jail as
>>
>>>being dangerous, but I feel that I can mitigate the danger by
>>
>>effectively
>>
>>>isolating the server as I would any other server running services
>>
>>accessable to
>>
>>>the outside world.
>>
>>By changing the config file an attacker can gain access to servers that
>>should be private. We feel that is a serious problem.
>>
>>
>>>We've thought about using the HA port to "down" a balanced webserver,
>>
>>but it
>>
>>>puts the function of removal onto the webserver itself, and not on
>>
>>pound. It
>>
>>>also doesn't solve the issue of migrating to a new config file during
>>
>>a period
>>
>>>of uptime.
>>
>>It doesn't have to be the server itself (even though some servers
>>support it). It can just as easily be a small monitor program that you
>>can start and stop at will. Would adding such a program to the Pound
>>distribution be useful?
>>
>>The second issue you did not address is the problem of sessions: by
>>changing the config in a running program you create mayhem for the
>>existing sessions. How should that be dealt with?
> 
> 
> If pound is running setuid/gid then one's config file can be owned by root or
> by some other non-pound user, and readable to pound but not writable. If
pound
> is compromised, the compromised pound would only be able to read the config
> file, but not change it. However, a true administrator could modify the
config
> and tell pound to reload.
> 
> As far as sessions go, here are some brainstorming ideas: 
> 
> Assign a checksum to each URLGroup. Compare old configfile urlgroups to the
new
> configfile's URLGroups, and if they match 100%, keep those sessions active.
If
> they don't match, let the current inquiries finish and terminate the
sessions.
> 
> One may also argue that when the config file changes, sessions become
> completely meaningless. This is a model used by classic Active Server
Pages...
> You may just want to let all the transactions-in-process finish, and start
new
> sessions for all new connections. 
> 
> Ideally, when pound receives a signal to reload, pound would have a split
> personality for a small amount of time, letting old connections finish, and
> forcing new connections to use the new ruleset. Can't this be done by
starting
> a new group of threads and then retiring the old threads when they become
idle?
> The memory structures pertaining to session information which should survive
> the transition can persist.
> 
> If none of this sounds good, and we finally abandon the idea of reloading
> config files, I would settle for a "graceful shutdown" option ... A "graceful
> shutdown" consists of letting all in-process connections complete for a
maximum
> amount of time (say, 30 seconds, or even n seconds specified in the config
> file... :) and deny all new incoming requests with a 503 until all
transactions
> complete or are finally terminated, at which time a new pound would load up.
> 
> 
> 		
> __________________________________ 
> Discover Yahoo! 
> Stay in touch with email, IM, photo sharing and more. Check it out! 
> http://discover.yahoo.com/stayintouch.html
> 

Hi Robert,

> By changing the config file an attacker can gain access to servers that
> should be private. We feel that is a serious problem.

I think there are other ways to do that. If the attacker can either
modify the filesystem (in the jail) or inject code into Pound (e.g. via
a buffer overflow) then they don't need to modify the config file to
gain access to those servers.

Cheers, Chris.
-- 
(aidworld) chris wilson | chief engineer (chris(at)aidworld.org)

In that case the config files are the least of a sysadmin's worries.

--- Chris Wilson <chris(at)aidworld.org> wrote:

> Hi Robert,
> 
> > By changing the config file an attacker can gain access to servers that
> > should be private. We feel that is a serious problem.
> 
> I think there are other ways to do that. If the attacker can either
> modify the filesystem (in the jail) or inject code into Pound (e.g. via
> a buffer overflow) then they don't need to modify the config file to
> gain access to those servers.
> 
> Cheers, Chris.
> -- 
> (aidworld) chris wilson | chief engineer (chris(at)aidworld.org)
> 
> 
> -- 
> To unsubscribe send an email with subject 'unsubscribe' to pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
>
http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-06/1118772641000/1118916496000
> 



		
__________________________________ 
Discover Yahoo! 
Find restaurants, movies, travel and more fun for the weekend. Check it out! 
http://discover.yahoo.com/weekend.html 

Except with a config "reload" new incoming requests would not be refused during
the "graceful shutdown" period. Incoming requests would instead be serviced by
the new config file rules.

--- Ed R Zahurak <ezahurak(at)atlanticbb.net> wrote:

> Really, couldn't this be "approximated" by a graceful shutdown of the 
> backend webservers *first*, assuming they're capable of doing so, then 
> killing pound and restarting it?
> 
> CG wrote:
> > 
> > --- Robert Segall <roseg(at)apsis.ch> wrote:
> > 
> > 
> >>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:
> >>
> >>
> >>>>This subject was discussed repeatedly on this list and the simple
> >>
> >>answer
> >>
> >>>>is that it doesn't work: if you run in a root jail the config file
> >>
> >>is
> >>
> >>>>not accessible at all, not to mention the issue with existing
> >>
> >>sessions.
> >>
> >>>I mean this in the nicest way possible: Repeated discussion might be a
> >>
> >>sign
> >>
> >>>that this feature may be useful to a wide group of pound users, and it
> >>
> >>could be
> >>
> >>>worthwile to spend some time in sincere exploration.
> >>
> >>That's the way I took it. I fully understand it is a desirable feature -
> >>I just don't know how it can be done...
> >>
> >>
> >>>I can think of two solutions for the chroot problem. Why not make the
> >>
> >>config
> >>
> >>>file a cellmate for pound to talk to? Or, Why not just run pound
> >>
> >>without chroot
> >>
> >>>if you want this feature?
> >>>
> >>>A quick thought... If jailed, Apache keeps its config files in its
> >>
> >>jail. If
> >>
> >>>compromised, Apache has equal or more potential for destruction than
> >>
> >>pound. I
> >>
> >>>understand that some may see keeping the config file in the chroot
> >>
> >>jail as
> >>
> >>>being dangerous, but I feel that I can mitigate the danger by
> >>
> >>effectively
> >>
> >>>isolating the server as I would any other server running services
> >>
> >>accessable to
> >>
> >>>the outside world.
> >>
> >>By changing the config file an attacker can gain access to servers that
> >>should be private. We feel that is a serious problem.
> >>
> >>
> >>>We've thought about using the HA port to "down" a balanced webserver,
> >>
> >>but it
> >>
> >>>puts the function of removal onto the webserver itself, and not on
> >>
> >>pound. It
> >>
> >>>also doesn't solve the issue of migrating to a new config file during
> >>
> >>a period
> >>
> >>>of uptime.
> >>
> >>It doesn't have to be the server itself (even though some servers
> >>support it). It can just as easily be a small monitor program that you
> >>can start and stop at will. Would adding such a program to the Pound
> >>distribution be useful?
> >>
> >>The second issue you did not address is the problem of sessions: by
> >>changing the config in a running program you create mayhem for the
> >>existing sessions. How should that be dealt with?
> > 
> > 
> > If pound is running setuid/gid then one's config file can be owned by root
> or
> > by some other non-pound user, and readable to pound but not writable. If
> pound
> > is compromised, the compromised pound would only be able to read the config
> > file, but not change it. However, a true administrator could modify the
> config
> > and tell pound to reload.
> > 
> > As far as sessions go, here are some brainstorming ideas: 
> > 
> > Assign a checksum to each URLGroup. Compare old configfile urlgroups to the
> new
> > configfile's URLGroups, and if they match 100%, keep those sessions active.
> If
> > they don't match, let the current inquiries finish and terminate the
> sessions.
> > 
> > One may also argue that when the config file changes, sessions become
> > completely meaningless. This is a model used by classic Active Server
> Pages...
> > You may just want to let all the transactions-in-process finish, and start
> new
> > sessions for all new connections. 
> > 
> > Ideally, when pound receives a signal to reload, pound would have a split
> > personality for a small amount of time, letting old connections finish, and
> > forcing new connections to use the new ruleset. Can't this be done by
> starting
> > a new group of threads and then retiring the old threads when they become
> idle?
> > The memory structures pertaining to session information which should
> survive
> > the transition can persist.
> > 
> > If none of this sounds good, and we finally abandon the idea of reloading
> > config files, I would settle for a "graceful shutdown" option ... A
> "graceful
> > shutdown" consists of letting all in-process connections complete for a
> maximum
> > amount of time (say, 30 seconds, or even n seconds specified in the config
> > file... :) and deny all new incoming requests with a 503 until all
> transactions
> > complete or are finally terminated, at which time a new pound would load
> up.
> > 
> > 
> > 		
> > __________________________________ 
> > Discover Yahoo! 
> > Stay in touch with email, IM, photo sharing and more. Check it out! 
> > http://discover.yahoo.com/stayintouch.html
> > 
> 
> 
> -- 
> To unsubscribe send an email with subject 'unsubscribe' to pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
>
http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-06/1118772641000/1118887663000
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

I think there might still be a way to do it.  How many web servers do 
you have behind pound, in your instance?

Ed


CG wrote:
> Except with a config "reload" new incoming requests would not be refused
during
> the "graceful shutdown" period. Incoming requests would instead be serviced
by
> the new config file rules.
> 
> --- Ed R Zahurak <ezahurak(at)atlanticbb.net> wrote:
> 
> 
>>Really, couldn't this be "approximated" by a graceful shutdown of the 
>>backend webservers *first*, assuming they're capable of doing so, then 
>>killing pound and restarting it?
>>
>>CG wrote:
>>
>>>--- Robert Segall <roseg(at)apsis.ch> wrote:
>>>
>>>
>>>
>>>>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com> wrote:
>>>>
>>>>
>>>>
>>>>>>This subject was discussed repeatedly on this list and the simple
>>>>
>>>>answer
>>>>
>>>>
>>>>>>is that it doesn't work: if you run in a root jail the config file
>>>>
>>>>is
>>>>
>>>>
>>>>>>not accessible at all, not to mention the issue with existing
>>>>
>>>>sessions.
>>>>
>>>>
>>>>>I mean this in the nicest way possible: Repeated discussion might be a
>>>>
>>>>sign
>>>>
>>>>
>>>>>that this feature may be useful to a wide group of pound users, and it
>>>>
>>>>could be
>>>>
>>>>
>>>>>worthwile to spend some time in sincere exploration.
>>>>
>>>>That's the way I took it. I fully understand it is a desirable feature -
>>>>I just don't know how it can be done...
>>>>
>>>>
>>>>
>>>>>I can think of two solutions for the chroot problem. Why not make the
>>>>
>>>>config
>>>>
>>>>
>>>>>file a cellmate for pound to talk to? Or, Why not just run pound
>>>>
>>>>without chroot
>>>>
>>>>
>>>>>if you want this feature?
>>>>>
>>>>>A quick thought... If jailed, Apache keeps its config files in its
>>>>
>>>>jail. If
>>>>
>>>>
>>>>>compromised, Apache has equal or more potential for destruction than
>>>>
>>>>pound. I
>>>>
>>>>
>>>>>understand that some may see keeping the config file in the chroot
>>>>
>>>>jail as
>>>>
>>>>
>>>>>being dangerous, but I feel that I can mitigate the danger by
>>>>
>>>>effectively
>>>>
>>>>
>>>>>isolating the server as I would any other server running services
>>>>
>>>>accessable to
>>>>
>>>>
>>>>>the outside world.
>>>>
>>>>By changing the config file an attacker can gain access to servers that
>>>>should be private. We feel that is a serious problem.
>>>>
>>>>
>>>>
>>>>>We've thought about using the HA port to "down" a balanced webserver,
>>>>
>>>>but it
>>>>
>>>>
>>>>>puts the function of removal onto the webserver itself, and not on
>>>>
>>>>pound. It
>>>>
>>>>
>>>>>also doesn't solve the issue of migrating to a new config file during
>>>>
>>>>a period
>>>>
>>>>
>>>>>of uptime.
>>>>
>>>>It doesn't have to be the server itself (even though some servers
>>>>support it). It can just as easily be a small monitor program that you
>>>>can start and stop at will. Would adding such a program to the Pound
>>>>distribution be useful?
>>>>
>>>>The second issue you did not address is the problem of sessions: by
>>>>changing the config in a running program you create mayhem for the
>>>>existing sessions. How should that be dealt with?
>>>
>>>
>>>If pound is running setuid/gid then one's config file can be owned by root
>>
>>or
>>
>>>by some other non-pound user, and readable to pound but not writable. If
>>
>>pound
>>
>>>is compromised, the compromised pound would only be able to read the config
>>>file, but not change it. However, a true administrator could modify the
>>
>>config
>>
>>>and tell pound to reload.
>>>
>>>As far as sessions go, here are some brainstorming ideas: 
>>>
>>>Assign a checksum to each URLGroup. Compare old configfile urlgroups to the
>>
>>new
>>
>>>configfile's URLGroups, and if they match 100%, keep those sessions active.
>>
>>If
>>
>>>they don't match, let the current inquiries finish and terminate the
>>
>>sessions.
>>
>>>One may also argue that when the config file changes, sessions become
>>>completely meaningless. This is a model used by classic Active Server
>>
>>Pages...
>>
>>>You may just want to let all the transactions-in-process finish, and start
>>
>>new
>>
>>>sessions for all new connections. 
>>>
>>>Ideally, when pound receives a signal to reload, pound would have a split
>>>personality for a small amount of time, letting old connections finish, and
>>>forcing new connections to use the new ruleset. Can't this be done by
>>
>>starting
>>
>>>a new group of threads and then retiring the old threads when they become
>>
>>idle?
>>
>>>The memory structures pertaining to session information which should
>>
>>survive
>>
>>>the transition can persist.
>>>
>>>If none of this sounds good, and we finally abandon the idea of reloading
>>>config files, I would settle for a "graceful shutdown" option ... A
>>
>>"graceful
>>
>>>shutdown" consists of letting all in-process connections complete for a
>>
>>maximum
>>
>>>amount of time (say, 30 seconds, or even n seconds specified in the config
>>>file... :) and deny all new incoming requests with a 503 until all
>>
>>transactions
>>
>>>complete or are finally terminated, at which time a new pound would load
>>
>>up.
>>
>>>
>>>		
>>>__________________________________ 
>>>Discover Yahoo! 
>>>Stay in touch with email, IM, photo sharing and more. Check it out! 
>>>http://discover.yahoo.com/stayintouch.html
>>>
>>
>>
>>-- 
>>To unsubscribe send an email with subject 'unsubscribe' to pound(at)apsis.ch.
>>Please contact roseg(at)apsis.ch for questions.
>>
> 
>
http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-06/1118772641000/1118887663000
> 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> 

Here's a few possible approaches

A. Re-read configuration file from one of the backend webservers
B. Start another pound instance, which listens on a backup socket
   Send HUP to initial pound instance, which forwards new sessions to
backup pound
   When existing sessions are all dead, backup pound takes over the
normal socket
 

-----Original Message-----
From: Ed R Zahurak [mailto:ezahurak(at)atlanticbb.net] 
Sent: Friday, 17 June 2005 9:18 AM
To: pound(at)apsis.ch
Subject: Re: [Pound Mailing List] Signal HUP to reload config files
without terminating connections

I think there might still be a way to do it.  How many web servers do
you have behind pound, in your instance?

Ed


CG wrote:
> Except with a config "reload" new incoming requests would not be 
> refused during the "graceful shutdown" period. Incoming requests would

> instead be serviced by the new config file rules.
> 
> --- Ed R Zahurak <ezahurak(at)atlanticbb.net> wrote:
> 
> 
>>Really, couldn't this be "approximated" by a graceful shutdown of the 
>>backend webservers *first*, assuming they're capable of doing so, then

>>killing pound and restarting it?
>>
>>CG wrote:
>>
>>>--- Robert Segall <roseg(at)apsis.ch> wrote:
>>>
>>>
>>>
>>>>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com>
wrote:
>>>>
>>>>
>>>>
>>>>>>This subject was discussed repeatedly on this list and the simple
>>>>
>>>>answer
>>>>
>>>>
>>>>>>is that it doesn't work: if you run in a root jail the config file
>>>>
>>>>is
>>>>
>>>>
>>>>>>not accessible at all, not to mention the issue with existing
>>>>
>>>>sessions.
>>>>
>>>>
>>>>>I mean this in the nicest way possible: Repeated discussion might 
>>>>>be a
>>>>
>>>>sign
>>>>
>>>>
>>>>>that this feature may be useful to a wide group of pound users, and

>>>>>it
>>>>
>>>>could be
>>>>
>>>>
>>>>>worthwile to spend some time in sincere exploration.
>>>>
>>>>That's the way I took it. I fully understand it is a desirable 
>>>>feature - I just don't know how it can be done...
>>>>
>>>>
>>>>
>>>>>I can think of two solutions for the chroot problem. Why not make 
>>>>>the
>>>>
>>>>config
>>>>
>>>>
>>>>>file a cellmate for pound to talk to? Or, Why not just run pound
>>>>
>>>>without chroot
>>>>
>>>>
>>>>>if you want this feature?
>>>>>
>>>>>A quick thought... If jailed, Apache keeps its config files in its
>>>>
>>>>jail. If
>>>>
>>>>
>>>>>compromised, Apache has equal or more potential for destruction 
>>>>>than
>>>>
>>>>pound. I
>>>>
>>>>
>>>>>understand that some may see keeping the config file in the chroot
>>>>
>>>>jail as
>>>>
>>>>
>>>>>being dangerous, but I feel that I can mitigate the danger by
>>>>
>>>>effectively
>>>>
>>>>
>>>>>isolating the server as I would any other server running services
>>>>
>>>>accessable to
>>>>
>>>>
>>>>>the outside world.
>>>>
>>>>By changing the config file an attacker can gain access to servers 
>>>>that should be private. We feel that is a serious problem.
>>>>
>>>>
>>>>
>>>>>We've thought about using the HA port to "down" a balanced 
>>>>>webserver,
>>>>
>>>>but it
>>>>
>>>>
>>>>>puts the function of removal onto the webserver itself, and not on
>>>>
>>>>pound. It
>>>>
>>>>
>>>>>also doesn't solve the issue of migrating to a new config file 
>>>>>during
>>>>
>>>>a period
>>>>
>>>>
>>>>>of uptime.
>>>>
>>>>It doesn't have to be the server itself (even though some servers 
>>>>support it). It can just as easily be a small monitor program that 
>>>>you can start and stop at will. Would adding such a program to the 
>>>>Pound distribution be useful?
>>>>
>>>>The second issue you did not address is the problem of sessions: by 
>>>>changing the config in a running program you create mayhem for the 
>>>>existing sessions. How should that be dealt with?
>>>
>>>
>>>If pound is running setuid/gid then one's config file can be owned by

>>>root
>>
>>or
>>
>>>by some other non-pound user, and readable to pound but not writable.

>>>If
>>
>>pound
>>
>>>is compromised, the compromised pound would only be able to read the 
>>>config file, but not change it. However, a true administrator could 
>>>modify the
>>
>>config
>>
>>>and tell pound to reload.
>>>
>>>As far as sessions go, here are some brainstorming ideas: 
>>>
>>>Assign a checksum to each URLGroup. Compare old configfile urlgroups 
>>>to the
>>
>>new
>>
>>>configfile's URLGroups, and if they match 100%, keep those sessions
active.
>>
>>If
>>
>>>they don't match, let the current inquiries finish and terminate the
>>
>>sessions.
>>
>>>One may also argue that when the config file changes, sessions become

>>>completely meaningless. This is a model used by classic Active Server
>>
>>Pages...
>>
>>>You may just want to let all the transactions-in-process finish, and 
>>>start
>>
>>new
>>
>>>sessions for all new connections. 
>>>
>>>Ideally, when pound receives a signal to reload, pound would have a 
>>>split personality for a small amount of time, letting old connections

>>>finish, and forcing new connections to use the new ruleset. Can't 
>>>this be done by
>>
>>starting
>>
>>>a new group of threads and then retiring the old threads when they 
>>>become
>>
>>idle?
>>
>>>The memory structures pertaining to session information which should
>>
>>survive
>>
>>>the transition can persist.
>>>
>>>If none of this sounds good, and we finally abandon the idea of 
>>>reloading config files, I would settle for a "graceful shutdown" 
>>>option ... A
>>
>>"graceful
>>
>>>shutdown" consists of letting all in-process connections complete for

>>>a
>>
>>maximum
>>
>>>amount of time (say, 30 seconds, or even n seconds specified in the 
>>>config file... :) and deny all new incoming requests with a 503 until

>>>all
>>
>>transactions
>>
>>>complete or are finally terminated, at which time a new pound would 
>>>load
>>
>>up.
>>
>>>
>>>		
>>>__________________________________
>>>Discover Yahoo! 
>>>Stay in touch with email, IM, photo sharing and more. Check it out! 
>>>http://discover.yahoo.com/stayintouch.html
>>>
>>
>>
>>--
>>To unsubscribe send an email with subject 'unsubscribe' to
pound(at)apsis.ch.
>>Please contact roseg(at)apsis.ch for questions.
>>
> 
> http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-06/11
> 18772641000/1118887663000
> 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com
> 


-- 
To unsubscribe send an email with subject 'unsubscribe' to
pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-06/1118
772641000/1118963890000

Approach B has some definite potential. The only issue I can see is that this
would kill all sessions which occur in SSL encrypted webspace. 

Client(SSL) --> Pound(Dying) --forwards without decryption--> Pound(Live) 

Pound(Live) decodes the request and finds that it doesn't recognise the
session. 

Of couse Pound(Dying) could continue to run and service active http sessions
until they were over, assuming that the old config is still valid for those
session. That's a pretty big assumption. It might be better to let the
transactions terminate and then pass over instead of waiting for the sessions
to finish.

CG

--- "Chui G. Tey" <chui.tey(at)advdata.com.au> wrote:

> Here's a few possible approaches
> 
> A. Re-read configuration file from one of the backend webservers
> B. Start another pound instance, which listens on a backup socket
>    Send HUP to initial pound instance, which forwards new sessions to
> backup pound
>    When existing sessions are all dead, backup pound takes over the
> normal socket
>  
> 
> -----Original Message-----
> From: Ed R Zahurak [mailto:ezahurak(at)atlanticbb.net] 
> Sent: Friday, 17 June 2005 9:18 AM
> To: pound(at)apsis.ch
> Subject: Re: [Pound Mailing List] Signal HUP to reload config files
> without terminating connections
> 
> I think there might still be a way to do it.  How many web servers do
> you have behind pound, in your instance?
> 
> Ed
> 
> 
> CG wrote:
> > Except with a config "reload" new incoming requests would not be 
> > refused during the "graceful shutdown" period. Incoming requests would
> 
> > instead be serviced by the new config file rules.
> > 
> > --- Ed R Zahurak <ezahurak(at)atlanticbb.net> wrote:
> > 
> > 
> >>Really, couldn't this be "approximated" by a graceful shutdown of the 
> >>backend webservers *first*, assuming they're capable of doing so, then
> 
> >>killing pound and restarting it?
> >>
> >>CG wrote:
> >>
> >>>--- Robert Segall <roseg(at)apsis.ch> wrote:
> >>>
> >>>
> >>>
> >>>>On Wed, 15 Jun 2005 06:10:42 -0700 (PDT) CG <cgg007(at)yahoo.com>
> wrote:
> >>>>
> >>>>
> >>>>
> >>>>>>This subject was discussed repeatedly on this list and the simple
> >>>>
> >>>>answer
> >>>>
> >>>>
> >>>>>>is that it doesn't work: if you run in a root jail the config file
> >>>>
> >>>>is
> >>>>
> >>>>
> >>>>>>not accessible at all, not to mention the issue with existing
> >>>>
> >>>>sessions.
> >>>>
> >>>>
> >>>>>I mean this in the nicest way possible: Repeated discussion might 
> >>>>>be a
> >>>>
> >>>>sign
> >>>>
> >>>>
> >>>>>that this feature may be useful to a wide group of pound users, and
> 
> >>>>>it
> >>>>
> >>>>could be
> >>>>
> >>>>
> >>>>>worthwile to spend some time in sincere exploration.
> >>>>
> >>>>That's the way I took it. I fully understand it is a desirable 
> >>>>feature - I just don't know how it can be done...
> >>>>
> >>>>
> >>>>
> >>>>>I can think of two solutions for the chroot problem. Why not make 
> >>>>>the
> >>>>
> >>>>config
> >>>>
> >>>>
> >>>>>file a cellmate for pound to talk to? Or, Why not just run pound
> >>>>
> >>>>without chroot
> >>>>
> >>>>
> >>>>>if you want this feature?
> >>>>>
> >>>>>A quick thought... If jailed, Apache keeps its config files in its
> >>>>
> >>>>jail. If
> >>>>
> >>>>
> >>>>>compromised, Apache has equal or more potential for destruction 
> >>>>>than
> >>>>
> >>>>pound. I
> >>>>
> >>>>
> >>>>>understand that some may see keeping the config file in the chroot
> >>>>
> >>>>jail as
> >>>>
> >>>>
> >>>>>being dangerous, but I feel that I can mitigate the danger by
> >>>>
> >>>>effectively
> >>>>
> >>>>
> >>>>>isolating the server as I would any other server running services
> >>>>
> >>>>accessable to
> >>>>
> >>>>
> >>>>>the outside world.
> >>>>
> >>>>By changing the config file an attacker can gain access to servers 
> >>>>that should be private. We feel that is a serious problem.
> >>>>
> >>>>
> >>>>
> >>>>>We've thought about using the HA port to "down" a balanced 
> >>>>>webserver,
> >>>>
> >>>>but it
> >>>>
> >>>>
> >>>>>puts the function of removal onto the webserver itself, and not on
> >>>>
> >>>>pound. It
> >>>>
> >>>>
> >>>>>also doesn't solve the issue of migrating to a new config file 
> >>>>>during
> >>>>
> >>>>a period
> >>>>
> >>>>
> >>>>>of uptime.
> >>>>
> >>>>It doesn't have to be the server itself (even though some servers 
> >>>>support it). It can just as easily be a small monitor program that 
> >>>>you can start and stop at will. Would adding such a program to the 
> >>>>Pound distribution be useful?
> >>>>
> >>>>The second issue you did not address is the problem of sessions: by 
> >>>>changing the config in a running program you create mayhem for the 
> >>>>existing sessions. How should that be dealt with?
> >>>
> >>>
> >>>If pound is running setuid/gid then one's config file can be owned by
> 
> >>>root
> >>
> >>or
> >>
> >>>by some other non-pound user, and readable to pound but not writable.
> 
> >>>If
> >>
> >>pound
> >>
> >>>is compromised, the compromised pound would only be able to read the 
> >>>config file, but not change it. However, a true administrator could 
> >>>modify the
> >>
> >>config
> >>
> >>>and tell pound to reload.
> >>>
> >>>As far as sessions go, here are some brainstorming ideas: 
> >>>
> >>>Assign a checksum to each URLGroup. Compare old configfile urlgroups 
> >>>to the
> >>
> >>new
> >>
> >>>configfile's URLGroups, and if they match 100%, keep those sessions
> active.
> >>
> >>If
> >>
> >>>they don't match, let the current inquiries finish and terminate the
> >>
> >>sessions.
> >>
> >>>One may also argue that when the config file changes, sessions become
> 
> >>>completely meaningless. This is a model used by classic Active Server
> >>
> >>Pages...
> >>
> >>>You may just want to let all the transactions-in-process finish, and 
> 
=== message truncated ===



		
____________________________________________________ 
Yahoo! Sports 
Rekindle the Rivalries. Sign up for Fantasy Football 
http://football.fantasysports.yahoo.com

Hi Robert,

--On 15 June 2005 17:01 +0200 Robert Segall <roseg(at)apsis.ch> wrote:

<-- snip -->
>> We've thought about using the HA port to "down" a balanced webserver,
> but it
>> puts the function of removal onto the webserver itself, and not on
> pound. It
>> also doesn't solve the issue of migrating to a new config file during
> a period
>> of uptime.
>
> It doesn't have to be the server itself (even though some servers
> support it). It can just as easily be a small monitor program that you
> can start and stop at will. Would adding such a program to the Pound
> distribution be useful?
>

FWIW, I would find such a program very useful.  For example, in a Zope/Zeo 
setup, it would be handy to take the ZEO "backend" client down, update the 
code, bring it back up and test it again, without making it accessible via 
Pound. A standalone monitor program sounds like the easiest way to achieve 
this.

Many thanks,

Dominic

Am Donnerstag, 30. Juni 2005 16:41 schrieb Dominic Hiles:
> > It doesn't have to be the server itself (even though some servers
> > support it). It can just as easily be a small monitor program that you
> > can start and stop at will. Would adding such a program to the Pound
> > distribution be useful?
>
> FWIW, I would find such a program very useful.  For example, in a Zope/Zeo
> setup, it would be handy to take the ZEO "backend" client down, update the
> code, bring it back up and test it again, without making it accessible via
> Pound. A standalone monitor program sounds like the easiest way to achieve
> this.

How about this script:

    $ cat backend_monitor.py
    #!/bin/env python
    from SocketServer import BaseRequestHandler, TCPServer
    
    class NoOpMonitor(BaseRequestHandler):
    
        def handle(self):
            pass
    
    server = TCPServer(('127.0.0.1', 55555), NoOpMonitor)
    server.serve_forever()


and a pound config like

   BackEnd 127.0.0.1,80,55555


run in background

    $ ./backend_monitor.py &
    [1] 26845


now pound can connect to the ha_port:

    $ telnet localhost 55555
    Trying 127.0.0.1...
    Connected to localhost.localdomain (127.0.0.1).
    Escape character is '^]'.
    Connection closed by foreign host.


if the monitor is gone...

    $ kill 26845


pound should assume the backends dead:

    $ telnet localhost 55555
    Trying 127.0.0.1...
    telnet: connect to address 127.0.0.1: Connection refused
    telnet: Unable to connect to remote host: Connection refused


I didn't test it with pound, and don't know how to make sure that no other 
program uses the same port while the monitor is down (maybe it would be best 
to listen on a low port to prevent this?), but it might help as a starting 
point.

I'd be very interesting if this works :-)


Cheers,

Sascha



-- 
Gallileus - the power of knowledge

Gallileus GmbH                   http://www.gallileus.info/

Pintschstraße 16                  fon +49-(0)30-41 93 43 43
10249 Berlin                      fax +49-(0)30-41 93 43 45
Germany

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
AKTUELLER HINWEIS (Juni 2005)

Gallilaus des Monats Juni:

Herr Karl-Heinz von Rothenburg ist bei vielen Schülern wegen
seiner Übersetzungen der Comic-Reihe "Asterix und Obelix"
bekannt. Lesen Sie mehr unter:

http://www.gallileus.info/events/gallilaus/news/111753701335
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Am Donnerstag, 30. Juni 2005 18:09 schrieb Sascha Ottolski:
> and a pound config like
>
>    BackEnd 127.0.0.1,80,55555

ups, I think it should read

    BackEnd 127.0.0.1,80,9,55555

I missed the priority.


Cheers, Sascha