I know that there has been a fair bit of discussion regarding both the 
perceived need for changing configuration on the fly as well as some 
objections to this potential feature.

I would like to put myself firmly in the camp of those who would dearly 
like to have such a feature.  Simply restarting pound doesn't suffice 
for a number of reasons.  These include:

a) session affinity is lost
b) unless some care is taken, requests or responses in transit at the 
time of killing can be lost

Questions have also been raised about the required semantics of the 
reconfigure process.  These questions include:

a) what happens if pound is chrooted into a place where it can't read 
the new config?
b) how should the reconfiguration be triggered?
c) what will happen with sessions instituted under the old configuration?
d) what will happen as in (c) when the old sessions conflict with the 
new configuration?

I don't think that any of these questions are all that difficult, except 
possibly for (d).  In (a), the obvious answer is that the 
reconfiguration fails and the old configuration applies.  The failure 
should be logged, of course.  In (b), the traditional answer is by kill 
-HUP, but sending an HTTP post to pound via a priveleged port is not a 
bad alternative which also helps with question (a).

For item (c), all sessions that were instituted under the old 
configuration that are viable under the new configuration should 
obviously stay in place.  If the sessions would not have been started, 
then they fall under (d).

This leaves (d).  To my mind, there are three viable policies which are 
each correct in different circumstances.  I think that which policy is 
applied should be selectable at reconfiguration time.

Policy (1) would be to discard all all existing sessions.  During the 
reconfiguration, new incoming requests would be queued and outgoing 
responses would be forwarded as usual.  This is just a clean 
re-implementation of the current program restart approach that avoids 
dropped requests or responses.

Policy (2) would be to discard only sessions that would not have been 
started under the new configuration.  This would otherwise be the same 
as policy (1).

Policy (3) would be to not discard any old session information, but to 
instead only allow new sessions to be instituted according to the new 
configuration.  This would allow a soft disconnect of servers.

It may be that this raises a new objection to this feature:

e) The current code organization makes it hard to implement a phased 
cutover.

My answer to this objection is that if this is the real problem, then 
that should be stated.  If it is the problem, then the solution 
shouldn't be all that difficult ... the question really just has to do 
with tracking the rationale for how a session was started.  This 
tracking information should make it relatively easy to determine whether 
the session should continue depending on which policy is in effect.

Any thoughts?

Ted Dunning wrote:
> I know that there has been a fair bit of discussion regarding both the 
> perceived need for changing configuration on the fly as well as some 
> objections to this potential feature.
> 
> I would like to put myself firmly in the camp of those who would dearly 
> like to have such a feature.  Simply restarting pound doesn't suffice 
> for a number of reasons.  These include:

Thanks for raising this again. I really think this is a huge factor
in getting Pound more widely adopted. For high availability 
environments, having graceful restarts and the ability to not lose
the session affinity is just crucial.

Robert, what can we do to help solve this?

Maurice

Even something as easy as being able to serialize the session structure
would be useful.  I have no problem doing a kill -USR1, having it dump
the session structure, and then read it on next load.  It could kill the
sessions that no longer make sense.  As long as the majority of the
sticky sessions survive, I'll be happy.

On a more complicated note, any plans to move to C++ (or other OO) to
make memory leaks less likely?  I haven't noticed anything, but I've
grown a distrust for C after years of OO training.  (Or shall we say a
respect for smaller objects that are easier to test and verify proper
operation)

Joe Gooch
K12 Systems

-----Original Message-----
From: Maurice Aubrey [mailto:maurice(at)redweek.com] 
Sent: Tuesday, August 30, 2005 4:23 PM
To: pound(at)apsis.ch
Subject: Re: [Pound Mailing List] changing conf without restart

Ted Dunning wrote:
> I know that there has been a fair bit of discussion regarding both the

> perceived need for changing configuration on the fly as well as some 
> objections to this potential feature.
> 
> I would like to put myself firmly in the camp of those who would
dearly 
> like to have such a feature.  Simply restarting pound doesn't suffice 
> for a number of reasons.  These include:

Thanks for raising this again. I really think this is a huge factor
in getting Pound more widely adopted. For high availability 
environments, having graceful restarts and the ability to not lose
the session affinity is just crucial.

Robert, what can we do to help solve this?

Maurice

-- 
To unsubscribe send an email with subject 'unsubscribe' to
pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-08/1125
362245000/1125433383000

This really would meet most of my needs.  The most common use cases for 
me are adding a new server and rolling new versions of servers.  When 
adding a new server, serializing the session structure would work just 
fine since no sessions would be invalidated.  For rolling new versions, 
the process is a bit more complex involving setting a priority to 0 for 
a server on some box and adding the new version on that same box at a 
low priority (to allow for waking the server up gently).  Once the new 
version is warmed up and and old sessions have expired, the next server 
can be rolled and the priority of the first server's new version can be 
ramped up to full bandwidth.

Joe Gooch wrote:

>Even something as easy as being able to serialize the session structure
>would be useful.  I have no problem doing a kill -USR1, having it dump
>the session structure, and then read it on next load.  It could kill the
>sessions that no longer make sense.  As long as the majority of the
>sticky sessions survive, I'll be happy.
>
>  
>


-- 
Ted Dunning
Chief Scientist
Veoh Networks

On Mon, 29 Aug 2005 17:37:25 -0700 Ted Dunning <tdunning(at)veoh.com>
wrote:

> I know that there has been a fair bit of discussion regarding both the
> perceived need for changing configuration on the fly as well as some 
> objections to this potential feature.
> 
> I would like to put myself firmly in the camp of those who would
> dearly like to have such a feature.  Simply restarting pound doesn't
> suffice for a number of reasons.  These include:
> 
> a) session affinity is lost
> b) unless some care is taken, requests or responses in transit at the 
> time of killing can be lost

Let me try to summarize what was said here:

1. the ability to change the configuration without stopping/restarting
Pound is desirable.

2. it is also desirable to keep the existing sessions through the
change, at least as much as possible.

Looking at the existing code (your hidden assumption - we want to change
as little as possible) we could implement the following:

1. allow for configuration changes on the fly. The change is triggered
by a signal (kill -HUP), never by an HTTP request (security). Support
for this feature could be reasonably done if only if Pound is compiled
in supervisor mode (configure --enable-super, which is the default). The
supervisor process, which has nothing to do with serving HTTP, would not
be root-jailed and thus able to re-read the configuration, stop the
worker process and restart it.

2. we can keep existing sessions by having them maintained in the
supervisor process. The old sessions remain in effect as long as the
back-ends respond - if not the back-end is declared dead and the
sessions invalidated. New requests would create sessions by referring
to the new configuration.

Such a solution still leaves a few questions open:

1. requests currently being processed would be mercilessly aborted.
Would this be acceptable?

2. how are we to deal with "mixed" responses? If you look at the code
you'll notice that sessions may be switched in mid-stream (e.g. we have
an HTTP/1.1 client that makes multiple requests, which may get split to
separate back-ends; some of the requests would "follow" the old
sessions, some the new).

Let me know.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

Robert,

Just thinking out loud here.  How about allowing for a graceful shutdown 
of an instance of pound?  You could then cascade a few instances of 
pound running behind one main instance, (with minimal configuration, 
that hopefully you'd not have to stop often,) like this:

                         [pound 'overlord']
                            |          |
                      [pound 'sub1'] [pound 'sub2']
                         |     |           |     |
                      [backend servers all go here.]

When a configuration on sub1 and sub2 need changed, you could issue a 
graceful shutdown to those instances, one at a time, to handle current 
connections, not take any new ones, and shut down when the last current 
connection terminates.  The HA already present in the overlord process 
would route new connections to the other subinstance, and when the 
shut-down subinstance is restarted, start routing connections to that 
instance.  The process could then be repeated for the other subinstance.

Obviously, this wouldn't work in a scenario where sessions *must* go to 
one backend box for the life of the session, but I can't think of 
practical means, aside from saving session information to disk to be 
read on restart that would solve that, not that I think it's a priority 
to solve that particular problem in the first place.

Robert Segall wrote:
> On Mon, 29 Aug 2005 17:37:25 -0700 Ted Dunning <tdunning(at)veoh.com>
> wrote:
> 
> 
>>I know that there has been a fair bit of discussion regarding both the
>>perceived need for changing configuration on the fly as well as some 
>>objections to this potential feature.
>>
>>I would like to put myself firmly in the camp of those who would
>>dearly like to have such a feature.  Simply restarting pound doesn't
>>suffice for a number of reasons.  These include:
>>
>>a) session affinity is lost
>>b) unless some care is taken, requests or responses in transit at the 
>>time of killing can be lost
> 
> 
> Let me try to summarize what was said here:
> 
> 1. the ability to change the configuration without stopping/restarting
> Pound is desirable.
> 
> 2. it is also desirable to keep the existing sessions through the
> change, at least as much as possible.
> 
> Looking at the existing code (your hidden assumption - we want to change
> as little as possible) we could implement the following:
> 
> 1. allow for configuration changes on the fly. The change is triggered
> by a signal (kill -HUP), never by an HTTP request (security). Support
> for this feature could be reasonably done if only if Pound is compiled
> in supervisor mode (configure --enable-super, which is the default). The
> supervisor process, which has nothing to do with serving HTTP, would not
> be root-jailed and thus able to re-read the configuration, stop the
> worker process and restart it.
> 
> 2. we can keep existing sessions by having them maintained in the
> supervisor process. The old sessions remain in effect as long as the
> back-ends respond - if not the back-end is declared dead and the
> sessions invalidated. New requests would create sessions by referring
> to the new configuration.
> 
> Such a solution still leaves a few questions open:
> 
> 1. requests currently being processed would be mercilessly aborted.
> Would this be acceptable?
> 
> 2. how are we to deal with "mixed" responses? If you look at the code
> you'll notice that sessions may be switched in mid-stream (e.g. we have
> an HTTP/1.1 client that makes multiple requests, which may get split to
> separate back-ends; some of the requests would "follow" the old
> sessions, some the new).
> 
> Let me know.

Hi Robert,

Thanks for all your hard work on Pound. I know that my company and I really
appreciate your product.

To answer your questions:

> 1. requests currently being processed would be mercilessly aborted.
> Would this be acceptable?

The major thing we would want to see out of this change is that when a new
config is loaded, current connections are NOT affected (they are allowed to
finish as normal). I think that without this, there is not much point to
reloading the config (accept maybe the sessions stuff but we don't use that).
So, that is a "no" to your question.

> 2. how are we to deal with "mixed" responses? If you look at the code
> you'll notice that sessions may be switched in mid-stream (e.g. we have
> an HTTP/1.1 client that makes multiple requests, which may get split to
> separate back-ends; some of the requests would "follow" the old
> sessions, some the new).

Like I mentioned in the other answer, all of our backends are capable of
handling any request at any time. But I know this is important to other people.
I don't think there would be any problem with dealing with each request as you
have already suggested.

Thanks again,

John D.

********** Original Email *********
** To:   pound(at)apsis.ch
** From: pound(at)apsis.ch, roseg(at)apsis.ch
** Date: Wed, 31 Aug 2005 16:22:34 +0200
**********

On Mon, 29 Aug 2005 17:37:25 -0700 Ted Dunning <tdunning(at)veoh.com>
wrote:

> I know that there has been a fair bit of discussion regarding both the
> perceived need for changing configuration on the fly as well as some 
> objections to this potential feature.
> 
> I would like to put myself firmly in the camp of those who would
> dearly like to have such a feature.  Simply restarting pound doesn't
> suffice for a number of reasons.  These include:
> 
> a) session affinity is lost
> b) unless some care is taken, requests or responses in transit at the 
> time of killing can be lost

Let me try to summarize what was said here:

1. the ability to change the configuration without stopping/restarting
Pound is desirable.

2. it is also desirable to keep the existing sessions through the
change, at least as much as possible.

Looking at the existing code (your hidden assumption - we want to change
as little as possible) we could implement the following:

1. allow for configuration changes on the fly. The change is triggered
by a signal (kill -HUP), never by an HTTP request (security). Support
for this feature could be reasonably done if only if Pound is compiled
in supervisor mode (configure --enable-super, which is the default). The
supervisor process, which has nothing to do with serving HTTP, would not
be root-jailed and thus able to re-read the configuration, stop the
worker process and restart it.

2. we can keep existing sessions by having them maintained in the
supervisor process. The old sessions remain in effect as long as the
back-ends respond - if not the back-end is declared dead and the
sessions invalidated. New requests would create sessions by referring
to the new configuration.

Such a solution still leaves a few questions open:

1. requests currently being processed would be mercilessly aborted.
Would this be acceptable?

2. how are we to deal with "mixed" responses? If you look at the code
you'll notice that sessions may be switched in mid-stream (e.g. we have
an HTTP/1.1 client that makes multiple requests, which may get split to
separate back-ends; some of the requests would "follow" the old
sessions, some the new).

Let me know.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

-- 
To unsubscribe send an email with subject 'unsubscribe' to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
http://192.168.1.2:8080/Apsis/pound/pound_list/archive/2005/2005-08/1125362245000/1125498154000

Robert Segall wrote:

> ... parent/child design suggestion, kill -USR to parent triggers 
> infanticide and reincarnation ....
>
>Such a solution still leaves a few questions open:
>
>1. requests currently being processed would be mercilessly aborted.
>Would this be acceptable?
>  
>
I don't think so.  But this could be avoided by catching the signal and 
setting a flag which would cause all threads to terminate immediately 
after processing the current transaction and which would prevent any 
threads from checking for another connection.  In general requests to 
back-end servers should not take all that long (especially compared to 
session life-time).

>2. how are we to deal with "mixed" responses? If you look at the code
>you'll notice that sessions may be switched in mid-stream (e.g. we have
>an HTTP/1.1 client that makes multiple requests, which may get split to
>separate back-ends; some of the requests would "follow" the old
>sessions, some the new).
>  
>
I don't think that this really matters.  If session splitting is 
acceptable at all, then any semantics after a reconfiguration would by 
just as acceptable.


-- 
Ted Dunning
Chief Scientist
Veoh Networks

Ted Dunning wrote:
> Robert Segall wrote:
> 
>> ... parent/child design suggestion, kill -USR to parent triggers 
>> infanticide and reincarnation ....
>>
>> Such a solution still leaves a few questions open:
>>
>> 1. requests currently being processed would be mercilessly aborted.
>> Would this be acceptable?
>>
> I don't think so.  But this could be avoided by catching the signal and 
> setting a flag which would cause all threads to terminate immediately 
> after processing the current transaction and which would prevent any 
> threads from checking for another connection.  In general requests to 
> back-end servers should not take all that long (especially compared to 
> session life-time).

I agree. A graceful restart that allows existing requests to finish is
one of the main things I'm looking for. Otherwise we're guaranteed to
interrupt service.

Having the supervisor process maintain the sessions sounds ideal to me.

>> 2. how are we to deal with "mixed" responses? If you look at the code
>> you'll notice that sessions may be switched in mid-stream (e.g. we have
>> an HTTP/1.1 client that makes multiple requests, which may get split to
>> separate back-ends; some of the requests would "follow" the old
>> sessions, some the new).
>>
> I don't think that this really matters.  If session splitting is 
> acceptable at all, then any semantics after a reconfiguration would by 
> just as acceptable.

Seems fine to me as well.

Maurice