(I sent this earlier, but it didn't seem to appear. This email should be 
considered a "precursor" to the email I've just sent!)


I'm looking at reverse proxy software for load balancing/failover of web 
traffic.

Pound is almost ideal - particularly because it can handle virtual hosts 
(although I know the website recommends against that) - but one show 
stopping aspect for me is the fairly basic config reloading facility - 
whereby we would just have to reload the thing in a fairly brutal manner 
(losing any current connections etc.)

So I saw people recommending HaProxy, which has kinder config reloading, 
but it doesn't do virtual hosting. (i.e. I can't redirect requests to 
different backend servers based on the incoming host name - or at least, 
I've not worked out a way to do it, in HaProxy itself...).

So I was wondering if anyone has worked out a way to get around the 
fairly abrupt nature of config reloads in Pound? By using CARP, or 
anything else? I also saw the idea of running a pound instance in front 
of two back pount instances, and reconfiguring one while other is 'live' 
- anyone tried this?

Any ideas anyone?
thanks!
Alex

Alex Hunsley wrote:[...]

One way to retain all the current sessions when you restart pound:
1) Drain all the connections from all but one of the backends
2) Wait for all of the sessions attached to other backends to expire
3) Restart pound, and immediately make the same backends inactive again
4) Wait for the session timeout period - all the active sessions are
forced to re-establish themselves with the original server, because it's
the only one active (any sessions that are not re-established within
this period are invalid anyway)
5) Re-activate the rest of the backends

It's not ideal, but it works. It'd be better if pound could simply dump
its current sessions to disk, and attempt to reload them (assuming the
config fiule hadn't changed significantly in the meantime) on startup.

Regards,

Russ
[...]

Hi

I wrote a graceful exit-patch for Pound-1.10 once. What it does is 
that when pound receives one of the TERM, INT, QUIT and HUP signals it 
shuts down the listening ports straight away, but waits for a configurable 
amount of time before exiting, and thus killing open tcp sessions.
The restart script fires up a new pound instance as soon as the old one is 
signalled to quit. There might be a millisecond or so of unavailability.

Using this with (in pound-1.10 config language):
   "Session IP -1"
makes the target server calculation static, so the new pound instance will 
send the clients to the same servers as the old one.

Perhaps something like this could be implemented in Pound-2.*?

The patch can be found here:
http://folesvaert.no/pound/patches/patch-pound-1.10-graceful_exit_20070313.diff.gz

The supervisor process must be disabled at compile time for the patch to 
work.

Regards
Rune

---
Rune Sætre <rune.saetre(at)netcom-gsm.no>
NetCom as
..

On Thu, 13 Sep 2007, Alex Hunsley wrote:
[...]

Alex Hunsley wrote:[...]

Hello Alex.

I'll probably tell you something you'll not want to hear, but trust me, 
that it's experience talking.

If it's even remotely possible, the best thing you can do, is make your 
application NOT dependent on sticking to a specific backend server. It's 
just not what pound and even HTTP, for that matter, was designed for.

We've been there. We took every precaution possible to ensure the client 
stays on one server for the duration of the session and spent half a 
year fixing and analyzing problems, when the switch happened anyway. We 
then redesigned the application, so that it stores all session data on 
shared resource (database in our case, but can be done in other ways). 
Now, we don't care, where the request goes. Maintenance is a breeze, we 
can remove/add backends at will, even reload pound to our heart's content.

Just my 2 cents.
[...]

Another option .. probably not applicable to many of you is to have 
session management managed centrally for your web applications. We have 
a session management server our web servers access, it adds a little 
overhead to the system but the benefit is sessions are evenly 
distributed across the cluster. Pound is configured to round robin the 
connections across the cluster. Restarting/reloading pound has no effect 
on our end users except for the brief moment it takes to stop and start it.

Russell Odom wrote:[...][...][...]

[...]

Jimmy Brake wrote:[...]

Agreed - it takes a huge amount of pain out of load balancing!

ASP.NET has 2 methods for sharing session state between servers (in your
web.config <sessionState mode="SQLServer" ... /> or "StateServer" -
instead of "InProc"). I also heard there is a Postgres session provider
for ASP.NET, but haven't got round to researching it yet - does anyone
have experience of this?

Just another gotcha for ASP.NET in a multi-server environment - make
sure you give all the servers the same machine key (in your web.config:
<machineKey validationKey="..." />), otherwise viewstate generated on
one machine cannot be decrypted by another machine in the cluster.

Regards,

Russ
[...]

On Thu, 2007-09-13 at 16:28 +0200, Rune Saetre wrote:[...]

A similar mechanism is implemented in the latest 2.x series via the
Grace keyword. See the man page for details. The idea for the change is
credited to you.[...]

Hi

Dammit, I should have my head examined! Or better still, replaced!
Writing this mail felt strangely familiar, but I couldn't figure out why..
I am sorry about that.

Adding sticky sessions (like in pound-1.10) for Session Type IP would 
seemingly solve Alex Hunsleys problem.

I guess a suitably motivated scripter could dump the session table from 
the old pound instance and insert them into the new one using poundctl as 
well, given that the config hasn't changed too much. That wouldn't require 
any changes to pound.

Regards
Rune

---
Rune Sætre <rune.saetre(at)netcom-gsm.no>
NetCom as
..

On Thu, 13 Sep 2007, Robert Segall wrote:
[...][...][...]

Rune Saetre wrote:[...]
Thanks for all your replies - this has been a most useful discussion for me!

Rune, if by "sticky sessions for sessions type IP" you mean a certain IP 
would be hashed and then always hit the same backend (e.g. number 2) 
over any amount of sessions, then yes, that helps!

I just found this very interesting page, 
http://www.andrewtaylor.me.uk/gnu/pound/manual1.9.txt,
 which contains:

       Session TYPE [id] seconds
              Specify the time (in seconds) that a session will be kept.  May
be used exactly once and only in an UrlGroup directive. TYPE is
              one of:

              IP  -  the session is kept based on client IP address. No id is
allowed. The time may be negative, in which case Pound will use

              sticky sessions: instead of keeping a normal session the client
IP address is always mapped (hashed) to the same back-end.



However, I can't find this in the current Pound documentation - it's basically
what I need I think. I did actually try putting in a negative TTL in the
current pound release, but it spat and me and didn't like it....




[...]

Michal Taborsky - Internet Mall wrote:[...][...][...]
[snip]
Hi Michal
Thanks for your thoughtful message. You bring up a very good point, one 
which we are now considering here!
Alex
[...]

Alex Hunsley wrote:[...][...][...]
....and the same I found at 
http://www.penguin-soft.com/penguin/man/8/pound.html.
However, this is looking like an old version of Pound, and I can't see 
any way, or mention, of giving a negative 'time' (or TTL) for Session 
type IP in the current pound....
lex

Hi Alex

This is indeed from the man page for an older version of pound. The 
config file format changed in Pound-2, and it seems the sticky session 
concept disappeared as well.

Sticky sessions worked by calculating a hash on the source IP if the TTL 
was negative and Session type was set to IP. This would be really nice to 
have for Session Type IP and HEADER (and even for COOKIE, though this 
would require a dedicated persistence cookie assignment server to work 
properly).

I take it from your earlier mail that you are not intimidated by the 
prospect of implementing this yourself. It seems fairly straightforward to 
alter get_backend() in svc.c to calculate a number n from the IP 
address so that (n >= 0) and (n < svc->tot_pri). The call

   res = rand_backend(svc->backends, n);

will then always return the same backend as long as no backends health 
status changes or no backends are disabled or enabled. This works because 
rand_backend() does not call random() itself, but is called with a random 
priority. rand_backend() itself is entirely deterministic.

In addition a mechanism is of course needed to tell get_backend() that the 
static mapping should be used. The simplest way is probably to allow a TTL 
value of 0 for Session Type IP, and checking for (svc->sess_ttl==0) in 
get_backend(). Search for TTL and sess_ttl in config.c (and grep for it 
everywhere else to make sure using sess_ttl==0 is safe).

If you feel specially crafty one evening you can even write an alternative 
to rand_backend() that doesn't rehash all clients if backends are 
disabled, enabled or change health status, but only those ending up on the 
affected backends.

Roberts original algorithm from Pound-1.10 for calculating the magical n 
works quite well. The source is still available from Pounds website.
Incidentally it resided in my paste buffer, so here it is:

             /* "sticky" mappings */
             in_addr_t   t;

             t = from_host.s_addr;
             orig = 0;
             while(t) {
                 orig = (orig << 3) ^ (t & 0xff);
                 t = (t >> 8);
             orig = n = (orig & 0xffff) % g->tot_pri;

Variable names probably need adjusting, of course.

My paste buffer is in an exceptional generous mood tonight:

             int i = 0;
             orig  = 0;
             for (i=0; key[i] != '\0'; i++) {
                 orig = (orig << 3) ^ ((int)key[i] & 0xff);
             }
             orig = n = (orig & 0xffff) % g->tot_pri;

This is a variation of Roberts original code, using the text string 
key (e.g. the value of a header or a cookie) for the same calculations. I 
use it for header based sticky sessions in Pound-1.10.

Good luck to you if you choose to implement this!

Regards
Rune

---
Rune Sætre <rune.saetre(at)netcom-gsm.no>
NetCom as
..

On Thu, 13 Sep 2007, Alex Hunsley wrote:
[...][...]
>>> Hi
>>> 
>>> Dammit, I should have my head examined! Or better still, replaced!
>>> Writing this mail felt strangely familiar, but I couldn't figure
out why..
>>> I am sorry about that.
>>> 
>>> Adding sticky sessions (like in pound-1.10) for Session Type IP
would 
>>> seemingly solve Alex Hunsleys problem.
>>> [...][...]