We're currently using a spread/wackamole solution for high availability &
load balancing. We've recently got bitten by the arp issue with that "ip
shuffling approach" as our PIX that the webservers are behind ignores the
arps of the moved IP's. Unless you set the arp timeout to 60, and we don't
care for the possibility of 60 seconds of downtime (especially when the
webservers are rsynced and bounced in rapid sucession for new code often).
So we looked at hardware loadbalancers, and after the sticker shock decided
to look at software solutions. From my reading on your website, pound looks
like a great solution. I have several exploratory questions I didn't quite
see addressed in the FAQ or archives and I was hoping folks could clue me in
on them (as well as this email being a trial balloon to see if the support
response is existant).

1) We can't have the pound load balancer be a single point of failure. It
appears the commonly suggested solution from the archives is to use
Heartbeat from the linux HA project. We are strictly a FreeBSD shop. Is this
the most common approach to address the SPOF issue? I saw some comments that
pound will never offer HA, but that seemed to be with respect to the backend
servers, and we're looking at the pound box itself. Any thoughts?

2) I don't see anything on the algorithm that pound uses to balance the load
across the backend webservers. I'm assuming it just does a sequential or
round-robin approach? Has there been any discussion or plans for any other
load balancing strategy? I was most impressed by the strategies offered by
SGcluster but that package doesn't include source and doesn't appear to be
maintained. I'm thinking of having a small process run on each webserver
that sends load information (cpu, packets per second, memory use, latency,
etc.) to the load balancer and it would distribute load based on "least
busy" server. Any new thoughts on this since the last message I saw in the
archive which simply said "no"? If it was based on load you wouldn't have to
worry about disparaties in machine performance within the cluster.

3) Perhaps I'm missing something obvious here with regards to routing. We
have public IP's on the outside of the PIX, webservers on the DMZ using
private address space, and database servers on the inside interface on
different private address space. Does a pound server have to be multihomed
and talk on two separate interfaces? I'm not sure how to fit a pound server
into that, without creating an additional subnet, which seems quite messy.
Any thoughts?

4) I was also toying with implementing mod_backhand, which people on this
list may have messed with. It will handle the balancing based on load,
making the pound box more of a high availability device than load balancing.
Has anyone found good pros or cons for implementing both a pound box and
something like mod_backhand?

I realize the above is a long email, thanks so much for reading it and I
would greatly appreciate any comments or thoughts folks here may have on
these topics.

Regards,

Jay West
Web/Network Administrator
Knight's Direct

---
[This E-mail scanned for viruses by Declude Virus]

On Friday 21 May 2004 22.41, Jay West wrote:
> We're currently using a spread/wackamole solution for high availability &
> load balancing. We've recently got bitten by the arp issue with that "ip
> shuffling approach" as our PIX that the webservers are behind ignores the
> arps of the moved IP's. Unless you set the arp timeout to 60, and we don't
> care for the possibility of 60 seconds of downtime (especially when the
> webservers are rsynced and bounced in rapid sucession for new code often).
> So we looked at hardware loadbalancers, and after the sticker shock decided
> to look at software solutions. From my reading on your website, pound looks
> like a great solution. I have several exploratory questions I didn't quite
> see addressed in the FAQ or archives and I was hoping folks could clue me
> in on them (as well as this email being a trial balloon to see if the
> support response is existant).

You'll get responses, but in final instance you get what you pay for. If 60 
seconds downtime is not acceptable I suspect you need paid support 
(preferably in-house, or at least on-site, or at least in your time-zone). 
People on this list answer quite often, but this is still voluntary.

> 1) We can't have the pound load balancer be a single point of failure. It
> appears the commonly suggested solution from the archives is to use
> Heartbeat from the linux HA project. We are strictly a FreeBSD shop. Is
> this the most common approach to address the SPOF issue? I saw some
> comments that pound will never offer HA, but that seemed to be with respect
> to the backend servers, and we're looking at the pound box itself. Any
> thoughts?

Use anything that satisfies your needs (we even did this with some shell 
scripting - it's really quite easy). Pound itself does not support redundancy 
- this is best done at the system level.

> 2) I don't see anything on the algorithm that pound uses to balance the
> load across the backend webservers. I'm assuming it just does a sequential
> or round-robin approach? Has there been any discussion or plans for any
> other load balancing strategy? I was most impressed by the strategies
> offered by SGcluster but that package doesn't include source and doesn't
> appear to be maintained. I'm thinking of having a small process run on each
> webserver that sends load information (cpu, packets per second, memory use,
> latency, etc.) to the load balancer and it would distribute load based on
> "least busy" server. Any new thoughts on this since the last message I saw
> in the archive which simply said "no"? If it was based on load you wouldn't
> have to worry about disparaties in machine performance within the cluster.

Pound uses a random algorithm for distributing requests (uniform weighted). It 
was a design decision to use static balancing in order to avoid the extra 
complications of queue management and/or load measurement on the remote 
machines. If you really want to go that way you have a host of alternatives 
to choose from.

> 3) Perhaps I'm missing something obvious here with regards to routing. We
> have public IP's on the outside of the PIX, webservers on the DMZ using
> private address space, and database servers on the inside interface on
> different private address space. Does a pound server have to be multihomed
> and talk on two separate interfaces? I'm not sure how to fit a pound server
> into that, without creating an additional subnet, which seems quite messy.
> Any thoughts?

Pound does not need anything like that. As long as it can talk to the 
back-ends it does not know or care how the packets get there.

> 4) I was also toying with implementing mod_backhand, which people on this
> list may have messed with. It will handle the balancing based on load,
> making the pound box more of a high availability device than load
> balancing. Has anyone found good pros or cons for implementing both a pound
> box and something like mod_backhand?

I'll leave this to others. For me it is a question of code and installation 
complexity.

> I realize the above is a long email, thanks so much for reading it and I
> would greatly appreciate any comments or thoughts folks here may have on
> these topics.

-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904

On Monday 24 May 2004 14.59, you wrote:
> Robert wrote...
>
> > You'll get responses, but in final instance you get what you pay for. If
>
> 60
>
> > seconds downtime is not acceptable I suspect you need paid support
> > (preferably in-house, or at least on-site, or at least in your
> > time-zone). People on this list answer quite often, but this is still
> > voluntary.
>
> I wasn't referring to needing 60 seconds response to questions on the
> mailing list *grin*. I was referring to the time the old software
> (spread/wackamole) took to effect a failover due to arp cache timeout.
> Separately, I just wanted to make sure there was SOME level of activity on
> the support list, and there is. Fantastic!
>
> re: pound server failover
>
> > Use anything that satisfies your needs (we even did this with some shell
> > scripting - it's really quite easy). Pound itself does not support
>
> redundancy
>
> > - this is best done at the system level.
>
> Ok, I'll look at implementing Heartbeat from the linux HA project for this,
> or cobbling together some scripts.
>
> > Pound uses a random algorithm for distributing requests (uniform
>
> weighted). It
>
> > was a design decision to use static balancing in order to avoid the extra
> > complications of queue management and/or load measurement on the remote
> > machines. If you really want to go that way you have a host of
>
> alternatives
>
> > to choose from.
>
> No problem. I'll ponder integrating mod_backhand into the picture as an
> add-on to the pound server setup for this at some point in the future. Or
> maybe I'll take the time to create backend agents to pass load information
> to pound and adapt the balancing code. For now, all the webservers are
> identical hardware so this isn't a big issue, was just curious about future
> ideas. I'd love to hear any suggestions from people on what some of the
> alternatives available might be (not alternatives to pound, but
> alternatives to balance based on load such as mod_backhand).

I repeat: not using queue/load based balancing was a design decision - we just 
don't want the extra complexity. The Squid add-ons or mod_backhand (to take 
just a few examples) do a fine job in that respect.

> re: multihomed
>
> > Pound does not need anything like that. As long as it can talk to the
> > back-ends it does not know or care how the packets get there.
>
> Excellent!
>
> Lastly - I didn't see this on the website - is there a central repository
> for user contributed patches, add-ons, etc. for pound? Not saying it's
> necessary, just wanted to know if such a thing was around to peruse.

Not really. We either integrate the changes in the Pound code (quite often), 
or put a pointer to user-contributed patches on the Pound Web page.

> Thanks *very* much for the input!!! Pound looks most impressive and
> capable. We'll start setting up a testbed on the development side and give
> it a whirl.

Please make sure in the future you reply to the list and not to the 
contributors personally.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904