POUND - REVERSE-PROXY AND LOAD-BALANCER
The Pound program is a reverse proxy, load balancer and
HTTPS front-end for Web server(s). Pound was developed
to enable distributing the load among several Web-servers
and to allow for a convenient SSL wrapper for those Web
servers that do not offer it natively. Pound is distributed
under the GPL - no warranty, it's free to use, copy and
give away.
WHAT POUND IS:
a reverse-proxy: it passes requests from client
browsers to one or more back-end servers.
a load balancer: it will distribute the requests from
the client browsers among several back-end servers,
while keeping session information.
an SSL wrapper: Pound will decrypt HTTPS requests
from client browsers and pass them as plain HTTP
to the back-end servers.
an HTTP/HTTPS sanitizer: Pound will verify requests
for correctness and accept only well-formed ones.
a fail over-server: should a back-end server fail,
Pound will take note of the fact and stop passing
requests to it until it recovers.
Pound is a very small program, easily audited for security
problems. It can run as setuid/setgid and/or in a chroot
jail. Pound does not access the hard-disk at all (except
for reading certificate file(s) on start, if required)
and should thus pose no security threat to any machine.
WHAT POUND IS NOT:
Pound is not a Web server: by itself, Pound serves no
content - it contacts the back-end server(s) for that
purpose.
Pound is not a Web accelerator: no caching is done -
every request is passed "as is" to a back-end server.
STATUS
As of release 1.0 Pound is declared to be production-quality code.
Quite a few people have reported using Pound successfully in production
environments. The largest volume reported to date is a site with an
average of about 30M requests per day, peaking at over 600 requests/sec.
Pound was successfully used in production with a variety of Web servers,
including Apache, IIS, Zope, WebLogic, Jakarta/Tomcat, iPlanet, etc. In
general Pound passes requests and responses back and forth unchanged,
so we have no reason to think that any web server would be incompatible.
Client browsers that were tested:
IE 5.0 and up (Windows) HTTP/HTTPS
Netscape 4.7 (Windows/Linux) HTTP/HTTPS
Firefox (Windows/Linux) HTTP/HTTPS
Konqueror (Linux) HTTP/HTTPS
Galleon (Linux) HTTP/HTTPS
Opera (Linux/Windows) HTTP/HTTPS
Lynx (Linux) HTTP
Given that Pound is in production and no problems were reported, we have
no reason to believe that other browsers would present a problem. A few
issues were observed with problematic SSL implementations, most notably
with Opera 6, but these should be OK in the present version.
INSTALLATION
Probably the easiest way to install Pound is to use a pre-compiled package
if you can find one. While Apsis offers no such packages, they are available
for quite a few systems (Suse, Debian and derivatives such as Ubuntu), as
well as some private packages.
Failing that you should install from sources.
Required packages:
- the nanomsg library (available as a package on most distributions)
- the libyaml library (available as a package on most distributions)
- the mbedSSL library (the library previously known as PolarSSL, available as a package on most distributions)
- a threads library (commonly libpthread)
- the uthash library. A copy is included in the Pound distribution, but you
may want to download the most up-to-date version. The copyright notice is here
- the hpack library. A copy is included in the Pound distribution, with adjustments for Linux compatability.
The copyright is held by Reyk Floeter (BSD license).
The uthash and hpack libraries are included as packages for them are usually not included in distributions.
Optional, but recommended packages:
- tcmalloc (previously part of the Google performance tools, available as a package on most distributions)
- pcreposix (available as a package on most distributions)
These packages will be automatically used by Pound if available.
Download the latest version Pound-3.0.tgz file
and unpack it. The archive is signed.
My signature is available here.
Alternately see below for stable versions.
Pound uses CMake for its build, so just do the usual:
cd build
cmake ..
make
COPYRIGHT
Pound is copyrighted by Apsis GmbH and is distributed under
the terms of the GNU Public License. Basically, this means that you can
use it free of charge, copy it, distribute it (provided the copyright is
maintained and the full package is distributed), modify it,
or line a bird-cage with it.
We would be happy to hear from you if you use it and
suggestions and improvements are gladly accepted.
CONTACT
Robert Segall
Apsis GmbH
P O Box
CH-8707 Uetikon am See
Switzerland
+41-44-920 4904
MAILING LIST
Pound has its own mailing list now: go here
to subscribe. You will receive confirmation and instructions in the reply.
All messages are available and indexed (searcheable) in the
archive.
The mailing list is the primary support forum for Pound - please
post there any questions you may have. The developpers' address is
given here for information purposes only.
LOGO
We would be grateful if you would include the Pound logo
with a link to
Apsis GmbH somewhere on your web page.
SESSIONS
Pound has the ability to keep track of sessions between a client
browser and a back-end server. Unfortunately, HTTP is defined as
a stateless protocol, which complicates matters: many schemes have
been invented to allow keeping track of sessions, none of which works
perfectly. Even worse, sessions are critical in order to allow
web-based applications to function correctly - it is vital that once
a session is established all subsequent requests from the same browser
be directed to the same back-end server.
The way this is supported in Pound is by keeping track of the association
between an originating IP address and a specific Backend. This is done
at the Service level, and the directive Session defines for how long this
association is kept.
A note on cookie injection: some applications have no session-tracking mechanism at
all but would still like to have the client always directed to the same back-end
time after time. Some reverse proxies use a mechanism called "cookie injection" in
order to achieve this: a cookie is added to the back-end responses and tracked by the
reverse proxy.
Pound was designed to be as transparent as possible, and this mechanism is not
supported.
THREADS AND LIMITS
A few people ran into problems when installing Pound because of the
various threading models and how they interact with system-imposed
limits. Please keep in mind the following requirements:
on most System V derived Unices (of which Linux up to 2.4 is one),
a thread is a process. This means that when doing a 'ps' you will see
as many processes with the name 'pound' as there are active threads.
Each such process uses only two file descriptors, but the system needs
to support the required number of processes, both in total and per
user (possibly also per process group). In bash, this is 'ulimit -u',
in csh this is 'limit maxproc'.
on BSD style systems all threads run in the same process space. Do
a ps and you see a single 'pound' process. The process needs two
file descriptors per active request (bash: 'ulimit -n', csh
'limit maxfiles'/'limit openfiles').
on most systems the thread library comes with a built-in limit on the
maximal number of concurrent threads allowed - on older systems it usually
is 1024, on newer systems quite a bit higher. In very
rare cases (very high load and long response times) you may run into
this limitation - the symptom is log messages saying "can't create
thread". Your only solution is to recompile the system threads library
(and possibly the kernel itself) with a higher limit.
Please note that your kernel needs to be configured to support the
required resources - the above are just the shell commands.
SIMILAR SYSTEMS
Quite a few people asked "What is wrong with Apache/Squid/
stunnel/your_favorite? Do we really need another proxy
system?". The simple answer is that there is nothing wrong -
they are all excellent systems that do their jobs very well.
The reasoning behind Pound is however slightly different:
In my experience, a load-balancer may easily become a
bottle-neck in itself. If you have a heavily loaded site,
there are few things more depressing than seeing your
"load-balancer" slow down the entire network. This means that
the load-balancer should be kept as light-weight as possible.
Security: auditing a large system for security issues is a
major undertaking for anybody (ask Bill Gates about it). This
implies that in order to avoid introducing new vulnerabilities
into a system (after all, your installation is only as secure
as its weakest component) the proxy/load-balancer should be
kept as small as possible.
Protection: I assume Pound will be the only component exposed
to the Internet - your back-end servers will run in a protected
network behind it. This means that Pound should filter requests
and make sure only valid, correctly formed ones are passed to the
back-end servers, thus protecting them from malicious clients.
Taking these criteria into consideration, it is easy to see why
the other systems mentioned above do not fit:
Apache (with modproxy and modbackhand): great system, but very
large. Imposes a significant load on the system, complex set-up
procedure (and it is so easy to get it wrong: check how many Apache
servers allow proxying from and to external hosts). While Apache
has proven remarkably exploit free, I wouldn't wish to go into a
security audit for the tens of thousands of lines of code involved,
not to mention all the additional modules.
Squid: great caching proxy, but even should load-balancing
features become available in the future, do you really need
caching on the load-balancer? After all, Pound can easily run on a
disk-less system, whereas with Squid you'd better prepare a high
throughput RAID. Squid is still perfectly usable as a caching
proxy between Pound and the actual Web server, should it lack
its own cache (which Zope happily has).
stunnel: probably comes closest to my understanding of software
design (does one job only and does it very well). However, it
lacks the load balancing and HTTP filtering features that I
considered necessary. Using stunnel in front of Pound (for HTTPS)
would have made sense, except that integrating HTTPS into Pound
proved to be so simple that it was not worth the trouble.
your favourite system: let me know how it looks in light of the
above criteria - I am always interested in new ideas.
OTHER ISSUES
The following problems were reported by various people who use pound:
Pound fails to start; HTTPS is enabled and the message "can't read
private key from file xxx" appears in the log.
Solution: make sure that the certificate file includes:
The file should be in PEM format. The OpenSSL command to generate a
self-signed certificate in the correct format would be something like::
openssl req -x509 -newkey rsa:1024 -keyout test.pem -out test.pem \
-days 365 -nodes
Note the '-nodes' flag - it's important!
Pound fails to operate correctly with SSL when RootJail is specified.
Solution: OpenSSL requires access to /dev/urandom, so make sure such a
device is accessible from the root jail directory. Thus if your root
jail is something like /var/pound:
mkdir /var/pound/dev
mknod /var/pound/dev/urandom c 1 9
or whatever major/minor number are appropriate for your system.
In chroot mode logging may stop functioning.
Solution: make sure /dev and the root jail are on the same filesystem
and create a hard link in the root jail to /dev/log:
mkdir /chroot/jail/dev
ln /dev/log /chroot/jail/dev/log
Alternately you can have syslog (or syslog-ng) listen on another
socket - see the man page for details.
In chroot mode name resolution (and especially redirects) may stop
functioning. Solution: make sure your resolver works correctly in the
jail. You probably need copies of /etc/resolv.conf and (at least part)
of /etc/hosts. Depending on your system additional files may be required
check your resolver man page for details. Should name resolution fail the
translation of host names to IP addresses would fail, thereby defeating
the mechanism Pound uses to identify when should a Redirect be rewritten.
Linux-specific: some people use various redundant Pound solutions for
Linux which require Pound instances on separate machines to bind to the
same address. The default configuration of Linux does not allow a
program to bind() to non-local addresses, which may cause a problem.
Solution: add
echo 1 > /proc/sys/net/ipv4/ipnonlocalbind
in your start-up script, or just set
net.ipv4.ipnonlocalbind = 1
in /etc/sysctl.conf (if you have one).
(Thanks to Rune Saetre for the suggestion).
ACKNOWLEDGMENTS
Albert (of Alacra) for investigating and writing the TCP_NODELAY code.
Luuk de Boer did some serious testing and debugging of the WebDAV
code for Microsoft servers.
Alession Cervellin packages and makes available Solaris packages for
various Pound versions.
David Couture found some nasty, lurking bugs, as well as contributing
some serious testing on big hardware.
Frank Denis contributed a few excellent code patches and some good ideas.
Dmitriy Dvoinikov makes available a live-CD FreeBSD distribution that
includes a Pound binary.
Abner G. Jacobsen did a lot of testing in a production environment
and contributed some very nice ideas.
Akira Higuchi found a significant security issue in Pound and contributed
the code to fix it.
Ken Lalonde contributed very useful remarks and suggestions, as well as
correcting a few code errors.
Phil Lodwick contributed essential parts of the high-availability code and
came up with some good ideas. In addition, did some serious testing under
heavy loads.
Simon Matter packages and makes available RPMs for various Pound versions.
Jan-Piet Mens raised some interesting security points about the HTTPS
implementation and brought the original idea for SSL header filtering.
Andreas Roedl for testing and some ideas about logging in root jails.
Gurkan Sengun tested Pound on Solaris, contributed the Solaris cc flags
and makes a Solaris pre-compiled version available on his Web-site
(www.linuks.mine.nu)
Shinji Tanaka contributed a patch for controlling logging to disk files.
This is available at http://www.hatena-inc.co.jp/~stanaka/pound/
Jim Washington contributed the code for WebDAV and tested it.
Maxime Yve discovered a nasty bug in the session tracking code and
contributed the patch to fix it.
All the others who tested Pound and told me about their results.
STABLE VERSIONS
EXPERIMENTAL VERSIONS