Pound

POUND - REVERSE-PROXY AND LOAD-BALANCER

The Pound program is a reverse proxy, load balancer and HTTPS front-end for Web server(s). Pound was developed to enable distributing the load among several Web-servers and to allow for a convenient SSL wrapper for those Web servers that do not offer it natively. Pound is distributed under the GPL - no warranty, it's free to use, copy and give away.

WHAT POUND IS:

  1. a reverse-proxy: it passes requests from client browsers to one or more back-end servers.

  2. a load balancer: it will distribute the requests from the client browsers among several back-end servers, while keeping session information.

  3. an SSL wrapper: Pound will decrypt HTTPS requests from client browsers and pass them as plain HTTP to the back-end servers.

  4. an HTTP/HTTPS sanitizer: Pound will verify requests for correctness and accept only well-formed ones.

  5. a fail over-server: should a back-end server fail, Pound will take note of the fact and stop passing requests to it until it recovers.

Pound is a very small program, easily audited for security problems. It can run as setuid/setgid and/or in a chroot jail. Pound does not access the hard-disk at all (except for reading certificate file(s) on start, if required) and should thus pose no security threat to any machine.

WHAT POUND IS NOT:

  1. Pound is not a Web server: by itself, Pound serves no content - it contacts the back-end server(s) for that purpose.

  2. Pound is not a Web accelerator: no caching is done - every request is passed "as is" to a back-end server.

STATUS

As of release 1.0 Pound is declared to be production-quality code.

Quite a few people have reported using Pound successfully in production environments. The largest volume reported to date is a site with an average of about 30M requests per day, peaking at over 600 requests/sec.

Pound was successfully used in production with a variety of Web servers, including Apache, IIS, Zope, WebLogic, Jakarta/Tomcat, iPlanet, etc. In general Pound passes requests and responses back and forth unchanged, so we have no reason to think that any web server would be incompatible.

Client browsers that were tested:

  • IE 5.0 and up (Windows) HTTP/HTTPS

  • Netscape 4.7 (Windows/Linux) HTTP/HTTPS

  • Firefox (Windows/Linux) HTTP/HTTPS

  • Konqueror (Linux) HTTP/HTTPS

  • Galleon (Linux) HTTP/HTTPS

  • Opera (Linux/Windows) HTTP/HTTPS

  • Lynx (Linux) HTTP

Given that Pound is in production and no problems were reported, we have no reason to believe that other browsers would present a problem. A few issues were observed with problematic SSL implementations, most notably with Opera 6, but these should be OK in the present version.

INSTALLATION

Probably the easiest way to install Pound is to use a pre-compiled package if you can find one. While Apsis offers no such packages, they are available for quite a few systems (Suse, Debian and derivatives such as Ubuntu), as well as some private packages.

Failing that you should install from sources.

Required packages:

  • the nanomsg library (available as a package on most distributions)
  • the libyaml library (available as a package on most distributions)
  • the mbedSSL library (the library previously known as PolarSSL, available as a package on most distributions)
  • a threads library (commonly libpthread)
  • the uthash library. A copy is included in the Pound distribution, but you may want to download the most up-to-date version. The copyright notice is here
  • the hpack library. A copy is included in the Pound distribution, with adjustments for Linux compatability. The copyright is held by Reyk Floeter (BSD license).

The uthash and hpack libraries are included as packages for them are usually not included in distributions.

Optional, but recommended packages:

  • tcmalloc (previously part of the Google performance tools, available as a package on most distributions)
  • pcreposix (available as a package on most distributions)

These packages will be automatically used by Pound if available.

Download the latest version Pound-3.0d.tgz file and unpack it. The archive is signed.

My signature is available here.

Alternately see below for stable versions.

Pound uses CMake for its build, so just do the usual:

cd build
cmake ..
make

COPYRIGHT

Pound is copyrighted by Apsis GmbH and is distributed under the terms of the GNU Public License. Basically, this means that you can use it free of charge, copy it, distribute it (provided the copyright is maintained and the full package is distributed), modify it, or line a bird-cage with it.

We would be happy to hear from you if you use it and suggestions and improvements are gladly accepted.

CONTACT

Robert Segall
Apsis GmbH
P O Box
CH-8707 Uetikon am See
Switzerland
+41-44-920 4904

MAILING LIST

Pound has its own mailing list now: go here to subscribe. You will receive confirmation and instructions in the reply.

All messages are available and indexed (searcheable) in the archive.

The mailing list is the primary support forum for Pound - please post there any questions you may have. The developpers' address is given here for information purposes only.

LOGO

We would be grateful if you would include the Pound logo Pound logo with a link to Apsis GmbH somewhere on your web page.

SESSIONS

Pound has the ability to keep track of sessions between a client browser and a back-end server. Unfortunately, HTTP is defined as a stateless protocol, which complicates matters: many schemes have been invented to allow keeping track of sessions, none of which works perfectly. Even worse, sessions are critical in order to allow web-based applications to function correctly - it is vital that once a session is established all subsequent requests from the same browser be directed to the same back-end server.

The way this is supported in Pound is by keeping track of the association between an originating IP address and a specific Backend. This is done at the Service level, and the directive Session defines for how long this association is kept.

A note on cookie injection: some applications have no session-tracking mechanism at all but would still like to have the client always directed to the same back-end time after time. Some reverse proxies use a mechanism called "cookie injection" in order to achieve this: a cookie is added to the back-end responses and tracked by the reverse proxy.

Pound was designed to be as transparent as possible, and this mechanism is not supported.

THREADS AND LIMITS

A few people ran into problems when installing Pound because of the various threading models and how they interact with system-imposed limits. Please keep in mind the following requirements:

  • on most System V derived Unices (of which Linux up to 2.4 is one), a thread is a process. This means that when doing a 'ps' you will see as many processes with the name 'pound' as there are active threads. Each such process uses only two file descriptors, but the system needs to support the required number of processes, both in total and per user (possibly also per process group). In bash, this is 'ulimit -u', in csh this is 'limit maxproc'.

  • on BSD style systems all threads run in the same process space. Do a ps and you see a single 'pound' process. The process needs two file descriptors per active request (bash: 'ulimit -n', csh 'limit maxfiles'/'limit openfiles').

  • on most systems the thread library comes with a built-in limit on the maximal number of concurrent threads allowed - on older systems it usually is 1024, on newer systems quite a bit higher. In very rare cases (very high load and long response times) you may run into this limitation - the symptom is log messages saying "can't create thread". Your only solution is to recompile the system threads library (and possibly the kernel itself) with a higher limit.

Please note that your kernel needs to be configured to support the required resources - the above are just the shell commands.

SIMILAR SYSTEMS

Quite a few people asked "What is wrong with Apache/Squid/ stunnel/your_favorite? Do we really need another proxy system?". The simple answer is that there is nothing wrong - they are all excellent systems that do their jobs very well. The reasoning behind Pound is however slightly different:

  • In my experience, a load-balancer may easily become a bottle-neck in itself. If you have a heavily loaded site, there are few things more depressing than seeing your "load-balancer" slow down the entire network. This means that the load-balancer should be kept as light-weight as possible.

  • Security: auditing a large system for security issues is a major undertaking for anybody (ask Bill Gates about it). This implies that in order to avoid introducing new vulnerabilities into a system (after all, your installation is only as secure as its weakest component) the proxy/load-balancer should be kept as small as possible.

  • Protection: I assume Pound will be the only component exposed to the Internet - your back-end servers will run in a protected network behind it. This means that Pound should filter requests and make sure only valid, correctly formed ones are passed to the back-end servers, thus protecting them from malicious clients.

Taking these criteria into consideration, it is easy to see why the other systems mentioned above do not fit:

  • Apache (with modproxy and modbackhand): great system, but very large. Imposes a significant load on the system, complex set-up procedure (and it is so easy to get it wrong: check how many Apache servers allow proxying from and to external hosts). While Apache has proven remarkably exploit free, I wouldn't wish to go into a security audit for the tens of thousands of lines of code involved, not to mention all the additional modules.

  • Squid: great caching proxy, but even should load-balancing features become available in the future, do you really need caching on the load-balancer? After all, Pound can easily run on a disk-less system, whereas with Squid you'd better prepare a high throughput RAID. Squid is still perfectly usable as a caching proxy between Pound and the actual Web server, should it lack its own cache (which Zope happily has).

  • stunnel: probably comes closest to my understanding of software design (does one job only and does it very well). However, it lacks the load balancing and HTTP filtering features that I considered necessary. Using stunnel in front of Pound (for HTTPS) would have made sense, except that integrating HTTPS into Pound proved to be so simple that it was not worth the trouble.

  • your favourite system: let me know how it looks in light of the above criteria - I am always interested in new ideas.

OTHER ISSUES

The following problems were reported by various people who use pound:

  • Pound fails to start; HTTPS is enabled and the message "can't read private key from file xxx" appears in the log.

    Solution: make sure that the certificate file includes:

    • (optional) a chain of certificates from a known certificate authority to your server certificate

    • the server certificate

    • the private key; the key may NOT be password-protected

    The file should be in PEM format. The OpenSSL command to generate a self-signed certificate in the correct format would be something like::

    openssl req -x509 -newkey rsa:1024 -keyout test.pem -out test.pem \ -days 365 -nodes

    Note the '-nodes' flag - it's important!

  • Pound fails to operate correctly with SSL when RootJail is specified. Solution: OpenSSL requires access to /dev/urandom, so make sure such a device is accessible from the root jail directory. Thus if your root jail is something like /var/pound:

    mkdir /var/pound/dev mknod /var/pound/dev/urandom c 1 9

    or whatever major/minor number are appropriate for your system.

  • In chroot mode logging may stop functioning. Solution: make sure /dev and the root jail are on the same filesystem and create a hard link in the root jail to /dev/log:

    mkdir /chroot/jail/dev ln /dev/log /chroot/jail/dev/log

    Alternately you can have syslog (or syslog-ng) listen on another socket - see the man page for details.

  • In chroot mode name resolution (and especially redirects) may stop functioning. Solution: make sure your resolver works correctly in the jail. You probably need copies of /etc/resolv.conf and (at least part) of /etc/hosts. Depending on your system additional files may be required check your resolver man page for details. Should name resolution fail the translation of host names to IP addresses would fail, thereby defeating the mechanism Pound uses to identify when should a Redirect be rewritten.

  • Linux-specific: some people use various redundant Pound solutions for Linux which require Pound instances on separate machines to bind to the same address. The default configuration of Linux does not allow a program to bind() to non-local addresses, which may cause a problem. Solution: add

    echo 1 > /proc/sys/net/ipv4/ipnonlocalbind

    in your start-up script, or just set

    net.ipv4.ipnonlocalbind = 1

    in /etc/sysctl.conf (if you have one).

    (Thanks to Rune Saetre for the suggestion).

ACKNOWLEDGMENTS

  • Albert (of Alacra) for investigating and writing the TCP_NODELAY code.

  • Luuk de Boer did some serious testing and debugging of the WebDAV code for Microsoft servers.

  • Alession Cervellin packages and makes available Solaris packages for various Pound versions.

  • David Couture found some nasty, lurking bugs, as well as contributing some serious testing on big hardware.

  • Frank Denis contributed a few excellent code patches and some good ideas.

  • Dmitriy Dvoinikov makes available a live-CD FreeBSD distribution that includes a Pound binary.

  • Abner G. Jacobsen did a lot of testing in a production environment and contributed some very nice ideas.

  • Akira Higuchi found a significant security issue in Pound and contributed the code to fix it.

  • Ken Lalonde contributed very useful remarks and suggestions, as well as correcting a few code errors.

  • Phil Lodwick contributed essential parts of the high-availability code and came up with some good ideas. In addition, did some serious testing under heavy loads.

  • Simon Matter packages and makes available RPMs for various Pound versions.

  • Jan-Piet Mens raised some interesting security points about the HTTPS implementation and brought the original idea for SSL header filtering.

  • Andreas Roedl for testing and some ideas about logging in root jails.

  • Gurkan Sengun tested Pound on Solaris, contributed the Solaris cc flags and makes a Solaris pre-compiled version available on his Web-site (www.linuks.mine.nu)

  • Shinji Tanaka contributed a patch for controlling logging to disk files. This is available at http://www.hatena-inc.co.jp/~stanaka/pound/

  • Jim Washington contributed the code for WebDAV and tested it.

  • Maxime Yve discovered a nasty bug in the session tracking code and contributed the patch to fix it.

All the others who tested Pound and told me about their results.

STABLE VERSIONS

EXPERIMENTAL VERSIONS