A rather strange problem I'm having.  Searching through the  
archives I see this exact question was asked back in February but got  
no responses.  The issue is that the throughput by a server running  
Pound falls flat, and generates failed requests on the magnitude of  
70-80% by Apache bench, even when performing a request as small as 50  
with 1 concurrent user.  If I bypass Pound and direct the requests  
straight to the back end web server, failed requests drop to zero.   
Necessary background info would be that the OS is FreeBSD 6.1-RC2,  
that I get this problem with pound 1.9 out of ports, and that it  
persists even with 2.0.4 built against a separate OpenSSL with  
threading enabled.  Since I'd prefer to use 2.x, here's the current  
config:

User "www"
Group "www"
LogLevel 2

ListenHTTP
   Address 10.10.10.32
   Port    8080
End

    Service
       BackEnd
           Address 10.10.10.35
           Port    80
       End
       BackEnd
           Address 10.10.10.36
           Port    80
       End
    End

      Of note is that despite LogLevel 2, no complaints are showing  
up in /var/log/messages.  It logs the startup notification and  
nothing else so I don't have anything useful to include from that at  
the moment.  Here's some ab info to illustrate what's going on:

With Pound 2.0.4:

[ stl(at)grimes : ~ ] $ ab -k -n 50 -c 1 http://192.168.0.205/
...
Concurrency Level:      1
Time taken for tests:   0.188027 seconds
Complete requests:      50
Failed requests:        40
Requests per second:    265.92 [#/sec] (mean)
...

Without Pound 2.0.4:

[ stl(at)grimes : ~ ] $ ab -k -n 50 -c 1 http://192.168.0.205/
...
Concurrency Level:      1
Time taken for tests:   0.45350 seconds
Complete requests:      50
Failed requests:        0
Requests per second:    1102.54 [#/sec] (mean)
...


-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com

You say something about being out of ports with 1.9.  That seems 
impossible if you are sending 50 requests with no request concurrency.

What do the server logs say when you are using pound?

Also, what kind of transactions are these?  Is there a possibility that 
pound is rejecting many of them as ill-formed?

Scott Larson wrote:
>      A rather strange problem I'm having.  Searching through the 
> archives I see this exact question was asked back in February but got 
> no responses.  The issue is that the throughput by a server running 
> Pound falls flat, and generates failed requests on the magnitude of 
> 70-80% by Apache bench, even when performing a request as small as 50 
> with 1 concurrent user.  If I bypass Pound and direct the requests 
> straight to the back end web server, failed requests drop to zero.  
> Necessary background info would be that the OS is FreeBSD 6.1-RC2, 
> that I get this problem with pound 1.9 out of ports, and that it 
> persists even with 2.0.4 built against a separate OpenSSL with 
> threading enabled.  Since I'd prefer to use 2.x, here's the current 
> config:
>
> User "www"
> Group "www"
> LogLevel 2
>
> ListenHTTP
>   Address 10.10.10.32
>   Port    8080
> End
>
>    Service
>       BackEnd
>           Address 10.10.10.35
>           Port    80
>       End
>       BackEnd
>           Address 10.10.10.36
>           Port    80
>       End
>    End
>
>      Of note is that despite LogLevel 2, no complaints are showing up 
> in /var/log/messages.  It logs the startup notification and nothing 
> else so I don't have anything useful to include from that at the 
> moment.  Here's some ab info to illustrate what's going on:
>
> With Pound 2.0.4:
>
> [ stl(at)grimes : ~ ] $ ab -k -n 50 -c 1 http://192.168.0.205/
> ...
> Concurrency Level:      1
> Time taken for tests:   0.188027 seconds
> Complete requests:      50
> Failed requests:        40
> Requests per second:    265.92 [#/sec] (mean)
> ...
>
> Without Pound 2.0.4:
>
> [ stl(at)grimes : ~ ] $ ab -k -n 50 -c 1 http://192.168.0.205/
> ...
> Concurrency Level:      1
> Time taken for tests:   0.45350 seconds
> Complete requests:      50
> Failed requests:        0
> Requests per second:    1102.54 [#/sec] (mean)
> ...
>
>
> --Scott Larson
> Network Administrator
> IOWA Interactive
> 4212 Glencoe Ave
> Marina Del Rey, CA 90292
>
> t 310.823.8238
> f 310.823.7108
> stl(at)iowainteractive.com
> http://www.iowainteractive.com
> http://www.wiredrive.com
>
>
>
> --To unsubscribe send an email with subject 'unsubscribe' to 
> pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
> http://www.apsis.ch/pound/pound_list/archive/2006/2006-05/1146767005000
>
>
> --No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.1.392 / Virus Database: 268.5.3/331 - Release Date: 5/3/2006
>
>

-- 
Ted Dunning
Chief Scientist
Veoh Networks

      I initially tried 2.0.4 built by hand against a separate  
OpenSSL built with threading enabled, ran Apache bench against it  
which showed all the failed requests, so I then built the version out  
of ports which is 1.9 to see if that would give me any insight as to  
what was happening.  1.9 out of ports generated the same issue  
however.  By ports, I'm referring to the FreeBSD ports/packages build  
system, not an OS service port.
      The Pound process isn't logging anything useful.  I've tried  
running it directly as root thinking that as user www it may be  
running into errors writing out, but still there was nothing despite  
LogLevel being set to 2.  It logs the startup and that is all.  As  
for what gets logged on the web server for the client request, it  
looks like this:

192.168.0.5 - - [04/May/2006:11:14:39 -0700] "GET / HTTP/1.0" 200 17  
"-" "ApacheBench/2.0.41-dev"

      The transaction is very simple, ApacheBench is just getting a  
one line text file.  It's about as unsophisticated and basic as can  
be.  As it just makes the same request over and over, I would think  
that if ApacheBench were making a malformed request they should all  
fail.  The request path of my test setup looks like this:

With Pound:

FreeBSD client -> FreeBSD firewall -> FreeBSD Pound server -> FreeBSD  
Apache server

Without Pound:

FreeBSD client -> FreeBSD firewall -> FreeBSD Apache server

      With the possibility of the dedicated Pound server being a pile  
of crap, I then moved Pound 2.0.4 directly onto the firewall machine,  
which then made the request path look like this:

FreeBSD client -> FreeBSD firewall/Pound -> FreeBSD Apache server

      Still, 70-80% failed request rate.  It would seem that Pound  
and FreeBSD are not getting along at this point.

On May 4, 2006, at 11:45 AM, Ted Dunning wrote:

>
> You say something about being out of ports with 1.9.  That seems  
> impossible if you are sending 50 requests with no request concurrency.
>
> What do the server logs say when you are using pound?
>
> Also, what kind of transactions are these?  Is there a possibility  
> that pound is rejecting many of them as ill-formed?

-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com

On Thu, 2006-05-04 at 11:23 -0700, Scott Larson wrote:
>       A rather strange problem I'm having.  Searching through the  
> archives I see this exact question was asked back in February but got  
> no responses.  The issue is that the throughput by a server running  
> Pound falls flat, and generates failed requests on the magnitude of  
> 70-80% by Apache bench, even when performing a request as small as 50  
> with 1 concurrent user.  If I bypass Pound and direct the requests  
> straight to the back end web server, failed requests drop to zero.   
> Necessary background info would be that the OS is FreeBSD 6.1-RC2,  
> that I get this problem with pound 1.9 out of ports, and that it  
> persists even with 2.0.4 built against a separate OpenSSL with  
> threading enabled.  Since I'd prefer to use 2.x, here's the current  
> config:
> 
> User "www"
> Group "www"
> LogLevel 2
> 
> ListenHTTP
>    Address 10.10.10.32
>    Port    8080
> End
> 
>     Service
>        BackEnd
>            Address 10.10.10.35
>            Port    80
>        End
>        BackEnd
>            Address 10.10.10.36
>            Port    80
>        End
>     End
> 
>       Of note is that despite LogLevel 2, no complaints are showing  
> up in /var/log/messages.  It logs the startup notification and  
> nothing else so I don't have anything useful to include from that at  
> the moment.  Here's some ab info to illustrate what's going on:
> 
> With Pound 2.0.4:
> 
> [ stl(at)grimes : ~ ] $ ab -k -n 50 -c 1 http://192.168.0.205/
> ...
> Concurrency Level:      1
> Time taken for tests:   0.188027 seconds
> Complete requests:      50
> Failed requests:        40
> Requests per second:    265.92 [#/sec] (mean)
> ...
> 
> Without Pound 2.0.4:
> 
> [ stl(at)grimes : ~ ] $ ab -k -n 50 -c 1 http://192.168.0.205/
> ...
> Concurrency Level:      1
> Time taken for tests:   0.45350 seconds
> Complete requests:      50
> Failed requests:        0
> Requests per second:    1102.54 [#/sec] (mean)
> ...

Something must show in the log files - at the very least the 50
requests, together with their status. You may want to try LogLevel 4 for
a more verbose display.

In any case I'd be curious to see what a "failed" request means: was
there no response, or a wrong response, or perhaps a 5xx response?

More info required...
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

      I've made some progress on this.  It appears that when Pound is  
left in round-robin mode, and it is spraying around connections to  
multiple backends, that is when all the failed connections are  
generated.  I enabled IP based sessions and my failures went away  
immediately.  When causing the failures the only clue as to what is  
going on in the log is this:

May  5 14:52:58 fencer pound: backend 10.10.10.36:80 connect:  
Operation not permitted

      Then it resurrects, and fails again.  It almost as if the  
backend is being overwhelmed, but I don't think that is the case.  If  
you look over the ApacheBench numbers from previous emails you'll see  
that when going through Pound, the requests per second is much lower  
than when going to the web server directly.  For comparison purposes,  
I loaded lighttpd on the backend to see if it was Apache now being  
the source of the problem, but it topped out at the same requests per  
second.
      To strip this test to the bare minimum, there are now only  
three things involved:  My 12" PowerBook, and matching Sempron 2200+  
servers, both running FBSD 6.1-RC2.  So here is the latest runs of AB  
using this scenario:

12" PB -> Apache 2.2.2 server
Server Software:        Apache/2.2.2
Server Hostname:        10.10.10.36
Server Port:            80

Document Path:          /
Document Length:        17 bytes

Concurrency Level:      10
Time taken for tests:   1.137 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      265265 bytes
HTML transferred:       17017 bytes
Requests per second:    879.51 [#/sec] (mean)
Time per request:       11.37 [ms] (mean)
Time per request:       1.14 [ms] (mean, across all concurrent requests)
Transfer rate:          233.30 [Kbytes/sec] received

Connnection Times (ms)
               min   avg   max
Connect:        0     0    18
Processing:     1    10    43
Total:          1    11    43


12" PB -> Pound 2.0.4 server -> Apache 2.2.2 server
Server Software:        Apache/2.2.2
Server Hostname:        10.10.10.35
Server Port:            8080

Document Path:          /
Document Length:        17 bytes

Concurrency Level:      10
Time taken for tests:   2.693 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      265265 bytes
HTML transferred:       17017 bytes
Requests per second:    371.33 [#/sec] (mean)
Time per request:       26.93 [ms] (mean)
Time per request:       2.69 [ms] (mean, across all concurrent requests)
Transfer rate:          98.50 [Kbytes/sec] received

Connnection Times (ms)
               min   avg   max
Connect:        0     0    19
Processing:    13    26    49
Total:         13    26    49

      The things to note this time is that the failed connections are  
non-existent, since all requests are being directed at the same  
server each time.  The other thing which jumps out is the halved  
requests per second and increased testing time.  So while I've worked  
around one issue it's left me with another.  For the hell of it I  
rebuilt the kernels to use the ULE scheduler rather than the old 4BSD  
but the difference in performance was impossible to detect.
      One other point to make is that the Sempron machines can  
actually output more than the ~800 requests per second.  The 800 mark  
is a limit imposed by my PowerBook, throwing a 2.6Ghz HP at it drives  
that number to over 1700.


On May 5, 2006, at 9:47 AM, Robert Segall wrote:

>
> Something must show in the log files - at the very least the 50
> requests, together with their status. You may want to try LogLevel  
> 4 for
> a more verbose display.
>
> In any case I'd be curious to see what a "failed" request means: was
> there no response, or a wrong response, or perhaps a 5xx response?
>
> More info required...
> -- 
> Robert Segall
> Apsis GmbH
> Postfach, Uetikon am See, CH-8707
> Tel: +41-44-920 4904
>
>
> -- 
> To unsubscribe send an email with subject 'unsubscribe' to  
> pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
> http://www.apsis.ch/pound/pound_list/archive/ 
> 2006/2006-05/1146767005000/1146847643000

-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com

      More fuel for the fire.  I took another spare machine, loaded  
up CentOS, and installed 2.0.4 then used the same configuration as  
I've been using on the FreeBSD Pound server.  The good news is that  
it gave me the exact same performance, topping out around 400  
requests per second, and giving the failed connections when using  
round-robin to two backends without IP sessions enabled.  That's also  
the bad news.  Clearly I'm doing something wrong here.

-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com

On Fri, 2006-05-05 at 16:24 -0700, Scott Larson wrote:
> May  5 14:52:58 fencer pound: backend 10.10.10.36:80 connect:  
> Operation not permitted

That is strange. From the man 2 connect (see the section on errors):

    EACCES, EPERM
        The user tried to connect to a broadcast address without  having
        the  socket  broadcast  flag  enabled  or the connection request
        failed because of a local firewall rule.

Any idea how this could happen?

BTW: Pound doesn't do round-robin...
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-44-920 4904

On Mon, May 08, 2006 at 06:31:09PM +0200, Robert Segall wrote:
> On Fri, 2006-05-05 at 16:24 -0700, Scott Larson wrote:
> > May  5 14:52:58 fencer pound: backend 10.10.10.36:80 connect:  
> > Operation not permitted
> 
> That is strange. From the man 2 connect (see the section on errors):
> 
>     EACCES, EPERM
>         The user tried to connect to a broadcast address without  having
>         the  socket  broadcast  flag  enabled  or the connection request
>         failed because of a local firewall rule.
> 
> Any idea how this could happen?

In the case of the pf firewall, it will happen when pf's state table
reaches its max.

-Eric

      I've been thinking about this over the weekend but avoided  
working on it until now.  My assumption was that Pound did round- 
robin based on that fact that when I don't set any session  
configuration, if I throw 1000 connections at it with two IPs  
configured as backends, it roughly gets divided between the two.   
That scenario is also the one which shows a ton of failed  
connections, and this strange error.  After I define some sort of  
session, IP in the case of sorting this out, then all the traffic  
gets directed to the same server and the failures disappear.  We have  
to keep track of sessions anyway so if that's the solution it's  
perfectly acceptable.
      Anyway, using the session tracking stopped those failure  
errors, but still left me with relatively low request handling.   
Having tried different networking gear, computers, and OSes and seen  
the same thing across the board this morning I decided to eliminate  
the physical networking entirely.  This is on one machine, looping  
back to itself.  Apache listens on 80, Pound listens on 8080 with a  
single backend of 127.0.0.1 being defined:

First, hitting Apache directly on localhost, port 80:

Server Software:        Apache/2.2.2
Server Hostname:        127.0.0.1
Server Port:            80

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      10
Time taken for tests:   1.43769 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      264000 bytes
HTML transferred:       16000 bytes
Requests per second:    958.07 [#/sec] (mean)
Time per request:       10.438 [ms] (mean)
Time per request:       1.044 [ms] (mean, across all concurrent  
requests)
Transfer rate:          246.22 [Kbytes/sec] received

Connection Times (ms)
               min   avg   max
Connect:        0     0    1
Processing:     1     9   19
Total:          1     9   20


Second, hitting Pound on localhost 8080 which forwards to the just  
tested Apache on localhost 80:

Server Software:        Apache/2.2.2
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      10
Time taken for tests:   3.282322 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      264000 bytes
HTML transferred:       16000 bytes
Requests per second:    304.66 [#/sec] (mean)
Time per request:       32.823 [ms] (mean)
Time per request:       3.282 [ms] (mean, across all concurrent  
requests)
Transfer rate:          78.30 [Kbytes/sec] received

Connection Times (ms)
               min   avg   max
Connect:        0     0    1
Processing:    10    31   59
Total:         10    31   60


      What's "strange" about the slow request handling is that I can  
up the concurrency level, and Pound will continue to handle it with  
little to no dropoff.  But for whatever reason it's peak performance  
caps off well below what the server behind it can handle.  Before  
coming into work this morning I tried this same loopback test on a  
machine of mine at home and direct to Lighttpd had a reqs/s rate of  
1800, while once it routed through Pound and then to Lighttpd the  
reqs/s rate dropped to 430.  Disabling all logging got me an extra 20  
reqs/s.  I just tested Pound on a CentOS 4.2 machine, a P3 700, using  
the loopback style again.  Direct to Lighttpd, 1600 reqs/s.  To  
Pound, forwarding to Lighttpd on loopback, 380 reqs/s.  I'd expect  
some dropoff but nothing quite that severe.
      I'm going to be working on this most of today so if there are  
some suggestions or requests of things for me to do, I'm open to  
trying them.

On May 8, 2006, at 9:31 AM, Robert Segall wrote:

> On Fri, 2006-05-05 at 16:24 -0700, Scott Larson wrote:
>> May  5 14:52:58 fencer pound: backend 10.10.10.36:80 connect:
>> Operation not permitted
>
> That is strange. From the man 2 connect (see the section on errors):
>
>     EACCES, EPERM
>         The user tried to connect to a broadcast address without   
> having
>         the  socket  broadcast  flag  enabled  or the connection  
> request
>         failed because of a local firewall rule.
>
> Any idea how this could happen?
>
> BTW: Pound doesn't do round-robin...
> -- 
> Robert Segall
> Apsis GmbH
> Postfach, Uetikon am See, CH-8707
> Tel: +41-44-920 4904
>
>
> -- 
> To unsubscribe send an email with subject 'unsubscribe' to  
> pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
> http://www.apsis.ch/pound/pound_list/archive/ 
> 2006/2006-05/1146767005000/1147105869000

-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com

      I actually ran into the state table limit when working on  
another issue, so that's been tweaked in my test environment  
already.  In the case of this error though I didn't realize it was  
the PF state table.

On May 8, 2006, at 10:44 AM, Eric McCarthy wrote:

> On Mon, May 08, 2006 at 06:31:09PM +0200, Robert Segall wrote:
>> On Fri, 2006-05-05 at 16:24 -0700, Scott Larson wrote:
>>> May  5 14:52:58 fencer pound: backend 10.10.10.36:80 connect:
>>> Operation not permitted
>>
>> That is strange. From the man 2 connect (see the section on errors):
>>
>>     EACCES, EPERM
>>         The user tried to connect to a broadcast address without   
>> having
>>         the  socket  broadcast  flag  enabled  or the connection  
>> request
>>         failed because of a local firewall rule.
>>
>> Any idea how this could happen?
>
> In the case of the pf firewall, it will happen when pf's state table
> reaches its max.
>
> -Eric
>
>
> -- 
> To unsubscribe send an email with subject 'unsubscribe' to  
> pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
> http://www.apsis.ch/pound/pound_list/archive/ 
> 2006/2006-05/1146767005000/1147110290000

-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com

      Further progress on squeezing out more performance.  I noticed  
the mention of PCRE improving performance dramatically, and looking  
through the config.log noticed that it was not finding the libs &  
headers, which are installed under /usr/local.  After passing the  
LDFLAGS & CPPFLAGS the proper paths, it linked against PCRE, and I've  
been able to raise the reqs/s when going through Pound from 304 reqs/ 
s to 648 reqs/s on the last run.  Still quite a disparity between  
that and a direct connection but it closed the gap considerably.

On May 8, 2006, at 9:31 AM, Robert Segall wrote:

> On Fri, 2006-05-05 at 16:24 -0700, Scott Larson wrote:
>> May  5 14:52:58 fencer pound: backend 10.10.10.36:80 connect:
>> Operation not permitted
>
> That is strange. From the man 2 connect (see the section on errors):
>
>     EACCES, EPERM
>         The user tried to connect to a broadcast address without   
> having
>         the  socket  broadcast  flag  enabled  or the connection  
> request
>         failed because of a local firewall rule.
>
> Any idea how this could happen?
>
> BTW: Pound doesn't do round-robin...
> -- 
> Robert Segall
> Apsis GmbH
> Postfach, Uetikon am See, CH-8707
> Tel: +41-44-920 4904
>
>
> -- 
> To unsubscribe send an email with subject 'unsubscribe' to  
> pound(at)apsis.ch.
> Please contact roseg(at)apsis.ch for questions.
> http://www.apsis.ch/pound/pound_list/archive/ 
> 2006/2006-05/1146767005000/1147105869000

-- 
Scott Larson
Network Administrator
IOWA Interactive
4212 Glencoe Ave
Marina Del Rey, CA 90292

t 310.823.8238
f 310.823.7108
stl(at)iowainteractive.com
http://www.iowainteractive.com
http://www.wiredrive.com