On 31 Aug 2004, at 20:31, Corin Langosch wrote:

> Hi,
>
> i'm using pound for quite a while, but now i noticed a lot
> of thse error messages in our logs.
> 31/Aug/2004 12:30:32 +0200: HTTP pthread_create: Cannot allocate memory
>
> free outputs:
>              total       used       free     shared    buffers     
> cached
> Mem:       1297444    1013140     284304          0      61696     
> 848848
> -/+ buffers/cache:     102596    1194848
> Swap:      1461872          0    1461872
> so there should really be enough memory?
>
> kernel is Linux 2.6.7-1-k7 #1 Thu Jul 8 06:45:35 EDT 2004 i686 
> GNU/Linux
> with latest debian testing. i also call ulimit -n 8192 before starting
> pound.

Ahuh :)

We get exactly the same problem. It's due to pound running out of 
threads.

According to the README (under THREADS AND LIMITS), "In very rare cases 
(very high load and long response times) you may run into this 
limitation - the symptom is log messages saying "can't create thread". 
Your only solution is to recompile the system threads library with a 
higher limit."

The pound we're running is front-ending three thttpd's serving static 
images: it seems to max out at 255 threads (ps axH | grep -c pound), on 
the same kernel/version as you.

I confess to being very puzzled as to what I'm supposed to increase to 
make this work - PTHREAD_THREAD_MAX seems not to be defined in any 
system include file: running the latest kernel and libc6-i686 doesn't 
seem to make any difference.

I'm sure I'm missing something obvious. But what?

> The pound we're running is front-ending three thttpd's serving static 
> images: it seems to max out at 255 threads (ps axH | grep -c pound), 
> on the same kernel/version as you.

I should point out that from some experiments using Squid, we seem to 
need to be able to serve upwards of 700 hits/s out of this setup. And 
I'll really LIKE to use pound :)

-- 
Mike Whitaker      | Phone: +44 1733 327544
System Architect   | Mobile: +44 7971 977375
WisdenCricinfo Ltd | Email: mike(at)wisdengroup.com

On Wednesday 01 September 2004 11.43, Mike Whitaker wrote:
> > The pound we're running is front-ending three thttpd's serving static
> > images: it seems to max out at 255 threads (ps axH | grep -c pound),
> > on the same kernel/version as you.
>
> I should point out that from some experiments using Squid, we seem to
> need to be able to serve upwards of 700 hits/s out of this setup. And
> I'll really LIKE to use pound :)

The 255 threads have nothing to do with how many requests Pound can answer per 
second, but rather how many concurrent requests can be answered. Pound has 
been shown to answer over 1500 requests/second (in synthetic settings) with 
an unmodified threads library.

As to the OP: when Pound starts showing "can't create thread" it probably 
means you have run into the threads limitation. You'll need to recompile your 
threads library to allow for more concurrent threads.

Before you try doing that please make sure this is not something else: for 
example, on pre-NPTL Linux, each thread is a separate process (threads are 
created via clone(2)). The 255 threads limitation looks much more like a 
ulimit-imposed limit on the number of processes per user rather than the 
limit on the number of threads (normally 1023 on a 2.4.x kernel).

First step: find and identify the limit on the number of threads. The 
parameter is PTHREAD_THREADS_MAX, and on Linux it is defined in 
the /usr/include/bits/local_lim.h file. Make sure that the value defined 
there fits what you observe in the ps listing - if not you have some other 
issue.

If you are satisfied that the issue really is the threads limit change the 
value, recompile and install the threads library. Enjoy, and don't forget you 
still need to deal with the ulimit-stuff.

An easier alternative is to upgrade your system to the 2.6 kernel series and 
NPTL. The default is 16K threads, it is quite a bit faster and all the 
threads run in a single process, so you have no ulimit issues (though you may 
need to increase the memory size per process allowed).
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904

> An easier alternative is to upgrade your system to the 2.6 kernel 
> series and
> NPTL. The default is 16K threads, it is quite a bit faster and all the
> threads run in a single process, so you have no ulimit issues (though 
> you may
> need to increase the memory size per process allowed).

new-master:~# grep PTHREAD_THREADS_MAX /usr/include/bits/local_lim.h
#define PTHREAD_THREADS_MAX     16384
new-master:~# uname -a
Linux new-master 2.6.8-new-hq-servers-02 #1 SMP Sun Aug 29 19:54:08 GMT 
2004 i686 GNU/Linux
new-master:~# ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) unlimited
virtual memory        (kbytes, -v) unlimited

I'm puzzled, frankly. This box is running the Debian optimised 
libc6-i686 package, including NTPL support, and as soon as the pound 
hits 254 threads, it starts throwing up the 'Cannot allocate memory' 
message. 

On Wednesday 01 September 2004 14.46, Mike Whitaker wrote:
> new-master:~# grep PTHREAD_THREADS_MAX /usr/include/bits/local_lim.h
> #define PTHREAD_THREADS_MAX     16384
> new-master:~# uname -a
> Linux new-master 2.6.8-new-hq-servers-02 #1 SMP Sun Aug 29 19:54:08 GMT
> 2004 i686 GNU/Linux
> new-master:~# ulimit -a
> core file size        (blocks, -c) 0
> data seg size         (kbytes, -d) unlimited
> file size             (blocks, -f) unlimited
> max locked memory     (kbytes, -l) unlimited
> max memory size       (kbytes, -m) unlimited
> open files                    (-n) 1024
> pipe size          (512 bytes, -p) 8
> stack size            (kbytes, -s) 8192
> cpu time             (seconds, -t) unlimited
> max user processes            (-u) unlimited
> virtual memory        (kbytes, -v) unlimited
>
> I'm puzzled, frankly. This box is running the Debian optimised
> libc6-i686 package, including NTPL support, and as soon as the pound
> hits 254 threads, it starts throwing up the 'Cannot allocate memory'
> message.

If you see the 254 threads in the ps listing as separate processes you are NOT 
using NPTL. With NPTL all the threads show as a single process. I suspect you 
are running in the backwards-compatability mode - check your system's 
documentation (I don't use Debian myself). We use SuSE 9.1 and latest Gentoo 
quite extensively and had no issues with this.

BTW, the output of ulimit does not indicate real limits, but rather what your 
shell allows. There may be additional limitations in your kernel which ulimit 
would not show.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904

> If you see the 254 threads in the ps listing as separate processes you 
> are NOT
> using NPTL. With NPTL all the threads show as a single process.

I see one process under ps ax, 254 under ps axH (which shows the 
threads).

Hence, as you can understand, my puzzlement.

On 1 Sep 2004, at 14:30, Mike Whitaker wrote:

>> If you see the 254 threads in the ps listing as separate processes 
>> you are NOT
>> using NPTL. With NPTL all the threads show as a single process.
>
> I see one process under ps ax, 254 under ps axH (which shows the 
> threads).
>
> Hence, as you can understand, my puzzlement.

Ah*hah* :)

It appears (though I haven't yet tested it under sufficient load to be 
sure) that under NPTL and a 2.6 kernel, we need to do what what's done 
for BSD in the #ifdef NEED_STACK section, in order to reduce the amount 
of stack needed per thread, and thus up the maximum number of threads.

On 2 Sep 2004, at 08:58, Mike Whitaker wrote:
> It appears (though I haven't yet tested it under sufficient load to be 
> sure) that under NPTL and a 2.6 kernel, we need to do what what's done 
> for BSD in the #ifdef NEED_STACK section, in order to reduce the 
> amount of stack needed per thread, and thus up the maximum number of 
> threads.

Looks like that's indeed the case for Linux 2.6 + NPTL - #define-ing 
NEED_STACK in pound.c - I'd supply a patch to configure, but I'm not 
sure how to detect that case.

My running pound just spiked to 441 threads, which is promising :)

On Thursday 02 September 2004 14.53, Mike Whitaker wrote:
> On 2 Sep 2004, at 08:58, Mike Whitaker wrote:
> > It appears (though I haven't yet tested it under sufficient load to be
> > sure) that under NPTL and a 2.6 kernel, we need to do what what's done
> > for BSD in the #ifdef NEED_STACK section, in order to reduce the
> > amount of stack needed per thread, and thus up the maximum number of
> > threads.
>
> Looks like that's indeed the case for Linux 2.6 + NPTL - #define-ing
> NEED_STACK in pound.c - I'd supply a patch to configure, but I'm not
> sure how to detect that case.
>
> My running pound just spiked to 441 threads, which is promising :)

Please keep us up to date. For your info - the BSD #define is conservative, 
you could probably set the stack size lower.

BTW - are you sure your kernel does not limit the total memory of a running 
process? That would explain it.
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904

On 2 Sep 2004, at 14:18, Robert Segall wrote:

> Please keep us up to date.

Running reasonbably with about 600 threads - it currently only has one 
backend server, for testing purposes.

One oddity is that if I telnet to port 80 on the pound and then do 
nothing for 10 seconds, this happens:

Trying XXX.XXX.XXX.XXX...
Connected to XXXXXXXXX
Escape character is '^]'.
HTTP/1.0 500 Internal Server Error
Content-Type: text/html
Content-Length: 178

<html><head><title>500 Internal Server 
Error</title></head><body><h1>500 Internal Server Error</h1><p>An 
internal server error occurred. Please try again 
later.</p></body></html>Connection closed by foreign host.

Is this normal?

> For your info - the BSD #define is conservative - you could probably 
> set the stack size lower.

I started out with 64K, which is definitely TOO low. 256K (1 << 18) 
seems ok, except for the above.

> BTW - are you sure your kernel does not limit the total memory of a 
> running
> process? That would explain it.

It does. 2GB, with a default stack size of 8192KB per thread. Which 
gives you exactly 256 threads, unless you change the per-thread stack 
size.

On Thursday 02 September 2004 17.23, Mike Whitaker wrote:
> On 2 Sep 2004, at 14:18, Robert Segall wrote:
> > Please keep us up to date.
>
> Running reasonbably with about 600 threads - it currently only has one
> backend server, for testing purposes.

OK.

> One oddity is that if I telnet to port 80 on the pound and then do
> nothing for 10 seconds, this happens:
>
> Trying XXX.XXX.XXX.XXX...
> Connected to XXXXXXXXX
> Escape character is '^]'.
> HTTP/1.0 500 Internal Server Error
> Content-Type: text/html
> Content-Length: 178
>
> <html><head><title>500 Internal Server
> Error</title></head><body><h1>500 Internal Server Error</h1><p>An
> internal server error occurred. Please try again
> later.</p></body></html>Connection closed by foreign host.
>
> Is this normal?

Yes - you see the effect of the Client timeout: you have connected, Pound 
waits for your request, and on reaching time-out Pound says "oops" and closes 
the connection.

> > For your info - the BSD #define is conservative - you could probably
> > set the stack size lower.
>
> I started out with 64K, which is definitely TOO low. 256K (1 << 18)
> seems ok, except for the above.

Fine.

> > BTW - are you sure your kernel does not limit the total memory of a
> > running
> > process? That would explain it.
>
> It does. 2GB, with a default stack size of 8192KB per thread. Which
> gives you exactly 256 threads, unless you change the per-thread stack
> size.

Nice piece of work. Do you think we should remove the NEED_STACK thing 
altogether and just set it for all systems?
-- 
Robert Segall
Apsis GmbH
Postfach, Uetikon am See, CH-8707
Tel: +41-1-920 4904

>> It does. 2GB, with a default stack size of 8192KB per thread. Which
>> gives you exactly 256 threads, unless you change the per-thread stack
>> size.
>
> Nice piece of work.

Thank Google :)

> Do you think we should remove the NEED_STACK thing
> altogether and just set it for all systems?

I'm not a pthreads expert: it's clearly needed if one uses NPTL, rather 
than the older LinuxThreads, on Linux. Whether it'll break the older 
implementation I don't know.
-- 
Mike Whitaker      | Phone: +44 1733 327544
System Architect   | Mobile: +44 7971 977375
WisdenCricinfo Ltd | Email: mike(at)wisdengroup.com