On 31 Aug 2004, at 20:31, Corin Langosch wrote:
[...]

Ahuh :)

We get exactly the same problem. It's due to pound running out of 
threads.

According to the README (under THREADS AND LIMITS), "In very rare cases 
(very high load and long response times) you may run into this 
limitation - the symptom is log messages saying "can't create thread". 
Your only solution is to recompile the system threads library with a 
higher limit."

The pound we're running is front-ending three thttpd's serving static 
images: it seems to max out at 255 threads (ps axH | grep -c pound), on 
the same kernel/version as you.

I confess to being very puzzled as to what I'm supposed to increase to 
make this work - PTHREAD_THREAD_MAX seems not to be defined in any 
system include file: running the latest kernel and libc6-i686 doesn't 
seem to make any difference.

I'm sure I'm missing something obvious. But what?

> The pound we're running is front-ending three thttpd's serving static
[...]

I should point out that from some experiments using Squid, we seem to 
need to be able to serve upwards of 700 hits/s out of this setup. And 
I'll really LIKE to use pound :)
[...]

On Wednesday 01 September 2004 11.43, Mike Whitaker wrote:[...]

The 255 threads have nothing to do with how many requests Pound can answer per 
second, but rather how many concurrent requests can be answered. Pound has 
been shown to answer over 1500 requests/second (in synthetic settings) with 
an unmodified threads library.

As to the OP: when Pound starts showing "can't create thread" it probably 
means you have run into the threads limitation. You'll need to recompile your 
threads library to allow for more concurrent threads.

Before you try doing that please make sure this is not something else: for 
example, on pre-NPTL Linux, each thread is a separate process (threads are 
created via clone(2)). The 255 threads limitation looks much more like a 
ulimit-imposed limit on the number of processes per user rather than the 
limit on the number of threads (normally 1023 on a 2.4.x kernel).

First step: find and identify the limit on the number of threads. The 
parameter is PTHREAD_THREADS_MAX, and on Linux it is defined in 
the /usr/include/bits/local_lim.h file. Make sure that the value defined 
there fits what you observe in the ps listing - if not you have some other 
issue.

If you are satisfied that the issue really is the threads limit change the 
value, recompile and install the threads library. Enjoy, and don't forget you 
still need to deal with the ulimit-stuff.

An easier alternative is to upgrade your system to the 2.6 kernel series and 
NPTL. The default is 16K threads, it is quite a bit faster and all the 
threads run in a single process, so you have no ulimit issues (though you may 
need to increase the memory size per process allowed).[...]

> An easier alternative is to upgrade your system to the 2.6 kernel [...]

new-master:~# grep PTHREAD_THREADS_MAX /usr/include/bits/local_lim.h
#define PTHREAD_THREADS_MAX     16384
new-master:~# uname -a
Linux new-master 2.6.8-new-hq-servers-02 #1 SMP Sun Aug 29 19:54:08 GMT 
2004 i686 GNU/Linux
new-master:~# ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) unlimited
virtual memory        (kbytes, -v) unlimited

I'm puzzled, frankly. This box is running the Debian optimised 
libc6-i686 package, including NTPL support, and as soon as the pound 
hits 254 threads, it starts throwing up the 'Cannot allocate memory' 
message.

On Wednesday 01 September 2004 14.46, Mike Whitaker wrote:[...]

If you see the 254 threads in the ps listing as separate processes you are NOT 
using NPTL. With NPTL all the threads show as a single process. I suspect you 
are running in the backwards-compatability mode - check your system's 
documentation (I don't use Debian myself). We use SuSE 9.1 and latest Gentoo 
quite extensively and had no issues with this.

BTW, the output of ulimit does not indicate real limits, but rather what your 
shell allows. There may be additional limitations in your kernel which ulimit 
would not show.[...]

> If you see the 254 threads in the ps listing as separate processes you
[...]

I see one process under ps ax, 254 under ps axH (which shows the 
threads).

Hence, as you can understand, my puzzlement.

On 1 Sep 2004, at 14:30, Mike Whitaker wrote:
[...][...]

Ah*hah* :)

It appears (though I haven't yet tested it under sufficient load to be 
sure) that under NPTL and a 2.6 kernel, we need to do what what's done 
for BSD in the #ifdef NEED_STACK section, in order to reduce the amount 
of stack needed per thread, and thus up the maximum number of threads.

On 2 Sep 2004, at 08:58, Mike Whitaker wrote:[...]

Looks like that's indeed the case for Linux 2.6 + NPTL - #define-ing 
NEED_STACK in pound.c - I'd supply a patch to configure, but I'm not 
sure how to detect that case.

My running pound just spiked to 441 threads, which is promising :)

On Thursday 02 September 2004 14.53, Mike Whitaker wrote:[...]

Please keep us up to date. For your info - the BSD #define is conservative, 
you could probably set the stack size lower.

BTW - are you sure your kernel does not limit the total memory of a running 
process? That would explain it.[...]

On 2 Sep 2004, at 14:18, Robert Segall wrote:
[...]

Running reasonbably with about 600 threads - it currently only has one 
backend server, for testing purposes.

One oddity is that if I telnet to port 80 on the pound and then do 
nothing for 10 seconds, this happens:

Trying XXX.XXX.XXX.XXX...
Connected to XXXXXXXXX
Escape character is '^]'.
HTTP/1.0 500 Internal Server Error
Content-Type: text/html
Content-Length: 178

<html><head><title>500 Internal Server 
Error</title></head><body><h1>500 Internal Server
Error</h1><p>An 
internal server error occurred. Please try again 
later.</p></body></html>Connection closed by foreign host.

Is this normal?
[...]

I started out with 64K, which is definitely TOO low. 256K (1 << 18) 
seems ok, except for the above.
[...]

It does. 2GB, with a default stack size of 8192KB per thread. Which 
gives you exactly 256 threads, unless you change the per-thread stack 
size.

On Thursday 02 September 2004 17.23, Mike Whitaker wrote:[...]

OK.
[...]

Yes - you see the effect of the Client timeout: you have connected, Pound 
waits for your request, and on reaching time-out Pound says "oops" and closes 
the connection.
[...]

Fine.
[...]

Nice piece of work. Do you think we should remove the NEED_STACK thing 
altogether and just set it for all systems?[...]

>> It does. 2GB, with a default stack size of 8192KB per thread.
Which[...][...]

Thank Google :)
[...]

I'm not a pthreads expert: it's clearly needed if one uses NPTL, rather 
than the older LinuxThreads, on Linux. Whether it'll break the older 
implementation I don't know.[...]