Hi,

I am wrestling with a very weird problem: This morning I wanted to
re-start a pound which has been disabled for some weeks/month. But for
some reason, the pound process failed to start. I went to the pound
homepage, downloaded the latest-and-greatest pound version but the
same problem.

Then I tried to start the pound binary on my development host (where
pound had been compiled) using the same config file ... again failure.

Ok, now the detailes: The devel host is a debian stable (sarge)
install. The openssl library is debians libssl0.9.7e (which probably
is security-patched). I'll try a vanilla 0.9.7i openssl lib
soon.

I first used strace on the binary. After some startup, pound started
to repeatedly call getpid and timeofday forever. Ok. Next step was to
fire up gdb. The crash happend while stepping over the init_RSAgen
call. Next run, stepping into init_RSAgen ... *and everything worked*
??? Now I was in deep waters.

I fired up valgrind, added some debug-printf's to init_RSAgen only to
assure, that the first call to RSA_generate_key fails horribly with
memory corruption or something along those lines.

What I really don't understand, is why the thing works OK under gdb
if single-stepping but not otherwise. Very weird.

I will now try a plain openssl library. Any one have an idea, what
*this* could be?

Stefan.

[...]

Stefan Bund <l-pound.z.xalan(at)xoxy.net> writes:[...]

Ok. Downloaded latest openssl 0.9.7, same problem. Here the first
valgrind error message:

# valgrind --num-callers=100 pound -f ./pound.https.conf
==3801== Memcheck, a memory error detector for x86-linux.
==3801== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==3801== Using valgrind-2.4.0, a program supervision framework for x86-linux.
==3801== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==3801== For more details, rerun with: -v
==3801==
==3801== Conditional jump or move depends on uninitialised value(s)
==3801==    at 0x80DCD1A: __umoddi3 (in /compile/pound/Pound-1.9.4/pound)
==3801==    by 0x80B62E1: BN_mod_word (in /compile/pound/Pound-1.9.4/pound)

The stack already seems corrupted. Further down the statck *certainly*
is corrupted. I'm completely lost ...

stefan.
[...]

On Thu, 2005-10-27 at 12:13 +0200, Stefan Bund wrote:[...]

I suspect you place a bit too much confidence in valgrind and/or gdb. In
my experience they gets horribly confused when you try to use them with
multi-threaded programs.

As to the problem itself: try running pound -v and tell us what it says
(or show us the log file for the start). As a first idea: if you use a
RootJail you may lack access to /dev/random. Second possibility: you run
Pound with the wrong shared library (compiled with 0.9.7i, but 0.9.7e is
found at run-time); try compiling with the static library to be on the
safe side.[...]

Robert Segall <roseg-Ws3YcLWMCps(at)public.gmane.org> writes:[...]

Hmm. I have never experienced problems and have used both extensively
with multi-threaded applications. 
[...]

I verified the libraries with ldd, no discrepancies found. And 'pound
-v' shows *no* output whatsoever. The call to init_RSAgen happens
pretty early ...

And the statically compiled version has the same problem. The valgrind
message is different (which I find weird since I never experienced the
thread problems) but the strace output is the same:

# strace ./pound -v -f ./pound.https.conf 
execve("./pound", ["./pound", "-v", "-f", "./pound.https.conf"], [/* 34 vars
*/]) = 0
[...]
open("/dev/urandom", O_RDONLY|O_NONBLOCK|O_NOCTTY) = 4
fstat64(4, {st_mode=S_IFCHR|0444, st_rdev=makedev(1, 9), ...}) = 0
select(5, [4], NULL, NULL, {0, 10000})  = 1 (in [4], left {0, 10000})
read(4, "\372k8\353\330\3645F\32\266\275\r;h\275i\231\344\2\240"..., 32) = 32
close(4)                                = 0
getuid32()                              = 0
time(NULL)                              = 1130422049
time([1130422049])                      = 1130422049
getpid()                                = 6289
time([1130422049])                      = 1130422049
getpid()                                = 6289
time([1130422049])                      = 1130422049
getpid()                                = 6289
time([1130422049])                      = 1130422049
getpid()                                = 6289
time([1130422049])                      = 1130422049
getpid()                                = 6289
time([1130422049])                      = 1130422049
getpid()                                = 6289
[...]

the getpid/time call is repeated ad infinitum with varying frequency
but at least about 10 times per second.

stefan.
[...]

On Thu, 2005-10-27 at 16:09 +0200, Stefan Bund wrote:
[...]

This looks bizarre, to say the least. As I'm sure you saw, all of this
(if strace is to be trusted) happens inside OpenSSL.

May I suggest you check your OpenSSL installation? Perhaps try running
genrsa just to make sure?[...]

Robert Segall <roseg(at)apsis.ch> writes:[...]

:-/
[...]

The last version with above strace output was produced the following
way:

- I removed all libssl-dev packages from my system
- I downloaded the latest openssl 0.9.7 tarball (0.9.7i), did ./conf,
  make, make test (!) and make install
- Then I built pound again
- Afterwards I removed the pound binary and rebuilt it manually adding
  '-static' to the link command line
- I used ldd to check, that the binary is indeed static

So the openssl version is statically compiled in and successfully ran
all openssl tests ...
[...]

# ./openssl genrsa
Generating RSA private key, 512 bit long modulus
...++++++++++++
.............++++++++++++
e is 65537 (0x10001)
-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBANyL02vNSkMz6gkUwDyw+oSUU0BZToG7uBDuVQQpwqEX5jxnkXW3
U1hiMWXrxjvvIKnBbkRg5oJewEOoKc1GMN0CAwEAAQJAUqSazvSYdWJIuqpp9tQi
q/eg0qYkgvzC1qVYejzjz9UguXJZ7YwnHGsdiymMn5kyHcrlMs+ypktSWbF59szu
AQIhAPFiUEntBQnMD4wLVbVvup+eBBVHq8MlFzT9I86A7Q3BAiEA6eaBZ8RX7C6g
eWGZzLxiBQHYy32gQOLV/u7UYIyYIh0CIQCY6RrLeeZzKl2+7X9CDKFAdC1J/bkF
Sf3Fh4ajTnDdAQIgfMSfvMmImQ0aPwVoNEU9CCy/B3Vg2ZvmyRF/Km107aUCIQC4
dzhZwhwhe2AxMQq6uUEJdLKoQc/Wmnt8EpJeadLe/Q==
-----END RSA PRIVATE KEY-----

Seems to work ...

stefan.
[...]

On Sat, 2005-10-29 at 10:35 +0200, Stefan Bund wrote:[...]

Sorry, I didn't explain it as well as I should have: by 'static' I meant
linking directly with the static version of the OpenSSL you compiled. In
most cases just adding -static doesn't do it. What you really want is to
replace -lssl -lcrypto with /usr/lib/libssl.a /usr/lib/libcrypto.a in
your Makefile (use the path of your new OpenSSL installation rather than
your existing system libraries!). This will ensure beyond doubt that you
are using the correct libraries. Secondly, make sure that you use the
header files from your version (configure Pound --with-ssl=/...), and
make sure the OpenSSL version you compile supports threads.[...]

Robert Segall <roseg(at)apsis.ch> writes:[...]

hrmpf ... of course ... on the other hand, since I removed all other
development packages for openssl there is little doubt but I will try
again

(please stand by)

ok, did it: reconfigured, built and installed openssl with

# make clean
# ./config threads --prefix=/usr/local/stow/openssl
# make install

 rebuild and installed. Then reconfigured and rebuilt pound with 

# make clean
# ./configure --with-ssl=/usr/local/stow/openssl
# make
# gcc -L/usr/local/stow/openssl/lib  -pthread -o pound pound.o http.o \
  config.o svc.o  /usr/local/stow/openssl/lib/libssl.a \
  /usr/local/stow/openssl/lib/libcrypto.a -ldl 

now rechecked with

# sudo strace -f ./pound -f pound.https.conf

same result :-(. Checking with ldd gives

# ldd ./pound
        libdl.so.2 => /lib/tls/libdl.so.2 (0x467ca000)
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x468bc000)
        libc.so.6 => /lib/tls/libc.so.6 (0x4666f000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x46656000)

One thing: When compiling against the newest openssl, I get a huge
number of warnings 'function declaration isn't a prototype' from all
the openssl headers.

Any more ideas ?

stefan.
[...]