Hi,

we run into some trouble using the pound 2.6 on CentOS 6.1. For security
reasons we need the RC4-SHA Cipher as prefered by the server. This can be
accomplished by the SSLHonorCipherOrder 1 directive. The normal 2.6
pound-version is not aware of this, but a 2.6f with a patch (http://goochfriend.org/pound_2.6f_ssl_renegotiation_and_ciphers.patch)
by
Joe Gooch ([Pound Mailing List] SSL renegotiation DDoS and Pound  http://www.mail-archive.com/pound(at)apsis.ch/msg01434.html)
the system is able to handle this.
We applied the patch to a 2.6f and the pound starts to work with the directive.
But we see a lot of seg-faults in the logfile:


Feb  1 13:34:26 pilotpound pound: 2xx.xxx.xxx.x47 GET / HTTP/1.1 - REDIRECT https://www.xxx.xx/
Feb  1 13:34:26 pilotpound kernel: pound[26808]: segfault at 4 ip 08051f5c sp
b761fce0 error 4 in pound[8048000+18000]
Feb  1 13:34:26 pilotpound pound: MONITOR: worker exited on signal 11,
restarting...
Feb  1 13:34:28 pilotpound kernel: pound[26936]: segfault at 4 ip 08051f5c sp
b77e6ce0 error 4 in pound[8048000+18000]
Feb  1 13:34:28 pilotpound pound: MONITOR: worker exited on signal 11,
restarting...
Feb  1 13:34:30 pilotpound kernel: pound[27067]: segfault at 4 ip 08051f5c sp
b77e6ce0 error 4 in pound[8048000+18000]
Feb  1 13:34:30 pilotpound pound: MONITOR: worker exited on signal 11,
restarting...
Feb  1 13:34:32 pilotpound kernel: pound[27198]: segfault at 4 ip 08051f5c sp
b77e6ce0 error 4 in pound[8048000+18000]
Feb  1 13:34:32 pilotpound pound: MONITOR: worker exited on signal 11,
restarting...
Feb  1 13:34:34 pilotpound pound: NULL get_thr_arg
Feb  1 13:34:34 pilotpound kernel: pound[27329]: segfault at 4 ip 08051f5c sp
b77e6ce0 error 4 in pound[8048000+18000]
Feb  1 13:34:34 pilotpound pound: MONITOR: worker exited on signal 11,
restarting...
Feb  1 13:34:36 pilotpound pound: NULL get_thr_arg
Feb  1 13:34:36 pilotpound pound: NULL get_thr_arg
Feb  1 13:34:36 pilotpound kernel: pound[27460]: segfault at 4 ip 08051f5c sp
b77e6ce0 error 4 in pound[8048000+18000]
Feb  1 13:34:36 pilotpound kernel: pound[27464]: segfault at 4 ip 08051f5c sp
b76e2ce0 error 4 in pound[8048000+18000]
Feb  1 13:34:36 pilotpound pound: MONITOR: worker exited on signal 11,
restarting...
...

There is no trouble with the 2.6f without the patch.
Is there a way to get a working pound with SSLHonorCipherOrder awareness for us
?

Any suggestions are welcome

fatcharly


[...]

Can you post the config you're using?  I'll take a look.

Joe
[...]

Hi Joe,

sure, here comes the pound.cfg:

#
# Default pound.cfg
#
# Pound listens on port 80 for HTTP and port 443 for HTTPS # and distributes
requests to 2 backends running on localhost.
# see pound(8) for configuration directives.
# You can enable/disable backends with poundctl(8).
#

User "pound"
Group "pound"
Control "/var/lib/pound/pound.cfg"
Loglevel 1
Alive 30
########################################


###website1
ListenHTTP
		Address 192.168.xxx.xxx
		Port 80
		Service
			Redirect "https://xxx.xxx.xxx.xxx"
			#Backend
			#	Address xxx.xxx.xxx.xxx
			#	Port 80
			#End
		End
End
ListenHTTPS
		Address 192.168.xxx.xxx
		Port 443
		Cert "/etc/pki/tls/pound/xxxxx/xxxxx.pem"
		Ciphers "RC4-SHA:HIGH:!ADH:!SSLv2:!AES"		
#Ciphers "ALL:RC4-SHA:!ADH:!EXPORT:!AES:!DES:!SSLv2:RC4+RSA:+HIGH:!MEDIUM:!LOW"
		Service
			Backend
				Address xxx.xxx.xxx.xxx
				Port 80
			End
		
			Session
				Type COOKIE
				ID "xxxxxxx"
				TTL 360
			End
		End
End
########################
###website2
ListenHTTP
                Address 192.168.xxx.xxx
                Port 80
                Service
                        Redirect "https://www.xxx.xx"
                        #Backend
                        #       Address xxx.xxx.xxx.xxx
                        #       Port 80
                        #End
                End
End
ListenHTTPS
                Address 192.168.xxx.xxx
                Port 443
                Cert "/etc/pki/tls/pound/xxx/xxx.pem"
		#SSLAllowClientRenegotiation 2
		SSLHonorCipherOrder 1
		Ciphers "RC4-SHA:HIGH:!ADH:!SSLv2:!AES"
		#Ciphers
"ALL:RC4-SHA:!ADH:!EXPORT:!AES:!DES:!SSLv2:RC4+RSA:+HIGH:!MEDIUM:!LOW" 
		Service
                        Backend
                                Address xxx.xxx.xxx.xxx
                                Port 80
                        End

                        Session
                                Type COOKIE
                                ID "xxxxxx"
                                TTL 360
                        End
                End
End
##########################
###website3
ListenHTTP
                Address 192.168.xxx.xxx
                Port 80
                Service
                        Redirect "https://xxx.xxx.xx"
                        #Backend
                        #       Address 85.183.207.89
                        #       Port 80
                        #End
                End
End
#########################
###website4
ListenHTTP
                Address 192.168.xxx.xxx
                Port 80
                Service
                        Redirect "https://www.xxxx.xx"
                        #Backend
                        #       Address xxx.xxx.xxx.xxx
                        #       Port 80
                        #End
                End
End
ListenHTTPS
                Address 192.168.xxx.xxx
                Port 443
                Cert "/etc/pki/tls/pound/xxxxx/xxxxx.pem"
		Ciphers "RC4-SHA:HIGH:!ADH:!SSLv2:!AES"                
#Ciphers "ALL:!ADH:!EXPORT:!AES:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW"
		Service
                        Backend
                                Address 192.168.xxx.xxx
                                Port 80
                        End

                        Session
                                Type COOKIE
                                ID "xxxxxxxxxxxx"
                                TTL 360
                        End
                End
End
#########################
###website5
ListenHTTP
                Address 192.168.xxx.xxx
                Port 80
                Service
                        Redirect "https://xxxx.xxxxxx.de"
                        #Backend
                        #       Address xxx.xxx.xxx.xxx
                        #       Port 80
                        #End
                End
End
ListenHTTPS
                Address 192.168.xxx.xxx
                Port 443
                Cert "/etc/pki/tls/pound/xxxxxx/xxxxxx.pem"
		Ciphers "RC4-SHA:HIGH:!ADH:!SSLv2:!AES"                
#Ciphers "ALL:RC4-SHA:!ADH:!EXPORT:!DES:!AES:!SSLv2:RC4+RSA:+HIGH:!MEDIUM:!LOW"
                Service
                        Backend
                                Address 192.168.xxx.xxxxx
                                Port 80
                        End

                        Session
                                Type COOKIE
                                ID "xxxxxxxxx"
                                TTL 360
                        End
                End
End



Kind Regards

fatcharly

[...]
[...]

I build a centos 6.1 x86_64 chroot on my workstation and compiled 2.6f with
that patch.

First, I assume you fixed the log messages in config.c, on lines 803, 807, 809,
814... Instead of servername it should be server_name.  Of course this has to
do with SNI.. Which really doesn't belong in that patch at all but was
mistakenly included.  You can remove that hunk from the patch if you like. 
(starts with (at)(at) -794,17 +800,18 (at)(at))

Second, I pared down my config a lot and used my cert with your ciphers string,
and everything works on unpatched 2.6f.  Then I patched and it kept working,
then I turned on honororder and renegotiation 2, and it's still working.

I tend to use global services instead of listener based services, so I went
through and made a redirect on non-ssl and moved my services inside the https
listener.... Still no problem.

So I can't get it to fail.  But I tried!

I did a quick look through the patch as well and don't see any obvious pointer
problems.

So, follow-ups
1) Does this happen on every request for you? Or is it sporadic?
2) 32 or 64 bit?  I can whip up a i386 chroot if need be.
3) Looking at the packages below do you see any blatant differences between my
setup and yours?
4) Anything else you can think of to help me track this down for you?

If all else fails, maybe you can tar.gz up your compiled directory (with binary
and .o files) and I'll try your binary directly. (possibly with gdb mojo)

Joe

Build environment had the following packages:
audit-libs-2.1.3-3.el6.x86_64
basesystem-10.0-4.el6.noarch
bash-4.1.2-8.el6.centos.x86_64
binutils-2.20.51.0.2-5.28.el6.x86_64
bzip2-libs-1.0.5-7.el6_0.x86_64
ca-certificates-2010.63-3.el6_1.5.noarch
centos-release-6-1.el6.centos.6.x86_64
chkconfig-1.3.47-1.el6.x86_64
cloog-ppl-0.15.7-1.2.el6.x86_64
coreutils-8.4-16.el6.x86_64
coreutils-libs-8.4-16.el6.x86_64
cpio-2.10-9.el6.x86_64
cpp-4.4.6-3.el6.x86_64
cracklib-2.8.16-4.el6.x86_64
cracklib-dicts-2.8.16-4.el6.x86_64
curl-7.19.7-26.el6_1.2.x86_64
cyrus-sasl-lib-2.1.23-13.el6.x86_64
db4-4.7.25-16.el6.x86_64
db4-utils-4.7.25-16.el6.x86_64
dbus-libs-1.2.24-5.el6_1.x86_64
elfutils-libelf-0.152-1.el6.x86_64
ethtool-2.6.33-0.3.el6.x86_64
expat-2.0.1-9.1.el6.x86_64
file-libs-5.04-11.el6.x86_64
filesystem-2.4.30-3.el6.x86_64
findutils-4.4.2-6.el6.x86_64
fipscheck-1.2.0-7.el6.x86_64
fipscheck-lib-1.2.0-7.el6.x86_64
gamin-0.1.10-9.el6.x86_64
gawk-3.1.7-6.el6.x86_64
gcc-4.4.6-3.el6.x86_64
gdbm-1.8.0-36.el6.x86_64
git-1.7.1-2.el6_0.1.x86_64
glib2-2.22.5-6.el6.x86_64
glibc-2.12-1.47.el6_2.5.x86_64
glibc-common-2.12-1.47.el6_2.5.x86_64
glibc-devel-2.12-1.47.el6_2.5.x86_64
glibc-headers-2.12-1.47.el6_2.5.x86_64
gmp-4.3.1-7.el6.x86_64
gnupg2-2.0.14-4.el6.x86_64
gpgme-1.1.8-3.el6.x86_64
gpg-pubkey-c105b9de-4e0fd3a3
grep-2.6.3-2.el6.x86_64
groff-1.18.1.4-21.el6.x86_64
gzip-1.3.12-18.el6.x86_64
hwdata-0.233-7.6.el6.noarch
info-4.13a-8.el6.x86_64
initscripts-9.03.27-1.el6.centos.x86_64
iproute-2.6.32-17.el6.x86_64
iptables-1.4.7-4.el6.x86_64
iputils-20071127-16.el6.x86_64
kernel-headers-2.6.32-220.4.1.el6.x86_64
keyutils-libs-1.4-3.el6.x86_64
keyutils-libs-devel-1.4-3.el6.x86_64
krb5-devel-1.9-22.el6_2.1.x86_64
krb5-libs-1.9-22.el6_2.1.x86_64
less-436-10.el6.x86_64
libacl-2.2.49-6.el6.x86_64
libattr-2.4.44-7.el6.x86_64
libblkid-2.17.2-12.4.el6.x86_64
libcap-2.16-5.5.el6.x86_64
libcom_err-1.41.12-11.el6.x86_64
libcom_err-devel-1.41.12-11.el6.x86_64
libcurl-7.19.7-26.el6_1.2.x86_64
libedit-2.11-4.20080712cvs.1.el6.x86_64
libffi-3.0.5-3.2.el6.x86_64
libgcc-4.4.6-3.el6.x86_64
libgcrypt-1.4.5-9.el6.x86_64
libgomp-4.4.6-3.el6.x86_64
libgpg-error-1.7-4.el6.x86_64
libidn-1.18-2.el6.x86_64
libnih-1.0.1-7.el6.x86_64
libselinux-2.0.94-5.2.el6.x86_64
libselinux-devel-2.0.94-5.2.el6.x86_64
libsepol-2.0.41-4.el6.x86_64
libsepol-devel-2.0.41-4.el6.x86_64
libssh2-1.2.2-7.el6_1.1.x86_64
libstdc++-4.4.6-3.el6.x86_64
libusb-0.1.12-23.el6.x86_64
libutempter-1.1.5-4.1.el6.x86_64
libuuid-2.17.2-12.4.el6.x86_64
libxml2-2.7.6-4.el6_2.1.x86_64
lua-5.1.4-4.1.el6.x86_64
make-3.81-19.el6.x86_64
MAKEDEV-3.24-6.el6.x86_64
mingetty-1.08-5.el6.x86_64
module-init-tools-3.9-17.el6.x86_64
mpfr-2.4.1-6.el6.x86_64
nano-2.0.9-7.el6.x86_64
ncurses-5.7-3.20090208.el6.x86_64
ncurses-base-5.7-3.20090208.el6.x86_64
ncurses-libs-5.7-3.20090208.el6.x86_64
net-tools-1.60-109.el6.x86_64
nspr-4.8.8-3.el6.x86_64
nss-3.12.10-17.el6_2.x86_64
nss-softokn-3.12.9-11.el6.x86_64
nss-softokn-freebl-3.12.9-11.el6.x86_64
nss-sysinit-3.12.10-17.el6_2.x86_64
nss-util-3.12.10-2.el6.x86_64
openldap-2.4.23-20.el6.x86_64
openssh-5.3p1-70.el6_2.2.x86_64
openssh-clients-5.3p1-70.el6_2.2.x86_64
openssl-1.0.0-20.el6_2.1.x86_64
openssl-devel-1.0.0-20.el6_2.1.x86_64
pam-1.1.1-10.el6.x86_64
patch-2.6-6.el6.x86_64
pcre-7.8-3.1.el6.x86_64
pcre-devel-7.8-3.1.el6.x86_64
perl-5.10.1-119.el6_1.1.x86_64
perl-Error-0.17015-4.el6.noarch
perl-Git-1.7.1-2.el6_0.1.noarch
perl-libs-5.10.1-119.el6_1.1.x86_64
perl-Module-Pluggable-3.90-119.el6_1.1.x86_64
perl-Pod-Escapes-1.04-119.el6_1.1.x86_64
perl-Pod-Simple-3.13-119.el6_1.1.x86_64
perl-version-0.77-119.el6_1.1.x86_64
pinentry-0.7.6-6.el6.x86_64
pkgconfig-0.23-9.1.el6.x86_64
popt-1.13-7.el6.x86_64
ppl-0.10.2-11.el6.x86_64
procps-3.2.8-21.el6.x86_64
psmisc-22.6-15.el6_0.1.x86_64
pth-2.0.7-9.3.el6.x86_64
pygpgme-0.1-18.20090824bzr68.el6.x86_64
python-2.6.6-29.el6.x86_64
python-iniparse-0.3.1-2.1.el6.noarch
python-libs-2.6.6-29.el6.x86_64
python-pycurl-7.19.0-8.el6.x86_64
python-urlgrabber-3.9.1-8.el6.noarch
readline-6.0-3.el6.x86_64
rpm-4.8.0-19.el6.x86_64
rpm-libs-4.8.0-19.el6.x86_64
rpm-python-4.8.0-19.el6.x86_64
rsync-3.0.6-5.el6_0.1.x86_64
sed-4.2.1-7.el6.x86_64
setup-2.8.14-13.el6.noarch
shadow-utils-4.1.4.2-13.el6.x86_64
sqlite-3.6.20-1.el6.x86_64
sysvinit-tools-2.87-4.dsf.el6.x86_64
tzdata-2011l-4.el6.noarch
udev-147-2.40.el6.x86_64
upstart-0.6.5-10.el6.x86_64
util-linux-ng-2.17.2-12.4.el6.x86_64
wget-1.12-1.4.el6.x86_64
xz-libs-4.999.9-0.3.beta.20091007git.el6.x86_64
yum-3.2.29-22.el6.centos.noarch
yum-metadata-parser-1.1.2-16.el6.x86_64
yum-plugin-fastestmirror-1.1.30-10.el6.noarch
zlib-1.2.3-27.el6.x86_64
zlib-devel-1.2.3-27.el6.x86_64

Hi Joe,

yes we did fix the patchfile. I did some further investigation on this and
there are some news I have to share. First some answers for your
questions:[...]
no, its much more than just sporadic, some request get answered and some
not.[...]
it´s plain 32 bit [...]
no, but I will put my list in a special mail to send it directly with the
tar-archive of our pound-directory to you[...]
Yes, I could zero in the problem a bit. First a bit about our setup:
The pound is in dmz-A, the webserver is in dmz-B, and the requesting Client
comes a) from the internet or b) from the internal network. When we start the
pound everything works fine, as long as the requests are coming from the
internal network and the request is send to an IP of the dmz-A network. So
everything worked with this setup for the internal network. But when there are
requests from the internet, we get segfaults. The request is received from the
firewall which does a NAT to pass the external IP of the website to the
internal IP of the dmz-A network. And some requests are working (as I can see
in the logfile of pound) and some cause segfaults. We can only test this by
switching between the pound and our loadbalancer-appliance (as this one works,
we are sure the NAT is not a problem) the productive path. So maybe there is a
problem with some IP´s which cause the segfault. The segfaults appear even
when there is no SSLHonorCipherOrder enabled. I´m not deep into this 
 segfault thing, but there the word "ip" mentioned:
Feb  2 11:45:52 pilotpound kernel: pound[28641]: segfault at 4 ip 08051f5c sp
b7610ce0 error 4 in pound[8048000+18000]

Is there anything else I can do to support you ?

Kind Regards

fatcharly

 
[...]
[...]

It still won't segfault for me. :-/

"ip" in this context means instruction pointer, not internet protocol.
http://stackoverflow.com/questions/2549214/interpreting-segfault-messages

addr2line -e pound 08051f5c
/root/download/Pound-2.6f/config.c:808

Which, is square in the middle of the SNI checking.

At the top of your config.c (say around line 74) can you do 
#undef SSL_CTRL_SET_TLSEXT_SERVERNAME_CB

And recompile?  That should disable SNI.  (Which IIRC you weren't using anyway)

And then let me know if you still see segfaults.

Further, could you provide the subject of all the certificates you're using? 
I.e. the output of:
openssl x509 -noout -in yourpemfile.pem -subject


Joe

[...]

Also, perhaps running it with -v, or setting LogFacility -, (or both) will
yield a bigger picture... That'll output all the logs on the console. (so
you'll see debug and info and everything else on the same screen)  In your msg
below I'm not seeing the LOG_DEBUG messages from SNI... So maybe syslog is
filtering those out, or saving them elsewhere...

Joe
[...]

Hi Joe,

good news, after we applied the line "#undef SSL_CTRL_SET_TLSEXT_SERVERNAME_CB"
 in the config.c and a new compile, we don´t see any segfaults. I´m afraid,
but it´s not possible for me to send you all of the x509-Information. But I
can tell you that we have 2 EV-SSL´s and two "normal" SSL-Certificates. Do you
need some more information or maybe some information than won´t show any
company information of the SSL-Certificate ?

Kind Regards

fatcharly




-------- Original-Nachricht --------[...]
[...]
[...]

No worries. You can PM the information to me, or really, what Pound extracts is
the CN information.  Or at least that's what the regex is supposed to pull.  I
was hoping to see the subject line so I could see if it's in a format pound
should parse properly, or if it's something else it's not expecting.

My thought is either your cert's subject line isn't being parsed properly,
which is causing a problem in fnmatch, or the value isn't being initialized at
all (but I'm not sure how that would happen)... Or somehow turning on the honor
cipher order option causes some other type of callback to occur with SNI....
But I can't see how Cipher Suites would be related to SNI servername
extensions.

But I certainly don't want to compromise your SSL security.

Joe
[...]

Got it.  It is introduced in that patch.

Remove the logmsg line before the fnmatch line.  The logmsg line was inserted
in the for loop but the for loop doesn't have a {}, so it's not calling the if
until AFTER the loop.

That should take care of everything.

I'm going to clean up the SSL patch and repost it without that SNI detritus.

Joe
[...]

Use this one instead.
http://goochfriend.org/pound_2.6f_ssl_renegotiation_and_ciphers_v2.patch

Should start with hash 1698011920aa9c.

Changes -
Remove the SNI logging information (that never belonged as part of this patch
and caused segfaults)
Redo the whitespace to use spaces instead of tabs to be consistent with pound
best practices

Joe
 [...]

And in addition, updated patches:
http://goochfriend.org/pound_2.6f_xss_redirect_fix_v2.patch

The only difference here is the whitespace handling.  No functional changes.

http://goochfriend.org/pound_2.6f_sni_optimization.patch

Only look at SNI headers/callback if we have more than one cert... No need to
do a callback otherwise.


Joe[...]