Hello!

Currently I'm trying to debug an annoying bug, that causes Pound to refuse SSL 
connections after a while. At the moment, we have to restart Pound every now 
and then to accept https.

Last night I tried to track it all down. I haven't wait for it to long and it 
happened again that our monitoring script reported ssl problems. Then I 
started ssldump [1] to dump my connection attempts. This is the output:

New TCP connection #53: p3E9922AC.dip0.t-ipconnect.de(44001) <-> 
www.native-instruments.de(443)
53 1  0.0487 (0.0487)  C>S SSLv2 compatible client hello
  Version 3.0
  cipher suites
  SSL_RSA_WITH_RC4_128_MD5
  SSL2_CK_RC4
  SSL_RSA_WITH_RC4_128_SHA
  SSL_RSA_WITH_3DES_EDE_CBC_SHA
  SSL2_CK_RC2
  SSL2_CK_3DES
  SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
  SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
  SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
  SSL_DHE_DSS_WITH_RC4_128_SHA
  SSL_DH_anon_WITH_RC4_128_MD5
  SSL2_CK_RC464
  SSL_DHE_DSS_WITH_RC2_56_CBC_SHA
  SSL_RSA_EXPORT1024_WITH_RC4_56_SHA
  SSL_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA
  SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA
  SSL_RSA_EXPORT1024_WITH_RC2_CBC_56_MD5
  SSL_RSA_EXPORT1024_WITH_RC4_56_MD5
  SSL_DHE_RSA_WITH_DES_CBC_SHA
  SSL_DHE_DSS_WITH_DES_CBC_SHA
  SSL_RSA_WITH_DES_CBC_SHA
  SSL_DH_anon_WITH_DES_CBC_SHA
  SSL2_CK_DES
53 2  0.0511 (0.0024)  S>C  Handshake
      ServerHello
        Version 3.0
        session_id[32]=
          a2 52 33 5e f9 c8 1b 2e b9 fe ae a2 d3 64 c9 cf
          52 23 56 fa 9c ca a3 31 7d e7 ba 92 95 64 83 36
        cipherSuite         SSL_RSA_WITH_RC4_128_MD5
        compressionMethod                   NULL
53 3  0.0511 (0.0000)  S>C  Handshake
      Certificate
53 4  0.0511 (0.0000)  S>C  Handshake
      ServerHelloDone
1    9.1273 (9.1273)  C>S  TCP RST
53 5  0.3404 (0.2893)  C>S  Handshake
      ClientKeyExchange
53 6  0.3404 (0.0000)  C>S  ChangeCipherSpec
53 7  0.3404 (0.0000)  C>S  Handshake
53 8  0.3477 (0.0073)  S>C  Alert
    level           fatal
    value           bad_record_mac
53    0.3479 (0.0001)  S>C  TCP FIN


As you can see, there is a fatal error at the end, reporting a 
"bad_record_mac". The TLS Internet Draft says:

   bad_record_mac
       This alert is returned if a record is received with an incorrect
       MAC. This alert also SHOULD be returned if a TLSCiphertext
       decrypted in an invalid way: either it wasn't an even multiple of
       the block length, or its padding values, when checked, weren't
       correct. This message is always fatal.


These alerts are AFAIK not caused by Pound itself, but by one of the SSL 
implementations (client or server). But it seems that Pound threads neither 
recognizes or honors these fatal alerts and simply hangs. Remember: only a 
restart of Pound solves this.

OpenSSL offers ways to set "info callbacks" for ctx [3]. I don't know, if it 
makes sense to add a callback that reacts on fatal ssl alerts and does 
something useful (kill thread, write log, etc). What do you think? BIO_CB_* 
is not the same.


[1] http://www.rtfm.com/ssldump/
[2] 
http://www.ietf.org/internet-drafts/draft-ietf-tls-rfc2246-bis-05.txt&e=7415
[3] http://www.openssl.org/docs/ssl/SSL_CTX_set_info_callback.html


Andi[...]

On Friday 05 September 2003 13:04, you wrote:[...]

This should certainly NOT happen - and we never saw it in our tests! Perhaps 
you could tell us on what system(s)/what library version(s) this was 
recorded? For example we have had several reports of major problems on RedHat 
9 (apparently they made some problematic changes in the threads library).

In your specific case I can offer a temporary work-around: set a time-out of 
non-zero, which should avoid the lock.[...]

Hello!

Am Samstag, 6. September 2003 12:34 schrieb Robert Segall:[...]
[...]
[...]

  glibc 2.3.2
  linux 2.4.20
  gcc 3.2.3
  openssl-0.9.6i

The openssl library is the original SuSE RPM with its own patches, for example 
the "timing attacks" patch. In later releases of openssl it is stated that 
the RSA blinding code wasn't thread safe and has been fixed. This could be 
one reason for our problem. I'll update update openssl asap.

The SuSE gcc 3.3 (prerelease) wasn't able to compile newer kernels, so I 
downgraded to gcc 3.2.3. In the meantime gcc 3.3 has been released and I will 
rebuild at least the kernel with it asap.
[...]

Client and Server is set to 20.


Andi[...]

Hello!

Update:

Am Samstag, 6. September 2003 15:02 schrieb Andreas Roedl:[...]

Done. Compiled and installed openssl-0.9.7b.
[...]

Done.

This time I compiled pound as a static binary with updated openssl and gcc 
3.3.1. The glibc is still the same 2.3.2, but I hope that this is not the 
reason. The static pound is currently running and now I have to wait for the 
next look-up.
[...]


Andi[...]

Hallo!

Am Montag, 8. September 2003 19:42 schrieb Andreas Roedl:[...]

Thats it. Pound doesn't work correctly (stable) on a SuSE 8.2 with its 
openssl-0.9.6i-12 RPM, because they included the so called "RSA Blinding 
Patch", that fixes a vulnerability. Unfortunately, this patch "is NOT 
thread-safe and will cause failures for programs that use threads and OpenSSL 
such as stunnel" [1]. The problem: SuSE built its patched OpenSSL-RPM with 
threads _enabled_. Do not use the SuSE openssl RPM (at least not 0.9.6i-12).

The RSA blinding code in newer versions of OpenSSL (since version 0.9.7b) 
_are_ thread safe.


http://www.suse.com/us/private/download/updates/82_i386.html

  Security Update!
  Security update for two different kind of attacks against the RSA
  implementation: timing and an extention of the "Bleichenbacher attack".
  Remote attackers can compute the private RSA key by observing the timing
  behavior of services using openssl. 


[1] http://www.debian.org/security/2003/dsa-288.en.html


Andi[...]

On Tuesday 09 September 2003 13:44, you wrote:[...]

Great job in tracking it down Andi. Many thanks for the info.[...]