Dear all

we've been running Pound (version 1.10) in front of a Subversion (1.2.3 on
Apache 2.0.55) repository for some time. Internal and external subversion
clients use a generic URL to connect against the repository. Pound maintains
this subversion URL together with other plain HTTPS based services and 
redirects
requests to different backend servers.

This setup has been in use for some time without much problems. We've been 
using it
with different subversion clients (command line and TortoiseSVN up to 
version 1.3.0)
on Linux and W2K.

Recently, with the start of a new project the load on subversion increased 
and
additional client configurations were entering the game and some time ago 
we came
across a problem with WinXP clients running a full checkout from the 
repository. As
far as I can tell, this problem only appears on WinXP on the initial 
checkout of a
project with code larger than 10MB. The problem can be reproduced in our 
setup on
internal and external WinXP clients using the URL provided by pound. The 
choice of
the subversion client (svn, TortoiseSVN, JavaSVN) has no effect on the 
problem.
Connecting directly to the backend server providing the repository the 
checkout
finishes successfully.

The error message on the client typically looks like this:

...
A 
src\DataAccessTier\toplink\DataAccess_1\descriptor\DataAccess.CountryInSalesRegion.ClassDescriptor.xml
svn: REPORT Anfrage fehlgeschlagen auf '/svn/xmt/!svn/vcc/default'
svn: REPORT von '/svn/xmt/!svn/vcc/default': Could not read response body: 
Secure connection truncated (https://.....)
...

Apache says the following:

[Wed Feb 08 17:26:28 2006] [info] [client 192.168.109.57] (32)Broken pipe: 
core_output_filter: writing data to the network
[Wed Feb 08 17:26:28 2006] [error] [client 192.168.109.57] Provider 
encountered an error while streaming a REPORT response.  [500, #0]
[Wed Feb 08 17:26:28 2006] [error] [client 192.168.109.57] A failure 
occurred while driving the update report editor  [500, #190004]
[Wed Feb 08 17:26:28 2006] [info] [client 192.168.109.57] (32)Broken pipe: 
core_output_filter: writing data to the network

I was running some tests to figure out what is happening and here is what
I found out:

Comparing the network dumps of failed and successful checkouts on W2K and
WinXP revealed one significant difference. Apart from the usual protocol
traffic the WinXP dump contained additional TCP Keep-Alive packages sent
from the client to the backend server. The backend server acknowledges these
packages but the client does not seem to receive these acknowledgements
since the client retransmits TCP Keep-Alive packages with the same sequence
number a couple of times until the keep-alive limits have exceeded. Then the
connections is terminated.

I'm not a network expert and I'm not entirely sure to understand the meaning
of these keep-alive packages but from my understanding it seems to me
that Pound is blocking the acknowledged packages sent from the backend 
server
to the client. Since the client is not receiving these packages the OS 
(WinXP, not
the application) considers the connection to be dead and shuts down the 
socket.
The application only realizes that the connection has been dropped.

A couple of workarounds have been considered and tested but none is 
delivering
satisfying results. First of all, rerunning the checkout may succeed if the
remaining code is smaller that the threshold. Once the code has been 
checkout
most of the subversion commands work properly as long as they are not 
running
into the keep-alive limits. The next idea was to disable sending TCP 
Keep-Alive
packages on OS level. According to Microsoft documentation the keep-alive 
feature
is something the application initializes during socket configuration. 
Changing the
related registry keys on system level has not shown any effect. If and how 
this
would be possible on application level is not known to me. Finally, if I 
could make
the system running the Pound service to answer the incoming keep-alive 
packages
before pound is involved (netfilter, iptables) the client would be happy 
and pound
would not be aware of these packages. This raises the question who in a 
proxy setup
should be answering these packages, the proxy or the backend server? Or 
would you
consider this to be a problem of the subversion implementation on WinXP?

Any help or suggestions would be much appreciated ...

Thanks,

   Eckard Koch

On Thu, 2006-02-09 at 10:26 +0100, Eckard Koch wrote:[...]

Can't help you much, except to say that Pound explicitly enables
KEEPALIVE on the sockets it uses. Beyond that it is up to the OS TCP
stack to deal with them - this is not something an application can do.

I suggest you have a careful look at your network setup - both at the OS
level and intervening routers/firewalls.[...]

--On Thursday, February 09, 2006 13:24:23 +0100 Robert Segall 
<roseg(at)apsis.ch> wrote:
[...][...][...]

So you are suggesting that the keepalive packages should be dealt
with by the system running the Pound service, not the backend server?
If so then it should be something to fix on Linux kernel level, am I
right?

   Eckard
[...]

On Thu, 2006-02-09 at 13:37 +0100, Eckard Koch wrote:
[...]

The way it works:

- the system has a default (respond or not to KEEPALIVE)
- an application may change that default; Pound explicitly tells the
system "do respond"
- the system deals with it - the packets are not even visible at the
application level (they are part of the TCP infrastructure)
- the client is only connected to the Pound server, so the back-end is
never involved at all

All of the above assumes you are talking about TCP KEEPALIVE.[...]

I guess my first assumption was wrong. The TCP keepalive packages, seem to
be a symptom but not the reason and they seem to be handled properly 
between
all three ends of the connection.

I've been looking deeper into the problem and have configured a less complex
setup for testing (no SSL, no firewalls, no virtual IP addresses). The
behavior of the WinXP subversion client remains the same.

It appears to me that it has something to do with the flow control
between client, proxy and server. The WinXP client seems to be unable
to digest the data at the speed the proxy is delivering. According to
ethereal, there are a lot of "TCP Window Full" and "TCP ZeroWindow" packages
followed by the "TCP Keep-Alive" dialog. In the end Pound seems to close
down the connection to the back-end server, flushing the remaining content
in its buffer to the client and then the connection to the client is closed.
[...]

On Mon, 2006-02-13 at 10:52 +0100, Eckard Koch wrote:[...]

Search the list for older messages re. network configuration (more
specifically size of network buffers). This is less to do with Pound and
more with the interaction between the WinXP and Linux TCP stacks.[...]

--On Monday, February 13, 2006 11:56:10 +0100 Robert Segall 
<roseg(at)apsis.ch> wrote:
[...][...][...]

Changing buffer sizes on Linux and on WinXP had no effect. The WinXP
client behavior is still the same, the checkout stops at about 13 MB.

What made a difference, however, was switching from kernel version
2.6.10 back to 2.4.26. I will check a recent 2.6 kernel to see if the
problem is still around.

Have you heart of any other problems with 2.6 kernel series?


[...]

On Mon, 2006-02-13 at 15:20 +0100, Eckard Koch wrote:[...]

No, can't say I heard, but a bug is always possible. In any case, I
doubt this has much to do with Pound, unless there was some intentional
change in the sockets behaviour which we missed.

Please let us know what you find out.[...]