Hi,

We are experiencing some strange problems with file-uploads through 
Pound. It seems that the time that Pound uses for it's incoming requests 
is too short for uploading files. For normal GET-requests or 
POST-requests using only plain forms, Pound works perfectly !

This is an example config for a site that needs uploads:

    Service "94.124.89.211_80__wepaintyou.massmovement.nl"
       HeadRequire "Host: wepaintyou.massmovement.nl"
       BackEnd
          Address 91.199.219.83
          Port 80
          Priority 5
          Timeout 300
       End
    End

Is it possible that "Timeout 300" is only used for the connection itself 
but individual requests have a different timeout ? Also, please note 
that uploads using Flash-sites nearly everytime succeed and it seems 
that https-uploads succeed as well...

Can someone help us ? Our client now wants to remove the loadbalancer as 
it is of "no use" to them... Can't let that happen :)

Best regards,
Marcus Smit

Hello,

MSDirect Internet Diensten - Support <support(at)msdirect.nl> (Fr 02 Dez 2011
07:49:29 CET):
> 
> We are experiencing some strange problems with file-uploads through
> Pound. It seems that the time that Pound uses for it's incoming
> requests is too short for uploading files. For normal GET-requests
> or POST-requests using only plain forms, Pound works perfectly !

For my understanding an file upload is a normal POST request, using a
the multipart/form encoding.

> This is an example config for a site that needs uploads:
> 
>    Service "94.124.89.211_80__wepaintyou.massmovement.nl"
>       HeadRequire "Host: wepaintyou.massmovement.nl"
>       BackEnd
>          Address 91.199.219.83
>          Port 80
>          Priority 5
>          Timeout 300
>       End
>    End
> 
> Is it possible that "Timeout 300" is only used for the connection
> itself but individual requests have a different timeout ? Also,
> please note that uploads using Flash-sites nearly everytime succeed
> and it seems that https-uploads succeed as well...

According to the docs the Timeout is the time pound waits for the
*response* from the backend. This should start counting as soon your
request (upload) is completed.

Did you watch the connection using tcpdump? Does the data stop flowing?
Does pound kill the connection, regardless receiving data from the
client?

-- 
Heiko :: dresden : linux : SCHLITTERMANN.de
GPG Key 48D0359B : 3061 CFBF 2D88 F034 E8D2 7E92 EE4E AC98 48D0 359B

Hi,

Thanks for your reply and sorry for my late response. I had no way of 
repoducing this problem until yesterday. It seems that the problem only 
occurs using Windows7. I tested using Firefox.

> For my understanding an file upload is a normal POST request, using a
> the multipart/form encoding.
That is how i understand it too :)

>> This is an example config for a site that needs uploads:
>>
>>     Service "94.124.89.211_80__wepaintyou.massmovement.nl"
>>        HeadRequire "Host: wepaintyou.massmovement.nl"
>>        BackEnd
>>           Address 91.199.219.83
>>           Port 80
>>           Priority 5
>>           Timeout 300
>>        End
>>     End
>>
>> Is it possible that "Timeout 300" is only used for the connection
>> itself but individual requests have a different timeout ? Also,
>> please note that uploads using Flash-sites nearly everytime succeed
>> and it seems that https-uploads succeed as well...
> According to the docs the Timeout is the time pound waits for the
> *response* from the backend. This should start counting as soon your
> request (upload) is completed.
>
> Did you watch the connection using tcpdump? Does the data stop flowing?
> Does pound kill the connection, regardless receiving data from the
> client?

In the attachement you will find the tcp stream for the POST request. As 
i see it, this is just a plan POST request that is reset with a FIN/ACK. 
But why the RST's ? The site went to production so i could not test it 
as it was but instead i had to create an extra alias as you can see in 
the dump (marcustest.massmovement.nl). Points to the same virtualhost in 
Apache.

I hope someone can help me solving this problem.
------------------------------------------------------------------------
Met vriendelijke groet,
Marcus Smit
MSDirect Internet Diensten
www.msdirect.nl <http://www.msdirect.nl>
Mobiel: 06 - 167 20 817

Hi,

>>> This is an example config for a site that needs uploads:
>>>
>>>     Service "94.124.89.211_80__wepaintyou.massmovement.nl"
>>>        HeadRequire "Host: wepaintyou.massmovement.nl"
>>>        BackEnd
>>>           Address 91.199.219.83
>>>           Port 80
>>>           Priority 5
>>>           Timeout 300
>>>        End
>>>     End
>>>
>>> Is it possible that "Timeout 300" is only used for the connection
>>> itself but individual requests have a different timeout ? Also,
>>> please note that uploads using Flash-sites nearly everytime succeed
>>> and it seems that https-uploads succeed as well...
>> According to the docs the Timeout is the time pound waits for the
>> *response* from the backend. This should start counting as soon your
>> request (upload) is completed.
>>
>> Did you watch the connection using tcpdump? Does the data stop flowing?
>> Does pound kill the connection, regardless receiving data from the
>> client?
>
> In the attachement you will find the tcp stream for the POST request. 
> As i see it, this is just a plan POST request that is reset with a 
> FIN/ACK. But why the RST's ? The site went to production so i could 
> not test it as it was but instead i had to create an extra alias as 
> you can see in the dump (marcustest.massmovement.nl). Points to the 
> same virtualhost in Apache.
>
> I hope someone can help me solving this problem.

I think i reduced the problem to two possible bugs in either Windows7 or 
Pound.

In a session that fails, i see a retransmission (by Windows) for a 
packet that already was ACKed. Ofcourse, this packet is not ACKed again, 
which for Windows is enough reason to close the session, making the 
fileupload fail. See the attached file for more info.

As you can see in packet 12 an ACK is sent for 8281, but the Windows 7 
workstation retransmits the packet in packet 13. Nothing happens for 10 
seconds, so Pound (or Linux, not sure) cleans up.

------------------------------------------------------------------------
Met vriendelijke groet,
Marcus Smit
MSDirect Internet Diensten
www.msdirect.nl <http://www.msdirect.nl>
Mobiel: 06 - 167 20 817

Hello,

 

The FIN/ACK is a TCP stack related issue, not pound. Do you have the most
recent patches and such for your host server?

 

I’ve seen this same behavior for a variety of hosts, not just Windows
clients. Plus, I’ve seen this on CentOS 4 and CentOS 5, and so far no relief
in the way of a TCP protocol bug fix. 

 

Do you have iptables running or any other software firewall on the pound host?
I have had iptables running on all of the boxes that host pound with this
error. I have not tried disabling iptables to see if it clears.

 

-- jake

 

 

From: MSDirect Internet Diensten - Support [mailto:support(at)msdirect.nl] 
Sent: Friday, December 16, 2011 2:48 AM
To: pound(at)apsis.ch
Subject: Re: [Pound Mailing List] Upload problem through Pound

 


Hi,




This is an example config for a site that needs uploads:
 
   Service "94.124.89.211_80__wepaintyou.massmovement.nl"
      HeadRequire "Host: wepaintyou.massmovement.nl"
      BackEnd
         Address 91.199.219.83
         Port 80
         Priority 5
         Timeout 300
      End
   End
 
Is it possible that "Timeout 300" is only used for the connection
itself but individual requests have a different timeout ? Also,
please note that uploads using Flash-sites nearly everytime succeed
and it seems that https-uploads succeed as well...

According to the docs the Timeout is the time pound waits for the
*response* from the backend. This should start counting as soon your
request (upload) is completed.
 
Did you watch the connection using tcpdump? Does the data stop flowing?
Does pound kill the connection, regardless receiving data from the
client?


In the attachement you will find the tcp stream for the POST request. As i see
it, this is just a plan POST request that is reset with a FIN/ACK. But why the
RST's ? The site went to production so i could not test it as it was but
instead i had to create an extra alias as you can see in the dump
(marcustest.massmovement.nl). Points to the same virtualhost in Apache.

I hope someone can help me solving this problem.


I think i reduced the problem to two possible bugs in either Windows7 or Pound.

In a session that fails, i see a retransmission (by Windows) for a packet that
already was ACKed. Ofcourse, this packet is not ACKed again, which for Windows
is enough reason to close the session, making the fileupload fail. See the
attached file for more info.

As you can see in packet 12 an ACK is sent for 8281, but the Windows 7
workstation retransmits the packet in packet 13. Nothing happens for 10
seconds, so Pound (or Linux, not sure) cleans up.

  _____  

Met vriendelijke groet,
Marcus Smit
MSDirect Internet Diensten
www.msdirect.nl
Mobiel: 06 - 167 20 817

 

Options on linux that have resolved network issues for me in the past:
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 0 > /proc/sys/net/ipv4/tcp_ecn

If they work you will want to use sysctl.conf to make them permanent, i.e.:
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_ecn=0

Other tuning here
http://www.bloke.com/linux/kernel/network.html

YMMV!

Joe

From: Jacob Anderson [mailto:jwa(at)beyond-ordinary.com]
Sent: Friday, December 16, 2011 11:37 AM
To: pound(at)apsis.ch
Subject: RE: [Pound Mailing List] Upload problem through Pound

Hello,

The FIN/ACK is a TCP stack related issue, not pound. Do you have the most
recent patches and such for your host server?

I’ve seen this same behavior for a variety of hosts, not just Windows
clients. Plus, I’ve seen this on CentOS 4 and CentOS 5, and so far no relief
in the way of a TCP protocol bug fix.

Do you have iptables running or any other software firewall on the pound host?
I have had iptables running on all of the boxes that host pound with this
error. I have not tried disabling iptables to see if it clears.

-- jake


From: MSDirect Internet Diensten - Support [mailto:support(at)msdirect.nl]
Sent: Friday, December 16, 2011 2:48 AM
To: pound(at)apsis.ch<mailto:pound(at)apsis.ch>
Subject: Re: [Pound Mailing List] Upload problem through Pound


Hi,


This is an example config for a site that needs uploads:



   Service "94.124.89.211_80__wepaintyou.massmovement.nl"

      HeadRequire "Host: wepaintyou.massmovement.nl"

      BackEnd

         Address 91.199.219.83

         Port 80

         Priority 5

         Timeout 300

      End

   End



Is it possible that "Timeout 300" is only used for the connection

itself but individual requests have a different timeout ? Also,

please note that uploads using Flash-sites nearly everytime succeed

and it seems that https-uploads succeed as well...

According to the docs the Timeout is the time pound waits for the

*response* from the backend. This should start counting as soon your

request (upload) is completed.



Did you watch the connection using tcpdump? Does the data stop flowing?

Does pound kill the connection, regardless receiving data from the

client?

In the attachement you will find the tcp stream for the POST request. As i see
it, this is just a plan POST request that is reset with a FIN/ACK. But why the
RST's ? The site went to production so i could not test it as it was but
instead i had to create an extra alias as you can see in the dump
(marcustest.massmovement.nl). Points to the same virtualhost in Apache.

I hope someone can help me solving this problem.

I think i reduced the problem to two possible bugs in either Windows7 or Pound.

In a session that fails, i see a retransmission (by Windows) for a packet that
already was ACKed. Ofcourse, this packet is not ACKed again, which for Windows
is enough reason to close the session, making the fileupload fail. See the
attached file for more info.

As you can see in packet 12 an ACK is sent for 8281, but the Windows 7
workstation retransmits the packet in packet 13. Nothing happens for 10
seconds, so Pound (or Linux, not sure) cleans up.
________________________________
Met vriendelijke groet,
Marcus Smit
MSDirect Internet Diensten
www.msdirect.nl<http://www.msdirect.nl>
Mobiel: 06 - 167 20 817

Hi,

Thanks for the responses.

I totally agree that the session initiation and maintainance is the 
responsibility of the operating system. This Centos 6.0 system is fully 
up-to-date with patches from one month ago.

I tested stopping iptables (had some weird ftp troubles some time ago 
which solved by restarting iptables), but with no luck,  I also tested 
the settings Joe Gooch suggested. Unfortunately no luck there.

Euhm... all this suddenly, the upload succeeds.. I did test some 
settings, but all changes are reverted.. Hm, how am i going to explain 
this to the customer...
------------------------------------------------------------------------
Met vriendelijke groet,
Marcus Smit
MSDirect Internet Diensten
www.msdirect.nl <http://www.msdirect.nl>
Mobiel: 06 - 167 20 817

Op 16-12-2011 19:27, Joe Gooch schreef:
>
> Options on linux that have resolved network issues for me in the past:
>
> echo 0 > /proc/sys/net/ipv4/tcp_timestamps
>
> echo 0 > /proc/sys/net/ipv4/tcp_ecn
>
> If they work you will want to use sysctl.conf to make them permanent, 
> i.e.:
>
> net.ipv4.tcp_timestamps=0
>
> net.ipv4.tcp_ecn=0
>
> Other tuning here
>
> http://www.bloke.com/linux/kernel/network.html
>
> YMMV!
>
> Joe
>
> *From:*Jacob Anderson [mailto:jwa(at)beyond-ordinary.com]
> *Sent:* Friday, December 16, 2011 11:37 AM
> *To:* pound(at)apsis.ch
> *Subject:* RE: [Pound Mailing List] Upload problem through Pound
>
> Hello,
>
> The FIN/ACK is a TCP stack related issue, not pound. Do you have the 
> most recent patches and such for your host server?
>
> I’ve seen this same behavior for a variety of hosts, not just Windows 
> clients. Plus, I’ve seen this on CentOS 4 and CentOS 5, and so far no 
> relief in the way of a TCP protocol bug fix.
>
> Do you have iptables running or any other software firewall on the 
> pound host? I have had iptables running on all of the boxes that host 
> pound with this error. I have not tried disabling iptables to see if 
> it clears.
>
> -- jake
>
> *From:*MSDirect Internet Diensten - Support [mailto:support(at)msdirect.nl]
> *Sent:* Friday, December 16, 2011 2:48 AM
> *To:* pound(at)apsis.ch <mailto:pound(at)apsis.ch>
> *Subject:* Re: [Pound Mailing List] Upload problem through Pound
>
>
> Hi,
>
>         This is an example config for a site that needs uploads:
>
>           
>
>             Service "94.124.89.211_80__wepaintyou.massmovement.nl"
>
>                HeadRequire "Host: wepaintyou.massmovement.nl"
>
>                BackEnd
>
>                   Address 91.199.219.83
>
>                   Port 80
>
>                   Priority 5
>
>                   Timeout 300
>
>                End
>
>             End
>
>           
>
>         Is it possible that "Timeout 300" is only used for the connection
>
>         itself but individual requests have a different timeout ? Also,
>
>         please note that uploads using Flash-sites nearly everytime succeed
>
>         and it seems that https-uploads succeed as well...
>
>     According to the docs the Timeout is the time pound waits for the
>
>     *response* from the backend. This should start counting as soon your
>
>     request (upload) is completed.
>
>       
>
>     Did you watch the connection using tcpdump? Does the data stop flowing?
>
>     Does pound kill the connection, regardless receiving data from the
>
>     client?
>
>
> In the attachement you will find the tcp stream for the POST request. 
> As i see it, this is just a plan POST request that is reset with a 
> FIN/ACK. But why the RST's ? The site went to production so i could 
> not test it as it was but instead i had to create an extra alias as 
> you can see in the dump (marcustest.massmovement.nl). Points to the 
> same virtualhost in Apache.
>
> I hope someone can help me solving this problem.
>
>
> I think i reduced the problem to two possible bugs in either Windows7 
> or Pound.
>
> In a session that fails, i see a retransmission (by Windows) for a 
> packet that already was ACKed. Ofcourse, this packet is not ACKed 
> again, which for Windows is enough reason to close the session, making 
> the fileupload fail. See the attached file for more info.
>
> As you can see in packet 12 an ACK is sent for 8281, but the Windows 7 
> workstation retransmits the packet in packet 13. Nothing happens for 
> 10 seconds, so Pound (or Linux, not sure) cleans up.
>
> ------------------------------------------------------------------------
>
> Met vriendelijke groet,
> Marcus Smit
> MSDirect Internet Diensten
> www.msdirect.nl <http://www.msdirect.nl>
> Mobiel: 06 - 167 20 817
>