hi,
we have tried to migrate from pound 2.4 to pound 2.6.
We setup a new system with centos 6.2 x86_64.
Before compiling pound, we have installed pcre, gperftools, openssl:
rpm -qa pcre*
pcre-7.8-3.1.el6.x86_64
pcre-devel-7.8-3.1.el6.x86_64
rpm -qa gperftools*
gperftools-libs-2.0-3.el6.2.x86_64
gperftools-devel-2.0-3.el6.2.x86_64
rpm -qa *ssl*
openssl-devel-1.0.0-20.el6_2.5.x86_64
openssl-1.0.0-20.el6_2.5.x86_64
Afterwards we compiled pound-2.6 this way:
./configure --enable-cert1l --enable-pcreposix --enable-tcmalloc
--disable-super
make
make install
It was compiled without any errors.
We use this configuration:
User "pound"
Group "pound"
Alive 120
Control "/var/run/pound/ctl_socket"
# syslog facility, see /etc/syslog.conf
LogFacility local5
# apache combined like log format - with service and backend
LogLevel 5
# close connection if client doesnt send anything for this period
Client 60
# how long pound will wait for response from the backend server
TimeOut 2060
##########################################
ListenHTTP
Address 212.x.x.x
Port 80
# Remove the X-SSL-Request header from incoming connections
# to prevent hackers from spoofing it
HeadRemove "X-SSL-Request"
AddHeader "X-Requested-Port: 80"
RewriteLocation 0
Service
BackEnd
Address 172.25.x.x
Port 80
End
Session
Type COOKIE
ID "PHPSESSID"
TTL 28800
End
End
End
ListenHTTPS
Address 212.x.x.x
Port 443
Cert "/etc/pki/tls/certs/cert.pem"
Ciphers
"ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL"
RewriteLocation 2
# Add an extra header to tell the server that
# the external connection is secure
AddHeader "X-SSL-Request: 1"
#AddHeader "X-Requested-Port: 443"
Service
BackEnd
Address 172.25.x.x
Port 80
End
End
End
We did some stress test with the apache benchmark tool:
ab -n 5000 -c 500 http://212.x.x.x/index.html
In our production environment, we have peaks of 400 connections per second
(awk over the pound log file and count the lines for an equal time stamp).
This test failed, in /var/log/messages we found this messages:
loadbalancer2 kernel: possible SYN flooding on port 80. Sending cookies.
We found out that this comes from a (d)dos attack prevention feature of
centos.
The only way to disable this feature, that we found out was this:
echo 0 > /proc/sys/net/ipv4/tcp_syncookies
The stress test with apache benchmark tool were successful after disabling
this feature.
Next step was to migrate the official ip to the new system.
When the heavy user traffic started, the website delivery slowed down.
It took up to some minutes before the browser could render the sites.
On the new system every looked fine, no high cpu or memory usage.
A tail -f /var/log/pound.log printed a lot of requests out, but no problems
or error messages.
When we looked at the connections with netstat, we saw a lot of filled
recieve queues.
Normal 0 21
false false false DE X-NONE
X-NONE
MicrosoftInternetExplorer4
tcp 120 0 212.x.x.x:443
62.x.x.x:38443 VERBUNDEN 29530/pound
tcp 352 0 212.x.x.x:80
194.x.x.x:42835 VERBUNDEN 29530/pound
tcp 137 0 212.x.x.x:443
81.x.x.x:28257 CLOSE_WAIT 29530/pound
tcp 109 0 212.x.x.x:443
217.x.x.x:1239 VERBUNDEN 29530/pound
tcp 0 8606 212.x.x.x:80
72.x.x.x:45921 CLOSE_WAIT 29530/pound
tcp 0 8606 212.x.x.x:80
213.x.x.x:50557 CLOSE_WAIT 29530/pound
tcp 862 0 212.x.x.x:80
62.x.x.x:32903 VERBUNDEN
29530/pound
Normal 0 21
false false false DE X-NONE
X-NONE
MicrosoftInternetExplorer4
tcp 0 8606 212.x.x.x:80
213.x.x.x:54691 CLOSE_WAIT 29530/pound
Also the filled send queue with 8606 Bytes looks strange.
CPU and memory usage where not too high all over the time. Also the load
was at 0.015.
The pound log shows entries like this:
Jun 14 08:20:00 loadbalancer2 pound: (7f1baad52700) error copy server cont:
Connection reset by peer
Jun 14 08:20:00 loadbalancer2 pound: web.example.com 212.x.x.x - -
[14/Jun/2012:08:20:00 +0200] "GET /exampe/service.gif HTTP/1.0" 200 807
"http://www.example.com/" "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30;
.NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729;
.NET4.0C; .NET4.0E)" (- -> 172.25.x.x:8080) 0.002 sec
Jun 14 08:20:00 loadbalancer2 pound: web.example.com 213.x.x.x - -
[14/Jun/2012:08:19:59 +0200] "GET /example.php HTTP/1.1" 200 15677 ""
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" (- ->
192.168.105.208:80) 0.703 sec
We have no clue, what the reason for the slow answers could be.
Do you have any hints, guesses or have you seen some similar behavior?
Thanks for reading
Matthias [...]
|
