|
/
Zope
/
Apsis
/
Pound Mailing List
/
Archive
/
2010
/
2010-03
/
Connection Timeouts
[
Problem with pound SSL / "Anne Moore" ... ]
[
Re: [Pound Mailing List] TPROXY / Falk ... ]
Connection Timeouts
David W King <davidwk(at)goshen.edu> |
2010-03-16 16:35:06 |
[ FULL ]
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I started a new job two weeks ago and with it I inherited a server
running pound as a front end for tomcat. I am a _complete_ pound noob
but it worked fine until about a week ago when my users started
reporting that intermittently they will get timeout messages in their
browsers when trying to access this server. About the same time I
started getting a lot of pound errors in syslog:
[...]
Some details about my environment:
$ pound -V
starting...
Version 2.4.3
Configuration switches:
--enable-cert1l
Exiting...
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 9.04
Release: 9.04
Codename: jaunty
$ cat /etc/pound/pound.cfg
######################################################################
## global options:
User "www-data"
Group "www-data"
#RootJail "/chroot/pound"
## Logging: (goes to syslog by default)
## 0 no logging
## 1 normal
## 2 extended
## 3 Apache-style (common log format)
LogLevel 2
## check backend every X secs:
Alive 30
## use hardware-accelleration card supported by openssl(1):
#SSLEngine "<hw>"
# poundctl control socket
Control "/var/run/pound/poundctl.socket"
TimeOut 30
######################################################################
## listen, redirect and ... to:
## redirect all requests on port 8080 ("ListenHTTP") to the local
webserver (see "Service" below):
ListenHTTPS
Address <hostname removed>
Port 443
Cert "/etc/pound/<hostname removed>.pem"
## allow PUT and DELETE also (by default only GET, POST and HEAD)?:
xHTTP 0
Service
BackEnd
Address 127.0.0.1
Port 8080
End
End
End
## EOF
I hope I'm just missing something obvious here. Can anyone else spot it?
Thanks!
- --
David King
Goshen College ITS
davidwk(at)goshen.edu
574-535-7726
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkufpSoACgkQH+/Vg7DylXYU1gCgjD/PHkSl7de5SH9OsSTW+E31
2AYAoJR0Tkhmf8221MEZlvnhWyp/iad3
=XwOh
-----END PGP SIGNATURE-----
|
|
|
Re: [Pound Mailing List] Connection Timeouts
"Simon Matter" <simon.matter(at)invoca.ch> |
2010-03-16 17:02:19 |
[ FULL ]
|
> -----BEGIN PGP SIGNED MESSAGE-----[...][...][...]
Could it be that 30 seconds is a bit small here? Depending on the kind of
application realistic values can require quite large timeouts, up to a
couple of minutes. Maybe you webapp logs will get you some more info.
Regards,
Simon
[...]
|
|
|
Re: [Pound Mailing List] Connection Timeouts
David W King <davidwk(at)goshen.edu> |
2010-03-16 17:10:17 |
[ FULL ]
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
The timeouts had actually been set down to 15. When I started getting
the errors I had the same thought so I upped them to 30 without success.
I'll try going higher both here and (if I can figure out how ;) in tomcat.
David King
Goshen College ITS
davidwk(at)goshen.edu
574-535-7726
Simon Matter wrote:[...]
>>>> Mar 15 21:53:04 login pound: (7f419410f950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 21:53:09 login pound: (7f4194150950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:02:01 login pound: (7f419410f950) error read from
>>>> 199.8.237.104: Connection timed out
>>>> Mar 15 22:02:40 login pound: (7f4194191950) error read from
>>>> 199.8.237.104: Connection timed out
>>>> Mar 15 22:13:12 login pound: (7f4194191950) error read from
>>>> 199.8.237.189: Connection timed out
>>>> Mar 15 22:15:02 login pound: (7f419410f950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:15:03 login pound: (7f4194191950) error read from
>>>> 199.8.237.189: Connection timed out
>>>> Mar 15 22:20:12 login pound: (7f419410f950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:21:54 login pound: (7f41941d2950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:22:48 login pound: (7f41941d2950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:23:24 login pound: (7f4194150950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:29:12 login pound: (7f4194150950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:41:40 login pound: (7f4194150950) error read from
>>>> 98.206.89.134: Connection timed out
>>>> Mar 15 22:43:23 login pound: (7f4194150950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 22:43:23 login pound: (7f4194191950) error read from
>>>> 199.8.239.245: Connection timed out
>>>> Mar 15 23:02:53 login pound: (7f41941d2950) error read from
>>>> 199.8.237.235: Connection timed
out[...][...][...][...][...][...]
To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.[...][...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkufrWkACgkQH+/Vg7DylXazWwCfV1lnjFpE3DRxG3MBhtlZBoNT
WBYAoIqA0gcGpxjwZRi8cososvIU09z4
=g3QU
-----END PGP SIGNATURE-----
|
|
|
RE: [Pound Mailing List] Connection Timeouts
"Jacob Anderson" <jwa(at)beyond-ordinary.com> |
2010-03-16 17:16:54 |
[ FULL ]
|
Hello David,
First step - try direct routing with a static NAT rule to jump your users
directly to one back end. If they timeout, then it's a back end issue. If it
works great, then read on.
Next, check your routing. The back end may have a firewall running that is
not allowing the traffic on its local interface? I see that you are running
against 127.0.0.1, so next check the web logs.
/path_to_tomcat/logs/catalina.out
/path_to_tomcat/logs/tomcat_xx-yy-zzzz.txt
If you are seeing connections entering tomcat, but not returning (in the
log), then you have a container problem. Check the catalina.out file to see
if there are any errors on the command line output for the tomcat process.
Lastly, if your application needs to hit a database, check that too. Make
sure that you can route and resolve the database. If you are using any NAMES
for hosts, then check your DNS. DNS name resolution is the #1 culprit for
poor application performance. Try to use IP addresses if you can, but that
makes administration harder.
Good luck! We use tomcat behind pound and it works great.
[...]
|
|
|
Re: [Pound Mailing List] Connection Timeouts
David W King <davidwk(at)goshen.edu> |
2010-03-16 20:37:04 |
[ FULL ]
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Thanks for the response, Jacob. I would like to avoid NATting for two
reasons 1) the problem only occurs 5% of the time or less so if I did
NAT and it worked, it wouldn't really prove anything since _most_ of the
time it works anyway and 2) the tomcat app is a single sign on thing for
a bunch of our web apps. The web apps (which I don't have access to
change) are directing the clients to an https connection but the NAT
would connect them directly to the http tomcat connection (bypassing
pound and, therefore, SSL).
I checked my tomcat/catalina logs but, unfortunately, tomcat isn't
logging connections. I'll get that fixed but not today.
As far as external services go, it relies on DNS and LDAP both of which
are functioning very well. One thing we have noticed that may be of
interest is that the timeout error only occurs when the initial login
page is loading, it never occurs (that we know of) when the login form
is submitted. That points to a tomcat issue but it may be pertinent
here as well.
David King
Goshen College ITS
davidwk(at)goshen.edu
574-535-7726
Jacob Anderson
wrote:[...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]
- --
To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
- --
To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkuf3d8ACgkQH+/Vg7DylXY/pgCeJF0iCaBHNn4NE+ECXt7gRLQP
m3gAn2gydl+D2ELEowm9egJ2ZKHvPkqJ
=f2zT
-----END PGP SIGNATURE-----
|
|
|
Re: [Pound Mailing List] Connection Timeouts
David W King <davidwk(at)goshen.edu> |
2010-03-17 15:38:34 |
[ FULL ]
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
One other thought on that, the timeouts don't seem to be happening
exclusively right after _server_ startup. In fact, they seem to
increase in frequency (although not significantly so) as the server's
uptime increases.
It may have something to do with _session_ startup, we've only observed
the timeouts when the initial login page is loading, never (that we have
observed) when the login page/credentials are submitted, the
authentication token is returned or when the other servers check the
authentication token that the client got from this server. The only
thing is that tomcat (or the apps tomcat is running) shouldn't have to
do DNS lookups _every_ time a new session is started, that's the whole
point of caching. Also, LDAP shouldn't be involved in serving up that
initial page (whether it is or not is a question for the CAS mailing
list). So I guess the whole question here is what else is happening on
session startup that's taking so long (this whole thing is seeming more
and more like a CAS problem)?
David King
Goshen College ITS
davidwk(at)goshen.edu
574-535-7726
Jacob Anderson
wrote:[...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]
>>> Mar 15 21:53:04 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 21:53:09 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:02:01 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 22:02:40 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 22:13:12 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 22:15:02 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 22:15:03 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 22:20:12 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 22:21:54 login pound: (7f41941d2950) error read
from[...][...]
>>> Mar 15 22:22:48 login pound: (7f41941d2950) error read
from[...][...]
>>> Mar 15 22:23:24 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:29:12 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:41:40 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:43:23 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:43:23 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 23:02:53 login pound: (7f41941d2950) error read
from[...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]
- --
To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkug6WkACgkQH+/Vg7DylXYHJgCdFzg2GVJdsl7hlxU5yo35NLf/
TLUAn2cobnSDd74icwgCVWMqSW0e4Vh2
=ANyn
-----END PGP SIGNATURE-----
|
|
|
Re: [Pound Mailing List] Connection Timeouts
David W King <davidwk(at)goshen.edu> |
2010-03-17 15:39:10 |
[ FULL ]
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Thanks, I'll try that. I had been hesitant to set the timeouts too high
because our users aren't going to want to wait 60-120 seconds for a page
to load and if they get impatient and go do something else we might as
well timeout the session and recycle the resources. I'll increase it,
though, and see what it does.
Our SSO is CAS from Ja-Sig. We're not horribly happy with it right now
for a number of reasons; I assume you're using CAMS? Do you like it?
David King
Goshen College ITS
davidwk(at)goshen.edu
574-535-7726
Jacob Anderson
wrote:[...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]
>>> Mar 15 21:53:09 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:02:01 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 22:02:40 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 22:13:12 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 22:15:02 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 22:15:03 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 22:20:12 login pound: (7f419410f950) error read
from[...][...]
>>> Mar 15 22:21:54 login pound: (7f41941d2950) error read
from[...][...]
>>> Mar 15 22:22:48 login pound: (7f41941d2950) error read
from[...][...]
>>> Mar 15 22:23:24 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:29:12 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:41:40 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:43:23 login pound: (7f4194150950) error read
from[...][...]
>>> Mar 15 22:43:23 login pound: (7f4194191950) error read
from[...][...]
>>> Mar 15 23:02:53 login pound: (7f41941d2950) error read
from[...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]
- --
To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkug6Y4ACgkQH+/Vg7DylXboLACfeSoyNk3vrVRjnl/BpiO6nhYm
yuwAoJCJfTAKJ2o5S4CQNQdP5QlV83SW
=TKkC
-----END PGP SIGNATURE-----
|
|
|
Re: [Pound Mailing List] Connection Timeouts
David W King <davidwk(at)goshen.edu> |
2010-03-19 15:23:26 |
[ FULL ]
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I've been doing a little more digging and here's what I've found:
1) The error fragment "error read from" only appears in the Pound source
code once. It's http.c on line 601. My C is a little rusty but what I
make of the source code is that it's reading the headers from a client
request at that point and the error is that it doesn't get any headers.
2) The only browser that I _know_ has experienced this problem is
Firefox 3.5.8 and higher. Others may be experiencing it but I haven't
gotten any confirmation that they are.
Is it possible that Firefox is sending a mal-formed request that is
confusing pound?
David King
Goshen College ITS
davidwk(at)goshen.edu
574-535-7726
David W King
wrote:[...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...][...]
>>> Hello David,
>>> First step - try direct routing with a static NAT rule to jump
your users
>>> directly to one back end. If they timeout, then it's a back end
issue. If[...]
>>> works great, then read on.
>>> Next, check your routing. The back end may have a firewall running
that is
>>> not allowing the traffic on its local interface? I see that you
are[...]
>>> against 127.0.0.1, so next check the web logs.
>>> /path_to_tomcat/logs/catalina.out
>>> /path_to_tomcat/logs/tomcat_xx-yy-zzzz.txt
>>> If you are seeing connections entering tomcat, but not returning
(in the
>>> log), then you have a container problem. Check the catalina.out
file to[...]
>>> if there are any errors on the command line output for the tomcat
process.
>>> Lastly, if your application needs to hit a database, check that
too. Make
>>> sure that you can route and resolve the database. If you are using
any[...]
>>> for hosts, then check your DNS. DNS name resolution is the #1
culprit for
>>> poor application performance. Try to use IP addresses if you can,
but that
>>> makes administration harder.
>>> Good luck! We use tomcat behind pound and it works great.
>>> -- jake[...]
>>> -----Original Message-----
>>> From: David W King [mailto:davidwk(at)goshen.edu]
>>> Sent: Tuesday, March 16, 2010 8:35 AM
>>> To: pound(at)apsis.ch
>>> Subject: [Pound Mailing List] Connection Timeouts
>>> I started a new job two weeks ago and with it I inherited a server
>>> running pound as a front end for tomcat. I am a _complete_ pound
noob
>>> but it worked fine until about a week ago when my users started
>>> reporting that intermittently they will get timeout messages in
their
>>> browsers when trying to access this server. About the same time I
>>> started getting a lot of pound errors in syslog:
>>>> Mar 15 21:53:04 login pound: (7f419410f950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 21:53:09 login pound: (7f4194150950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:02:01 login pound: (7f419410f950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:02:40 login pound: (7f4194191950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:13:12 login pound: (7f4194191950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:15:02 login pound: (7f419410f950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:15:03 login pound: (7f4194191950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:20:12 login pound: (7f419410f950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:21:54 login pound: (7f41941d2950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:22:48 login pound: (7f41941d2950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:23:24 login pound: (7f4194150950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:29:12 login pound: (7f4194150950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:41:40 login pound: (7f4194150950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:43:23 login pound: (7f4194150950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 22:43:23 login pound: (7f4194191950) error read
from[...]
>>> Connection timed out
>>>> Mar 15 23:02:53 login pound: (7f41941d2950) error read
from[...]
>>> Connection timed out
>>> Some details about my environment:
>>> $ pound -V
>>> starting...
>>> Version 2.4.3
>>> Configuration switches:
>>> --enable-cert1l
>>> Exiting...
>>> $ lsb_release -a
>>> No LSB modules are available.
>>> Distributor ID: Ubuntu
>>> Description: Ubuntu 9.04
>>> Release: 9.04
>>> Codename: jaunty
>>> $ cat /etc/pound/pound.cfg
>>>
######################################################################
>>> ## global options:
>>> User "www-data"
>>> Group "www-data"
>>> #RootJail "/chroot/pound"
>>> ## Logging: (goes to syslog by default)
>>> ## 0 no logging
>>> ## 1 normal
>>> ## 2 extended
>>> ## 3 Apache-style (common log format)
>>> LogLevel 2
>>> ## check backend every X secs:
>>> Alive 30
>>> ## use hardware-accelleration card supported by openssl(1):
>>> #SSLEngine "<hw>"
>>> # poundctl control socket
>>> Control "/var/run/pound/poundctl.socket"
>>> TimeOut 30[...]
>>>
######################################################################
>>> ## listen, redirect and ... to:
>>> ## redirect all requests on port 8080 ("ListenHTTP") to the local
>>> webserver (see "Service" below):
>>> ListenHTTPS
>>> Address <hostname removed>
>>> Port 443
>>> Cert "/etc/pound/<hostname removed>.pem"
>>> ## allow PUT and DELETE also (by default only GET, POST and
HEAD)?:
>>> xHTTP 0
>>> Service
>>> BackEnd
>>> Address 127.0.0.1
>>> Port 8080
>>> End
>>> End
>>> End
>>> ## EOF
>>> I hope I'm just missing something obvious here. Can anyone else
spot it?
>>> Thanks![...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkujiN4ACgkQH+/Vg7DylXYWTwCfbZ/eW4jg4pmoY9zpqTmJ76s+
BDcAnRssfDAG3L8aGXVikI9KBxiQ7uXD
=HZXi
-----END PGP SIGNATURE-----
|
|
|
|
