|
/
Zope
/
Apsis
/
Pound Mailing List
/
Archive
/
2005
/
2005-09
/
Performance problem with JSP's
[
Problem with Servlet Redirects / "Sarah ... ]
[
BackendServer access over HTTPS / KOZMAN Balint ... ]
Performance problem with JSP's
Andrew Hughes <ahughes(at)lisasoft.com> |
2005-09-07 09:57:41 |
[ FULL ]
|
Hi All,
I'm using pound to balance three workers at the backend... I have
noticed that pound is quicker than an apache rewrite balancing in every
instance, but one. Images from my application and html pages are
delivered between 1.4 and 2.0 times faster! Yay for pound!
However, when I begin hitting backend jsp's the performance of pound
slows massively. Im not talking about the actual response... I expect
jsp's to be slow... but compared to an apache rewrite pound is dead in
the water! Pound is around 70% slower than performing an apache
rewrite, and normally its 1.4 to 2.0 times faster!!! This is grinding
my application into the ground and I really need to fix things.
I noticed that the jsp application server Tomcat 5.5 was setting a
cookie, so I turned this off. It made no difference.
I made the most simple of jsp's just to say "Hello World". This has NO
session access and still runs much faster with Apache ReWrite
balancing. (code below)
<%(at) page contentType =
"text/html;charset=windows-1252"%>
<%(at) page import = "java.net.*"%>
<%(at) page import = "javax.servlet.*" %>
<html>
<head><title>TerraMap</title></head>
<body>Hello World from:
<%=InetAddress.getLocalHost().getHostName()%></body>
</html>
Further more, the html that is delivered by the jsp... I saved the html
source in a html file, and performed loadtesting on the file. This sent
me back to being twice as fast as an apache rewrite balance. So my
problems
dont appear to be anything to do with the html content I am returning,
but possibly in the jsp's http transmission....
My pound.conf:
ListenHTTP 192.168.83.135,81
CheckURL 0
LogLevel 2
UrlGroup "/serviceOne/.*"
Session IP 0
LogLevel 2
BackEnd huey.localdomain.com.au,8080,1
BackEnd louie.localdomain.com.au,8080,1
BackEnd dewey.localdomain.com.au,8080,1
EndGroup
UrlGroup "/serviceTwo/.*"
Session IP 0
BackEnd huey.localdomain.com.au,80,1
BackEnd louie.localdomain.com.au,80,1
BackEnd dewey.localdomain.com.au,80,1
EndGroup
If anyone has any idea why this is occuring I would really appreciate
some advice!
Many thanks in advance,
Andrew
|
|
|
|
|
Re: [Pound Mailing List] Performance problem with JSP's
david walters <dwalters1(at)gmail.com> |
2005-09-07 11:34:09 |
[ FULL ]
|
My suspicion goes towards the http header - could a rogue content-length
cause this kind of perfromance problem with pound?
/David
On 9/7/05, Andrew Hughes <ahughes(at)lisasoft.com> wrote: [...]
|
|
|
|
|
Re: [Pound Mailing List] Performance problem with JSP's
Andrew Hughes <ahughes(at)lisasoft.com> |
2005-09-08 06:33:59 |
[ FULL ]
|
Here I go again!!!
I've done some more testing and found that the slow performace is not
limited to jsp's only!
I actually did a wget on the helloworld.jsp... saved the (html)
contents as helloworld.html. I then performed a load test on
helloworld.html.... and found it to also be VERY slow!
The headers returned from a fast html page (where pound DOMINATES over
the apache rewrtie alternative) and the slow header are below:
Fast HTTP response header:
Slow HTTP response header (helloworld.html):
HTTP/1.1 200 OK
Server:
Apache-Coyote/1.1
ETag:
W/"24433-1126065997000"
Last-Modified:
Wed, 07 Sep 2005 04:06:37 GMT
Content-Type:
text/html
Content-Length:
24433
Date:
Thu, 08 Sep 2005 02:25:17 GMT
HTTP/1.1 200 OK
Server:
Apache-Coyote/1.1
ETag:
W/"132-1126144756000"
Last-Modified:
Thu, 08 Sep 2005 01:59:16 GMT
Content-Type:
text/html
Content-Length:
132
Date:
Thu, 08 Sep 2005 02:33:21 GMT
The source for the slow html is:
<html>
<head><title>Test</title></head>
<body>Hello World</body>
</html>
Also, when the html is hosted on both Apache and Tomcat as the backend
web server, the results is the same. So this does not appear to be an
issue regarding Tomcat.
I cant see anything in the header that would cause this to run soooo
slow! being able to serve 23/sec of the simple helloworld.html and
serving the more complicated, larger and backend service calling html,
with a fast response of 173/sec indicates that pound is really not
happy about something!!!!
I'm completely out of ideas now and would appreciate anyone who has any
ideas.
Thanks for reading,
Andrew
david walters wrote:
My suspicion goes towards the http header - could a rogue
content-length cause this kind of perfromance problem with pound?
Since I removed the jsp and replaced it with html, I doubt this is a
content-length issue. Plus the problem exists in both Apache and Tomcat
webservers.
/David
On 9/7/05, Andrew
Hughes <ahughes(at)lisasoft.com>
wrote:
Hi
All,
I'm using pound to balance three workers at the backend... I have
noticed that pound is quicker than an apache rewrite balancing in every
instance, but one. Images from my application and html pages are
delivered between 1.4 and 2.0 times faster! Yay for pound!
However, when I begin hitting backend jsp's the performance of pound
slows massively. Im not talking about the actual response... I expect
jsp's to be slow... but compared to an apache rewrite pound is dead in
the water! Pound is around 70% slower than performing an apache
rewrite, and normally its 1.4 to 2.0 times faster!!! This is grinding
my application into the ground and I really need to fix things.
I noticed that the jsp application server Tomcat 5.5 was setting a
cookie, so I turned this off. It made no difference.
I made the most simple of jsp's just to say "Hello World". This has NO
session access and still runs much faster with Apache ReWrite
balancing. (code below)
<%(at) page contentType =
"text/html;charset=windows-1252"%>
<%(at) page import = "java.net.*"%>
<%(at) page import = "javax.servlet.*
" %>
<html>
<head><title>TerraMap</title></head>
<body>Hello World from:
<%=InetAddress.getLocalHost().getHostName()%></body>
</html>
Further more, the html that is delivered by the jsp... I saved the html
source in a html file, and performed loadtesting on the file. This sent
me back to being twice as fast as an apache rewrite balance. So my
problems dont appear to be anything to do with the html content I am
returning, but possibly in the jsp's http transmission....
My pound.conf:
ListenHTTP 192.168.83.135,81
CheckURL 0
LogLevel 2
UrlGroup "/serviceOne/.*"
Session IP 0
LogLevel 2
BackEnd huey.localdomain.com.au,8080,1
BackEnd
louie.localdomain.com.au,8080,1
BackEnd dewey.localdomain.com.au,8080,1
EndGroup
UrlGroup "/serviceTwo/.*"
Session IP 0
BackEnd huey.localdomain.com.au,80,1
BackEnd
louie.localdomain.com.au,80,1
BackEnd dewey.localdomain.com.au,80,1
EndGroup
If anyone has any idea why this is occuring I would really appreciate
some advice!
Many thanks in advance,
Andrew[...]
|
|
|
|
|
Re: [Pound Mailing List] Performance problem with JSP's
"Simon Matter" <simon.matter(at)ch.sauter-bc.com> |
2005-09-08 06:56:57 |
[ FULL ]
|
> Here I go again!!![...]
Did you try tracing pound with something like strace to see what it is
doing? May this will tell us what's wrong.
Simon
[...]
|
|
|
Re: [Pound Mailing List] Performance problem with JSP's
Ted Dunning <tdunning(at)veoh.com> |
2005-09-08 17:47:51 |
[ FULL ]
|
Look at the content length. Your file is very short which is what Pound
is saying. If the server on the other side is saying that the content
is very long, but it really isn't, then Pound probably has to wait for
the end of file to know how short the file is. This could easily cause
major problems with throughput.
Andrew Hughes wrote:
[...]
[...]
|
|
|
|
|
Re: [Pound Mailing List] Performance problem with JSP's
Andrew Hughes <ahughes(at)lisasoft.com> |
2005-09-09 09:13:32 |
[ FULL ]
|
Thanks guys,
Ted Dunning wrote:
Look at the content length. Your file is very short which is what
Pound is saying. If the server on the other side is saying that the
content is very long, but it really isn't, then Pound probably has to
wait for the end of file to know how short the file is. This could
easily cause major problems with throughput.
Nope, the Fast and Slow headers are from a different URL/File, so the
differing lengths you see below aren't the issue. By using a larger
file as the faster example, I can rule out network overheads.
I'm using ethereal to view the differences between the apache rewrite
and pound tcp transmissions.... the major difference is that apache
sends the HTTP Header in the same packet as the html text. However,
Pound sends the header.... waits (---insert the overhead here---),
keeps waiting.... for client ACKnowledgement to arrive and then sends
the content (html/text).
Furthermore... I have a FAST example of the header being sent in a
seperate packet before the content packet. However pound does not wait
for an ACK that the header has been recieved! It immediately sends the
content.
Why would it do this?
Thanks very much for reading!
Below is the Ethereal log showing all the data transmissions....
Frame 13.... the incoming HTTP Header.....
No. Time
Source Destination Protocol Info
13 0.027730 pound.mydomain.com
client.mydomain.com HTTP HTTP/1.1 200 OK
Frame 13 (258 bytes on wire, 258 bytes captured)
Arrival Time: Sep 9, 2005 11:58:45.097780000
Time delta from previous packet: 0.001602000
seconds **** NOTE that this is very
low!
Time since reference or first frame: 0.027730000
seconds
Frame Number: 13
Packet Length: 258 bytes
Capture Length: 258 bytes
Protocols in frame: eth:ip:tcp:http
Ethernet II, Src: Intel_56:f9:a1 (00:d0:b7:56:f9:a1), Dst:
client.mydomain.com (00:13:20:59:47:e3)
Destination: client.mydomain.com (00:13:20:59:47:e3)
Source: Intel_56:f9:a1 (00:d0:b7:56:f9:a1)
Type: IP (0x0800)
Internet Protocol, Src: pound.mydomain.com
(pound.mydomain.com), Dst: client.mydomain.com (client.mydomain.com)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00:
Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint:
Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 244
Identification: 0xe9e9 (59881)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0x27c8 [correct]
Source: pound.mydomain.com (pound.mydomain.com)
Destination: client.mydomain.com (client.mydomain.com)
Transmission Control Protocol, Src Port: 81 (81), Dst
Port: 2056 (2056), Seq: 1, Ack: 511, Len: 204
Source port: 81 (81)
Destination port: 2056 (2056)
Sequence number: 1 (relative sequence number)
Next sequence number: 205 (relative sequence number)
Acknowledgement number: 511 (relative ack number)
Header length: 20 bytes
Flags: 0x0018 (PSH, ACK)
0... .... = Congestion Window Reduced (CWR): Not
set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...1 .... = Acknowledgment: Set
.... 1... = Push: Set
.... .0.. = Reset: Not set
.... ..0. = Syn: Not set
.... ...0 = Fin: Not set
Window size: 6432
Checksum: 0x7f2f [correct]
Hypertext Transfer Protocol
HTTP/1.1 200 OK\r\n
Request Version: HTTP/1.1
Response Code: 200
Server: Apache-Coyote/1.1\r\n
ETag: W/"111-1126147562000"\r\n
Last-Modified: Thu, 08 Sep 2005 02:46:02 GMT\r\n
Content-Type: text/html\r\n
Content-Length: 111\r\n
Date: Fri, 09 Sep 2005 02:45:36 GMT\r\n
\r\n
Frame 14.... The ACK for the Header..... it
arrives this takes 0.13452sec to arrive... Very SLOW!
No. Time
Source Destination Protocol Info
14 0.162250 client.mydomain.com
pound.mydomain.com TCP 2056 > 81 [ACK] Seq=511 Ack=205
Win=65331 Len=0
Frame 14 (60 bytes on wire, 60 bytes captured)
Arrival Time: Sep 9, 2005 11:58:45.232300000
Time delta from previous packet: 0.134520000
seconds **** NOTE that
this is now very HIGH!
:'(
Time since reference or first frame: 0.162250000
seconds
Frame Number: 14
Packet Length: 60 bytes
Capture Length: 60 bytes
Protocols in frame: eth:ip:tcp
Ethernet II, Src: client.mydomain.com (00:13:20:59:47:e3),
Dst: Intel_56:f9:a1 (00:d0:b7:56:f9:a1)
Destination: Intel_56:f9:a1 (00:d0:b7:56:f9:a1)
Source: client.mydomain.com (00:13:20:59:47:e3)
Type: IP (0x0800)
Trailer: 000000000000
Internet Protocol, Src: client.mydomain.com
(client.mydomain.com), Dst: pound.mydomain.com (pound.mydomain.com)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00:
Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint:
Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 40
Identification: 0x299f (10655)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 128
Protocol: TCP (0x06)
Header checksum: 0xa8de [correct]
Source: client.mydomain.com (client.mydomain.com)
Destination: pound.mydomain.com (pound.mydomain.com)
Transmission Control Protocol, Src Port: 2056 (2056), Dst
Port: 81 (81), Seq: 511, Ack: 205, Len: 0
Source port: 2056 (2056)
Destination port: 81 (81)
Sequence number: 511 (relative sequence number)
Acknowledgement number: 205 (relative ack number)
Header length: 20 bytes
Flags: 0x0010 (ACK)
0... .... = Congestion Window Reduced (CWR): Not
set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...1 .... = Acknowledgment: Set
.... 0... = Push: Not set
.... .0.. = Reset: Not set
.... ..0. = Syn: Not set
.... ...0 = Fin: Not set
Window size: 65331
Checksum: 0xda50 [correct]
SEQ/ACK analysis
This is an ACK to the segment in frame: 13
The RTT to ACK
the segment was: 0.134520000 seconds
Frame 15..... The HTML Content is then sent....
No. Time
Source Destination Protocol Info
15 0.162380 pound.mydomain.com
client.mydomain.com HTTP Continuation or non-HTTP traffic
Frame 15 (165 bytes on wire, 165 bytes captured)
Arrival Time: Sep 9, 2005 11:58:45.232430000
Time delta from previous packet: 0.000130000 seconds **** NOTE
that this is very low!
Time since reference or first frame: 0.162380000
seconds
Frame Number: 15
Packet Length: 165 bytes
Capture Length: 165 bytes
Protocols in frame: eth:ip:tcp:http:data
Ethernet II, Src: Intel_56:f9:a1 (00:d0:b7:56:f9:a1), Dst:
client.mydomain.com (00:13:20:59:47:e3)
Destination: client.mydomain.com (00:13:20:59:47:e3)
Source: Intel_56:f9:a1 (00:d0:b7:56:f9:a1)
Type: IP (0x0800)
Internet Protocol, Src: pound.mydomain.com
(pound.mydomain.com), Dst: client.mydomain.com (client.mydomain.com)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00:
Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint:
Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 151
Identification: 0xe9eb (59883)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: TCP (0x06)
Header checksum: 0x2823 [correct]
Source: pound.mydomain.com (pound.mydomain.com)
Destination: client.mydomain.com (client.mydomain.com)
Transmission Control Protocol, Src Port: 81 (81), Dst
Port: 2056 (2056), Seq: 205, Ack: 511, Len: 111
Source port: 81 (81)
Destination port: 2056 (2056)
Sequence number: 205 (relative sequence number)
Next sequence number: 316 (relative sequence number)
Acknowledgement number: 511 (relative ack number)
Header length: 20 bytes
Flags: 0x0018 (PSH, ACK)
0... .... = Congestion Window Reduced (CWR): Not
set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...1 .... = Acknowledgment: Set
.... 1... = Push: Set
.... .0.. = Reset: Not set
.... ..0. = Syn: Not set
.... ...0 = Fin: Not set
Window size: 6432
Checksum: 0x8882 [correct]
Hypertext Transfer Protocol
Data (111 bytes)
0000 3c 68 74 6d 6c 3e 0a 20 3c 68 65 61 64 3e 0a 20
<html>. <head>.
0010 20 3c 74 69 74 6c 65 3e 54 65 72 72 61 4d 61 70
<title>TerraMap
0020 3c 2f 74 69 74 6c 65 3e 0a 3c 2f 68 65 61 64 3e
</title>.</head>
0030 0a 3c 62 6f 64 79 3e 48 65 6c 6c 6f 20 57 6f 72
.<body>Hello Wor
0040 6c 64 20 66 72 6f 6d 3a 20 68 75 65 79 2e 61 64 ld
from: huey.ad
0050 6c 2e 61 72 64 65 63 2e 63 6f 6d 2e 61 75 3c 2f
l.ardec.com.au</
0060 62 6f 64 79 3e 0a 0a 3c 2f 68 74 6d 6c 3e 0a
body>..</html>.
Andrew Hughes wrote:
Fast HTTP response header:
Slow HTTP response header
(helloworld.html):
...
Content-Length:
24433
Date:
Thu, 08 Sep 2005 02:25:17 GMT
...Content-Length:
132
Date:
Thu, 08 Sep 2005 02:33:21 GMT
--
Ted Dunning
Chief Scientist
Veoh Networks
|
|
|
|
|
Re: [Pound Mailing List] Performance problem with JSP's
Robert Segall <roseg(at)apsis.ch> |
2005-09-09 13:00:06 |
[ FULL ]
|
On Fri, 09 Sep 2005 16:43:32 +0930 Andrew Hughes
<ahughes(at)lisasoft.com>
wrote:
[...]
Check your IP setup - this is not Pound's doing. Pound does send the
headers as a separate packet, but you can disable that easily - just
comment out in http.c lines 1086-1093:
if(BIO_flush(cl) != 1) {
if(errno)
logmsg(LOG_WARNING,
"error flush headers to %s: %s",
inet_ntoa(from_host), strerror(errno));
clean_all();
pthread_exit(NULL);
}
However, if you look at the code, you'll notice there is NO waiting for
anything from the client at this point - the content is sent on
regardless of client status.
I suggest you have a look at your IP settings, in particular the
availability of network buffers. The only way the response will be
delayed is if you run out of buffers, at which point the socket becomes
"unwritable" and Pound (or any other program) HAS to wait until it can
continue writing.
Finally, using ethereal is not very helpful: what it shows is the
interaction between the client and the network stack, rather than what
Pound does. Try strace or similar for a more informative printout.[...]
|
|
|
|
