|
/
Zope
/
Apsis
/
Pound Mailing List
/
Archive
/
2004
/
2004-07
/
New -current
[
Pound and ASP on IIS / John D ... ]
[
startup stalled for 70 seconds? / Paul Chvostek ... ]
New -current
Robert Segall <roseg(at)apsis.ch> |
2004-07-15 16:57:26 |
[ FULL ]
|
A new -current has just been uploaded to the Web site. It should fix the
memory leak some people may have observed under heavy SSL load.
Please try it and let me know how it works for you - your feedback is
important.[...]
|
|
|
Re: New -current
"Simon Matter" <simon.matter(at)ch.sauter-bc.com> |
2004-07-19 14:54:20 |
[ FULL ]
|
> A new -current has just been uploaded to the Web site. It should fix
the[...]
Unfortunately it's still leaking for me. This is on RedHat 7.3. It's
running for 3 days now but I can't tell you whether it's leaking less.
Could it be that there are more places where it leaks and not all were
fixed?
Simon
|
|
|
Re: New -current
Robert Segall <roseg(at)apsis.ch> |
2004-07-19 18:33:14 |
[ FULL ]
|
On Monday 19 July 2004 14.54, Simon Matter wrote:[...]
Thanks for the feedback Simon. We're also looking at it - the leak is
definitely in OpenSSL. Whether this is an OpenSSL bug or our mistake I still
can't tell - the whole thing is kind of obscure.[...]
|
|
|
Re: New -current
"Simon Matter" <simon.matter(at)ch.sauter-bc.com> |
2004-07-19 18:52:43 |
[ FULL ]
|
> On Monday 19 July 2004 14.54, Simon Matter wrote:[...][...]
Googling for 'openssl memory leak' shows lots of known and already fixed
problems. Without knowing much about pounds internals, I'm wondering
whether it would be possible to change pound to have something like a
MaxRequestsPerChild / MaxRequestsPerThread mechanism. I mean, if it's a
problem with older openssl libs, it could still work around it that way.
Sorry if it's just a stupid idea :)
|
|
|
Re: New -current
Robert Segall <roseg(at)apsis.ch> |
2004-07-19 19:01:30 |
[ FULL ]
|
On Monday 19 July 2004 18.52, Simon Matter wrote:[...]
I'm not really sure how that would help. The leak can be shown with a single
HTTPS request, using the latest OpenSSL version (0.9.7d). OpenSSL seems to
alloc a lot of small memory chunks (12 to 16 bytes size) which are not
freed.[...]
|
|
|
Re: New -current
Thierry Coopman <thierry(at)keytradebank.com> |
2004-07-20 19:16:07 |
[ FULL ]
|
Ok,
what I'm going to say here may be ludicrous, so you are warned.
I have put in a machine with 2GB of RAM. Linux needs a recompile, so
right now it's using only 896MB. The previous config was with 512MB of
RAM and started swapping at some times, slowing things down.
Now I have a Pound process on that machine taking half of the traffic
and hits (up to 50 hits/second, >3Mbps traffic). I stated it July 16. It
'behaves' better than before, I still have about 470MB free now (slower
period of the day, down to 30 hits/sec).
I notice that the 'main' process uses very little memory, and that the
threads are consuming more.
Right now it hasn't been needed to restart the process, but I'll keep an
eye on it.
top - 18:52:41 up 4 days, 4:21, 2 users, load average: 0.16, 0.22, 0.18
Tasks: 75 total, 1 running, 74 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.0% us, 0.5% sy, 0.0% ni, 90.0% id, 0.0% wa, 0.2% hi, 0.3% si
Mem: 904932k total, 438760k used, 466172k free, 21564k buffers
Swap: 1004052k total, 0k used, 1004052k free, 303204k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
5952 nobody 16 0 3512 1876 2652 S 0.0 0.2 0:00.00
pound
7333 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:05.25
pound
7334 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:12.74
pound
7335 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
7336 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
7337 nobody 25 0 134m 83m 2736 S 0.0 9.4 0:39.93
pound
29272 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29525 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29532 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29540 nobody 17 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29628 nobody 17 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29647 nobody 15 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29658 nobody 15 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29724 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29727 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29732 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29733 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29735 nobody 15 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29736 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29737 nobody 17 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29740 nobody 17 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29741 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29742 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29745 nobody 17 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
29746 nobody 16 0 134m 83m 2736 S 0.0 9.4 0:00.00
pound
I keep a remark in mind from Robert: It might be a memory leak, but it
might also be that you need a lot of memory to handle your requests :)
I do have the impression that we start asking 'Apache features', because
I was thinking a 'server-status' page would be cool :))
Anyways, let's hope we find the openssl leak. Again, I have found
somewhere there are Error structures that need to be released explicitly
per connection.... Maybe that's related, maybe not...
more or less noted here:
<http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=brcsab%24dn8%241%40FreeBSD.csie.NCTU.edu.tw>
Using OpenSSL indeed seems to be more or less like 'guessing' for
programmers. Too bad it's such a waste of time :(
Robert Segall wrote:
[...][...][...]
[...]
|
|
|
Re: New -current
Thierry Coopman <thierry(at)keytradebank.com> |
2004-07-20 19:25:25 |
[ FULL ]
|
just for comparisson, here is a stat from the other machine, with the
pound process started yesterday:
19013 nobody 18 0 625m 467m 2744 S 0.0 93.1 0:00.00
pound
top - 19:19:53 up 19 days, 7:47, 2 users, load average: 3.19, 1.59, 0.88
Tasks: 91 total, 1 running, 90 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.4% us, 1.3% sy, 0.0% ni, 44.1% id, 48.5% wa, 0.3% hi, 0.3% si
Mem: 514808k total, 511384k used, 3424k free, 208k buffers
Swap: 1004052k total, 313128k used, 690924k free, 4484k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
15767 nobody 16 0 600m 467m 2744 S 0.3 93.0 1:54.02
pound
14882 nobody 16 0 3516 1084 2660 S 0.0 0.2 0:00.00
pound
15766 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:47.41
pound
15768 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.05
pound
15769 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.11
pound
15770 nobody 21 0 600m 467m 2744 S 0.0 93.0 11:11.81
pound
19009 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19012 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19016 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19031 nobody 17 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19032 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19033 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19034 nobody 17 0 600m 467m 2744 D 0.0 93.0 0:00.00
pound
19035 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19036 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19037 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19038 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19039 nobody 24 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19040 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19041 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19042 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19043 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19044 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19045 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19046 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19047 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19048 nobody 16 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19049 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00
pound
19050 nobody 15 0 600m 467m 2744 S 0.0 93.0 0:00.00 pound
workload is equelly wel distributed over both machines.
sight, try to explain this... I'm puzzled, I'll see how the 'large
memory' machine does and look to replace this machine with one that has
more memory.
Thierry Coopman wrote:
[...][...]
>>> fixed
>>> problems. Without knowing much about pounds internals, I'm
wondering
>>> whether it would be possible to change pound to have something
like a
>>> MaxRequestsPerChild / MaxRequestsPerThread mechanism. I mean, if
it's a
>>> problem with older openssl libs, it could still work around it
that
>>> way.
>>> Sorry if it's just a stupid idea :)
>>> [...][...][...]
|
|
|
Re: New -current
Robert Segall <roseg(at)apsis.ch> |
2004-07-21 03:34:25 |
[ FULL ]
|
I just uploaded a new -current which should (mostly?) take care of the memory
leaks. It is available on the web - as always feedback is welcome.
Additionally the new -current allows you to disable the supervisor process
(try ./configure --disable-super), which should help those of us who use the
DJB daemontools.
Looking at our tests we see a major performance boost in using NPTL - about
40% more transactions. For Gentoo users: make sure you use the right flag
USE=nptl emerge libc
to get the new threading model. NPTL also shows all threads as a single
process (unlike the old model, which uses clone() and thus has a process per
thread), which should allow for better monitoring.[...]
|
|
|
Re: New -current
"Simon Matter" <simon.matter(at)ch.sauter-bc.com> |
2004-07-21 06:17:41 |
[ FULL ]
|
> I just uploaded a new -current which should (mostly?) take care of
the[...]
Hi Robert,
Thanks for all your effort! The new current is already running on a
production box here, let's see what the day brings.
For those who care and are using my rpms, a new package is already
available here http://www.invoca.ch/pub/packages/pound/
Simon
|
|
|
Re: New -current
Sascha Ottolski <sascha.ottolski(at)gallileus.de> |
2004-07-21 11:36:17 |
[ FULL ]
|
Am Mittwoch, 21. Juli 2004 03:34 schrieb Robert Segall:[...]
in combination with or instead of --disable-daemon?
Cheers,
Sascha
[...]
|
|
|
Re: New -current
Robert Segall <roseg(at)apsis.ch> |
2004-07-21 15:23:01 |
[ FULL ]
|
On Wednesday 21 July 2004 11.36, Sascha Ottolski wrote:[...]
You may use either by itself, or both together. The way it works:
Default: both enabled. Pound will daemonise itself and spawn the worker as a
subprocess. Top process used as a supervisor.
--disable daemon only: Pound runs in the foreground. Supervisor + worker
processes.
--disable-super only: Pound runs only the WORKER thread (no supervisor) as a
daemon.
Both disabled: Pound runs in the foreground without a supervisor process.[...]
|
|
|
Re: New -current
"Simon Matter" <simon.matter(at)ch.sauter-bc.com> |
2004-07-21 19:01:38 |
[ FULL ]
|
> I just uploaded a new -current which should (mostly?) take care of
the[...]
After one day of operation I don't see a significant increase of Pounds
size. In quiet periods, the size is still under 10M while before, it has
grown to more than 100M after the same number of requests. I have already
removed the restart cronjobs. I'll report again after some more time.
Congratulations, that's really a nice step forward!
Thanks,
Simon
|
|
|
|
