Hello,

we are using pound since several years now (great software), so far on 
i386 architecture. Last week we have installed pound on x86_64 and 
noticed that there is a memory leak: pound handles round about 5 million 
requests per day and allocates 200 MB additional memory per day. This 
memory is never freed. Therefore we have to restart pound once a week. 
Otherwise the server will swap and in the end the oom-killer will kill 
pound ...
If we use the google-perftools (tcmalloc) the memory consumption rises 
even much faster. Because of this we are currently using pound without 
the perftools but the memory leak still remains (i.e. the 
google-perftools only boost the problem).

Is anybody else using pound on x86_64 and has the same problem? How can 
we further investigate this issue? We can provide any information you 
need to debug the problem.

Any help would be highly appreciated!

Kind regards,
Leo

**********
Pound:
Version 2.4.3
  Configuration switches:
    --enable-cert1l
    --with-maxbuf=2048

Linux:
Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02 CEST 2008 x86_64 
x86_64 x86_64 GNU/Linux
*********

Hi,

We have four pound instances running to balance site getting ~30 million 
hits in a day. All these nodes are from the ground up x86_64. Pound runs 
without any issues for about a year now without restarts but the version 
is much older - 2.3.2.

Maybe you have some session tracking settings which make pound grow it's 
session state data?

j.

Leo rašė:[...]

Hi,

thanks for your reply.

We are using a rather simple pound configuration (see below). There is 
only a chroot environment and some URL filters (no session tracking). On 
i386 there have never been any problems. Only difference between old and 
new server is (besides the hardware) the architecture and the OS version:

old server: Fedora 6 on i386
new server: Fedora 9 on x86_64

Any further suggestions?

Kind regards,
Leo


*** pound configuration ***

LogLevel 0
DynScale 0
Control "/var/share/pound/pound.sock"
RootJail "/usr/local/pound"
TimeOut 20

ListenHTTP
        Address 192.168.1.1
        Port    80
        HeadRemove "X-forwarded-for"
        LogLevel 0
        Err500 "/path/to/error.html"
        Err503 "/path/to/error.html"
End

ListenHTTPS
        Address 192.168.1.1
        Port    443
        Cert    "/path/to/xxx.cert"
        AddHeader "X-loadbalancer:pound"
        HeadRemove "X-forwarded-for"
        LogLevel 0
        Err500 "/path/to/error.html"
        Err503 "/path/to/error.html"
End

Service
        URL 
"^(/(xxx|yyy)/.*$|/favicon\.ico$|/zzz\.css$|/index\.html(\?.*)?$|/(\?.*)?)$"

        BackEnd
                Address 192.168.1.10
                Port    888
                Priority 3
        End

        BackEnd
                Address 192.168.1.11
                Port    888
                Priority 8
        End
End

Service
        URL "^/abc/"

        BackEnd
                Address 192.168.1.20
                Port    888
        End

        Emergency
                Address 192.168.1.21
                Port    888
        End
End

Service
        URL "^/def/"

        BackEnd
                Address 192.168.1.30
                Port    888
        End
End

Service
        BackEnd
                Address 192.168.1.40
                Port    888
        End
End

*** pound configuration ***


tyntas wrote:[...][...][...]

Leo wrote:[...]
Isn't fedora desktop oriented Linux?
We are using 2.3.2 on x86_64 and do not have problems too.

I would suggest testing on same version of Pound on Fedora 9 i386, and 
if you see the memory leak there, the it's the fedora change that broke 
the things.[...][...]
>>> Hello,
>>>
>>> we are using pound since several years now (great software), so
far 
>>> on i386 architecture. Last week we have installed pound on x86_64 
>>> and noticed that there is a memory leak: pound handles round about
5 
>>> million requests per day and allocates 200 MB additional memory
per 
>>> day. This memory is never freed. Therefore we have to restart
pound 
>>> once a week. Otherwise the server will swap and in the end the 
>>> oom-killer will kill pound ...
>>> If we use the google-perftools (tcmalloc) the memory consumption 
>>> rises even much faster. Because of this we are currently using
pound 
>>> without the perftools but the memory leak still remains (i.e. the 
>>> google-perftools only boost the problem).
>>>
>>> Is anybody else using pound on x86_64 and has the same problem?
How 
>>> can we further investigate this issue? We can provide any 
>>> information you need to debug the problem.
>>>
>>> Any help would be highly appreciated!
>>>
>>> Kind regards,
>>> Leo
>>>
>>> **********
>>> Pound:
>>> Version 2.4.3
>>> Configuration switches:
>>> --enable-cert1l
>>> --with-maxbuf 48
>>>
>>> Linux:
>>> Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02 CEST 2008 
>>> x86_64 x86_64 x86_64 GNU/Linux
>>> *********
>>>
>>> -- 
>>> To unsubscribe send an email with subject unsubscribe to 
>>> pound(at)apsis.ch.
>>> Please contact roseg(at)apsis.ch for questions.
>>>
>>>[...][...]
[...]

Well,

as seen from the configuration it should not keep any state information, 
so it becomes to look like some mem-leak.

As far as no one else has got such problem your'e on your own. :-/ I 
could only suggest to try debugging(maybe pound developers could support 
you with that) or changing something like downgrading pound version, etc...

j.

Leo rašė:[...][...]
>>> Hello,
>>>
>>> we are using pound since several years now (great software), so
far 
>>> on i386 architecture. Last week we have installed pound on x86_64
and 
>>> noticed that there is a memory leak: pound handles round about 5 
>>> million requests per day and allocates 200 MB additional memory
per 
>>> day. This memory is never freed. Therefore we have to restart
pound 
>>> once a week. Otherwise the server will swap and in the end the 
>>> oom-killer will kill pound ...
>>> If we use the google-perftools (tcmalloc) the memory consumption 
>>> rises even much faster. Because of this we are currently using
pound 
>>> without the perftools but the memory leak still remains (i.e. the 
>>> google-perftools only boost the problem).
>>>
>>> Is anybody else using pound on x86_64 and has the same problem?
How 
>>> can we further investigate this issue? We can provide any
information 
>>> you need to debug the problem.
>>>
>>> Any help would be highly appreciated!
>>>
>>> Kind regards,
>>> Leo
>>>
>>> **********
>>> Pound:
>>> Version 2.4.3
>>>  Configuration switches:
>>>    --enable-cert1l
>>>    --with-maxbuf 48
>>>
>>> Linux:
>>> Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02 CEST 2008 
>>> x86_64 x86_64 x86_64 GNU/Linux
>>> *********
>>>
>>> -- 
>>> To unsubscribe send an email with subject unsubscribe to
pound(at)apsis.ch.
>>> Please contact roseg(at)apsis.ch for questions.
>>>
>>>[...][...]

I'm currently installing a server with Fedora 9 i386 to check if Fedora 
is the problem.

Can somebody tell me how to debug a possible memory leak in pound?

Kind regards,
Leo


tyntas wrote:[...][...]
>>> Hi,
>>>
>>> We have four pound instances running to balance site getting ~30 
>>> million hits in a day. All these nodes are from the ground up 
>>> x86_64. Pound runs without any issues for about a year now without

>>> restarts but the version is much older - 2.3.2.
>>>
>>> Maybe you have some session tracking settings which make pound
grow 
>>> it's session state data?
>>>
>>> j.
>>>
>>> Leo rašė:
>>>> Hello,
>>>>
>>>> we are using pound since several years now (great software),
so far 
>>>> on i386 architecture. Last week we have installed pound on
x86_64 
>>>> and noticed that there is a memory leak: pound handles round
about 
>>>> 5 million requests per day and allocates 200 MB additional
memory 
>>>> per day. This memory is never freed. Therefore we have to
restart 
>>>> pound once a week. Otherwise the server will swap and in the
end 
>>>> the oom-killer will kill pound ...
>>>> If we use the google-perftools (tcmalloc) the memory
consumption 
>>>> rises even much faster. Because of this we are currently using

>>>> pound without the perftools but the memory leak still remains
(i.e. 
>>>> the google-perftools only boost the problem).
>>>>
>>>> Is anybody else using pound on x86_64 and has the same
problem? How 
>>>> can we further investigate this issue? We can provide any 
>>>> information you need to debug the problem.
>>>>
>>>> Any help would be highly appreciated!
>>>>
>>>> Kind regards,
>>>> Leo
>>>>
>>>> **********
>>>> Pound:
>>>> Version 2.4.3
>>>>  Configuration switches:
>>>>    --enable-cert1l
>>>>    --with-maxbuf 48
>>>>
>>>> Linux:
>>>> Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02 CEST
2008 
>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>> *********
>>>>
>>>> -- 
>>>> To unsubscribe send an email with subject unsubscribe to 
>>>> pound(at)apsis.ch.
>>>> Please contact roseg(at)apsis.ch for questions.
>>>>
>>>>
>>>
>>>
>>>
>>> -- 
>>> To unsubscribe send an email with subject unsubscribe to 
>>> pound(at)apsis.ch.
>>> Please contact roseg(at)apsis.ch for questions.
>>>[...][...]

Stefan Lambrev wrote:[...][...][...]
That's a matter of opinion and usage. It's the community version of 
RedHat ...[...]
I'm currently working on it.[...]
>>> Hi,
>>>
>>> We have four pound instances running to balance site getting ~30 
>>> million hits in a day. All these nodes are from the ground up 
>>> x86_64. Pound runs without any issues for about a year now without

>>> restarts but the version is much older - 2.3.2.
>>>
>>> Maybe you have some session tracking settings which make pound
grow 
>>> it's session state data?
>>>
>>> j.
>>>
>>> Leo rašė:
>>>> Hello,
>>>>
>>>> we are using pound since several years now (great software),
so far 
>>>> on i386 architecture. Last week we have installed pound on
x86_64 
>>>> and noticed that there is a memory leak: pound handles round
about 
>>>> 5 million requests per day and allocates 200 MB additional
memory 
>>>> per day. This memory is never freed. Therefore we have to
restart 
>>>> pound once a week. Otherwise the server will swap and in the
end 
>>>> the oom-killer will kill pound ...
>>>> If we use the google-perftools (tcmalloc) the memory
consumption 
>>>> rises even much faster. Because of this we are currently using

>>>> pound without the perftools but the memory leak still remains
(i.e. 
>>>> the google-perftools only boost the problem).
>>>>
>>>> Is anybody else using pound on x86_64 and has the same
problem? How 
>>>> can we further investigate this issue? We can provide any 
>>>> information you need to debug the problem.
>>>>
>>>> Any help would be highly appreciated!
>>>>
>>>> Kind regards,
>>>> Leo
>>>>
>>>> **********
>>>> Pound:
>>>> Version 2.4.3
>>>> Configuration switches:
>>>> --enable-cert1l
>>>> --with-maxbuf 48
>>>>
>>>> Linux:
>>>> Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02 CEST
2008 
>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>> *********
>>>>
>>>> -- 
>>>> To unsubscribe send an email with subject unsubscribe to 
>>>> pound(at)apsis.ch.
>>>> Please contact roseg(at)apsis.ch for questions.
>>>>
>>>>
>>>
>>>
>>>
>>> -- 
>>> To unsubscribe send an email with subject unsubscribe to 
>>> pound(at)apsis.ch.
>>> Please contact roseg(at)apsis.ch for questions.
>>>[...][...]

Hi Leo,

Leo wrote:[...]
I think the most popular tool you can use under linux is 
http://valgrind.org/[...][...]
>>> Hi,
>>>
>>> thanks for your reply.
>>>
>>> We are using a rather simple pound configuration (see below).
There 
>>> is only a chroot environment and some URL filters (no session 
>>> tracking). On i386 there have never been any problems. Only 
>>> difference between old and new server is (besides the hardware)
the 
>>> architecture and the OS version:
>>>
>>> old server: Fedora 6 on i386
>>> new server: Fedora 9 on x86_64
>>>
>>> Any further suggestions?
>>>
>>> Kind regards,
>>> Leo
>>>
>>>
>>> *** pound configuration ***
>>>
>>> LogLevel 0
>>> DynScale 0
>>> Control "/var/share/pound/pound.sock"
>>> RootJail "/usr/local/pound"
>>> TimeOut 20
>>>
>>> ListenHTTP
>>> Address 192.168.1.1
>>> Port 80
>>> HeadRemove "X-forwarded-for"
>>> LogLevel 0
>>> Err500 "/path/to/error.html"
>>> Err503 "/path/to/error.html"
>>> End
>>>
>>> ListenHTTPS
>>> Address 192.168.1.1
>>> Port 443
>>> Cert "/path/to/xxx.cert"
>>> AddHeader "X-loadbalancer:pound"
>>> HeadRemove "X-forwarded-for"
>>> LogLevel 0
>>> Err500 "/path/to/error.html"
>>> Err503 "/path/to/error.html"
>>> End
>>>
>>> Service
>>> URL 
>>>
"^(/(xxx|yyy)/.*$|/favicon\.ico$|/zzz\.css$|/index\.html(\?.*)?$|/(\?.*)?)$" 
>>>
>>>
>>> BackEnd
>>> Address 192.168.1.10
>>> Port 888
>>> Priority 3
>>> End
>>>
>>> BackEnd
>>> Address 192.168.1.11
>>> Port 888
>>> Priority 8
>>> End
>>> End
>>>
>>> Service
>>> URL "^/abc/"
>>>
>>> BackEnd
>>> Address 192.168.1.20
>>> Port 888
>>> End
>>>
>>> Emergency
>>> Address 192.168.1.21
>>> Port 888
>>> End
>>> End
>>>
>>> Service
>>> URL "^/def/"
>>>
>>> BackEnd
>>> Address 192.168.1.30
>>> Port 888
>>> End
>>> End
>>>
>>> Service
>>> BackEnd
>>> Address 192.168.1.40
>>> Port 888
>>> End
>>> End
>>>
>>> *** pound configuration ***
>>>
>>>
>>> tyntas wrote:
>>>> Hi,
>>>>
>>>> We have four pound instances running to balance site getting
~30 
>>>> million hits in a day. All these nodes are from the ground up 
>>>> x86_64. Pound runs without any issues for about a year now
without 
>>>> restarts but the version is much older - 2.3.2.
>>>>
>>>> Maybe you have some session tracking settings which make pound
grow 
>>>> it's session state data?
>>>>
>>>> j.
>>>>
>>>> Leo rašė:
>>>>> Hello,
>>>>>
>>>>> we are using pound since several years now (great
software), so 
>>>>> far on i386 architecture. Last week we have installed
pound on 
>>>>> x86_64 and noticed that there is a memory leak: pound
handles 
>>>>> round about 5 million requests per day and allocates 200
MB 
>>>>> additional memory per day. This memory is never freed.
Therefore 
>>>>> we have to restart pound once a week. Otherwise the server
will 
>>>>> swap and in the end the oom-killer will kill pound ...
>>>>> If we use the google-perftools (tcmalloc) the memory
consumption 
>>>>> rises even much faster. Because of this we are currently
using 
>>>>> pound without the perftools but the memory leak still
remains 
>>>>> (i.e. the google-perftools only boost the problem).
>>>>>
>>>>> Is anybody else using pound on x86_64 and has the same
problem? 
>>>>> How can we further investigate this issue? We can provide
any 
>>>>> information you need to debug the problem.
>>>>>
>>>>> Any help would be highly appreciated!
>>>>>
>>>>> Kind regards,
>>>>> Leo
>>>>>
>>>>> **********
>>>>> Pound:
>>>>> Version 2.4.3
>>>>> Configuration switches:
>>>>> --enable-cert1l
>>>>> --with-maxbuf 48
>>>>>
>>>>> Linux:
>>>>> Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02 CEST
2008 
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>> *********
>>>>>
>>>>> -- 
>>>>> To unsubscribe send an email with subject unsubscribe to 
>>>>> pound(at)apsis.ch.
>>>>> Please contact roseg(at)apsis.ch for questions.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> To unsubscribe send an email with subject unsubscribe to 
>>>> pound(at)apsis.ch.
>>>> Please contact roseg(at)apsis.ch for questions.
>>>>
>>>
>>> -- 
>>> To unsubscribe send an email with subject unsubscribe to 
>>> pound(at)apsis.ch.
>>> Please contact roseg(at)apsis.ch for questions.
>>>
>>>[...][...]
[...]

We have a fairly busy website running Pound v2.4.3 on Fedora Core 6 x86_64
with kernel v2.6.19-1.2911.fc6. Building Pound with regular malloc (i.e. no
external memory allocator libraries linked in) produces an executable with
no memory leaks that I can detect. It has been about a week since the last
pound restart and it shows about 25 Mb of memory allocated with about 400
pound processes running on average.

However, when I tried building pound with tcmalloc (from google perftools),
there was a pretty apparent memory leak present. After about 2 days of
running, pound was taking over 800 Mb of memory and growing. I also tried
building it with hoard, it was still leaking memory. Not as bad as with
tcmalloc though. So, needless to say, we are running the regular malloc
version -- I'd much rather have it run a bit slow that restarting pound
daily...

We have a very simple setup, only HTTP traffic going through pound, no HTTPS
and no sessions.

Not sure if this helps with finding the memory leak, but I thought I'd
mention my experience...

-Mike

On 7/7/08, Leo <neleo(at)gmx.net> wrote:[...]

Hi,

first of all thanks for your suggestions.

I have made further tests: the problem is the chroot environment. 
Without chroot there is no memory leak. Obviously pound needs some 
additional libraries on x86_64 within the chroot environment. I will 
check this next week (sorry, but I can make tests only on our production 
server and this is not always possible).

Kind regards,
Leo

Stefan Lambrev wrote:[...][...][...][...]
>>> Well,
>>>
>>> as seen from the configuration it should not keep any state 
>>> information, so it becomes to look like some mem-leak.
>>>
>>> As far as no one else has got such problem your'e on your own. :-/
I 
>>> could only suggest to try debugging(maybe pound developers could 
>>> support you with that) or changing something like downgrading
pound 
>>> version, etc...
>>>
>>> j.
>>>
>>> Leo rašė:
>>>> Hi,
>>>>
>>>> thanks for your reply.
>>>>
>>>> We are using a rather simple pound configuration (see below).
There 
>>>> is only a chroot environment and some URL filters (no session 
>>>> tracking). On i386 there have never been any problems. Only 
>>>> difference between old and new server is (besides the
hardware) the 
>>>> architecture and the OS version:
>>>>
>>>> old server: Fedora 6 on i386
>>>> new server: Fedora 9 on x86_64
>>>>
>>>> Any further suggestions?
>>>>
>>>> Kind regards,
>>>> Leo
>>>>
>>>>
>>>> *** pound configuration ***
>>>>
>>>> LogLevel 0
>>>> DynScale 0
>>>> Control "/var/share/pound/pound.sock"
>>>> RootJail "/usr/local/pound"
>>>> TimeOut 20
>>>>
>>>> ListenHTTP
>>>> Address 192.168.1.1
>>>> Port 80
>>>> HeadRemove "X-forwarded-for"
>>>> LogLevel 0
>>>> Err500 "/path/to/error.html"
>>>> Err503 "/path/to/error.html"
>>>> End
>>>>
>>>> ListenHTTPS
>>>> Address 192.168.1.1
>>>> Port 443
>>>> Cert "/path/to/xxx.cert"
>>>> AddHeader "X-loadbalancer:pound"
>>>> HeadRemove "X-forwarded-for"
>>>> LogLevel 0
>>>> Err500 "/path/to/error.html"
>>>> Err503 "/path/to/error.html"
>>>> End
>>>>
>>>> Service
>>>> URL 
>>>>
"^(/(xxx|yyy)/.*$|/favicon\.ico$|/zzz\.css$|/index\.html(\?.*)?$|/(\?.*)?)$" 
>>>>
>>>>
>>>> BackEnd
>>>> Address 192.168.1.10
>>>> Port 888
>>>> Priority 3
>>>> End
>>>>
>>>> BackEnd
>>>> Address 192.168.1.11
>>>> Port 888
>>>> Priority 8
>>>> End
>>>> End
>>>>
>>>> Service
>>>> URL "^/abc/"
>>>>
>>>> BackEnd
>>>> Address 192.168.1.20
>>>> Port 888
>>>> End
>>>>
>>>> Emergency
>>>> Address 192.168.1.21
>>>> Port 888
>>>> End
>>>> End
>>>>
>>>> Service
>>>> URL "^/def/"
>>>>
>>>> BackEnd
>>>> Address 192.168.1.30
>>>> Port 888
>>>> End
>>>> End
>>>>
>>>> Service
>>>> BackEnd
>>>> Address 192.168.1.40
>>>> Port 888
>>>> End
>>>> End
>>>>
>>>> *** pound configuration ***
>>>>
>>>>
>>>> tyntas wrote:
>>>>> Hi,
>>>>>
>>>>> We have four pound instances running to balance site
getting ~30 
>>>>> million hits in a day. All these nodes are from the ground
up 
>>>>> x86_64. Pound runs without any issues for about a year now
without 
>>>>> restarts but the version is much older - 2.3.2.
>>>>>
>>>>> Maybe you have some session tracking settings which make
pound 
>>>>> grow it's session state data?
>>>>>
>>>>> j.
>>>>>
>>>>> Leo rašė:
>>>>>> Hello,
>>>>>>
>>>>>> we are using pound since several years now (great
software), so 
>>>>>> far on i386 architecture. Last week we have installed
pound on 
>>>>>> x86_64 and noticed that there is a memory leak: pound
handles 
>>>>>> round about 5 million requests per day and allocates
200 MB 
>>>>>> additional memory per day. This memory is never freed.
Therefore 
>>>>>> we have to restart pound once a week. Otherwise the
server will 
>>>>>> swap and in the end the oom-killer will kill pound ...
>>>>>> If we use the google-perftools (tcmalloc) the memory
consumption 
>>>>>> rises even much faster. Because of this we are
currently using 
>>>>>> pound without the perftools but the memory leak still
remains 
>>>>>> (i.e. the google-perftools only boost the problem).
>>>>>>
>>>>>> Is anybody else using pound on x86_64 and has the same
problem? 
>>>>>> How can we further investigate this issue? We can
provide any 
>>>>>> information you need to debug the problem.
>>>>>>
>>>>>> Any help would be highly appreciated!
>>>>>>
>>>>>> Kind regards,
>>>>>> Leo
>>>>>>
>>>>>> **********
>>>>>> Pound:
>>>>>> Version 2.4.3
>>>>>> Configuration switches:
>>>>>> --enable-cert1l
>>>>>> --with-maxbuf 48
>>>>>>
>>>>>> Linux:
>>>>>> Linux poundserver 2.6.25.9 #1 SMP Wed Jun 25 15:09:02
CEST 2008 
>>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>> *********
>>>>>>
>>>>>> -- 
>>>>>> To unsubscribe send an email with subject unsubscribe
to 
>>>>>> pound(at)apsis.ch.
>>>>>> Please contact roseg(at)apsis.ch for questions.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> To unsubscribe send an email with subject unsubscribe to 
>>>>> pound(at)apsis.ch.
>>>>> Please contact roseg(at)apsis.ch for questions.
>>>>>
>>>>
>>>> -- 
>>>> To unsubscribe send an email with subject unsubscribe to 
>>>> pound(at)apsis.ch.
>>>> Please contact roseg(at)apsis.ch for questions.
>>>>
>>>>
>>>
>>>
>>>
>>> -- 
>>> To unsubscribe send an email with subject unsubscribe to 
>>> pound(at)apsis.ch.
>>> Please contact roseg(at)apsis.ch for questions.
>>>[...][...]