Unfortunately, there is no silver bullet. Some disruption will always
occur.

- if you kill the child you loose the requests being currently
processed - i.e. the ones that are in transit. This was the original
idea.

- if you wait until they complete you'll probably loose new requests.
The accept queue on sockets is of limited length, so on a medium site
you'll get failures in the connect.[...]

On Wed, 31 Aug 2005 10:41:31 -0400 Ed R Zahurak
<ezahurak(at)atlanticbb.net> wrote:
[...]

This is the sort of solution you could implement right now with three
Pound instances. As the master sees that one of the sub-servers is
off-line (because it's reloading its configuration) it would be
considered dead and all requests would go to the second server. When it
comes back on-line with the new configuration it gets resurrected and
then you can take the second one off-line for reconfiguration.

However this only works because you have separate listening sockets. As
a monolithic solutions this fails.[...]

I would rather have the requests die immediately, relying on the user to
try again.  And when that happens, the supervisor running the new config
with the old sessions (having been restored) will just create another
child and proceed from there.

It's no worse than killing pound completely, which is what I have to do
now.  It's certainly better than disrupting current running requests AND
making everyone's session restart.

Joe

-----Original Message-----
From: Robert Segall [mailto:roseg(at)apsis.ch] 
Sent: Thursday, September 01, 2005 9:18 AM
To: pound(at)apsis.ch
Subject: Re: [Pound Mailing List] changing conf without restart

Unfortunately, there is no silver bullet. Some disruption will always
occur.

- if you kill the child you loose the requests being currently
processed - i.e. the ones that are in transit. This was the original
idea.

- if you wait until they complete you'll probably loose new requests.
The accept queue on sockets is of limited length, so on a medium site
you'll get failures in the connect.[...]

Robert Segall wrote:[...]

Why can't we have someone accepting and queuing the requests until
the workers complete?

Joe Gooch wrote:[...]

But it's not better than just running straight Apache and giving it a
graceful restart, which seems capable of both reloading the config and
not disrupting any requests. I'd really like to retain that behavior
when introducing Pound to the mix.

And esp. in ecommerce applications, having a request killed mid-stream
while your CC data is in transit does not give you a warm feeling.

Maurice

On Thu, 01 Sep 2005 13:27:12 -0600 Maurice Aubrey
<maurice(at)redweek.com>
wrote:
[...]

Are you volunteering?

Have a look at the code - there are no queues at all. Each connection is
accepted and immediately passed to a new thread. You stop doing that and
you loose connections.

Keep in mind that there is a single socket - and that can't be changed.[...]

When I get my head above water, I might do just that.

What about encapsulating the global variables (aka make a pound 
"object") and then when the signal is received

a) instantiate a new pound object from the new config file
b) tell the old pound object to stop accepting connections
c) copy session data from the old to the new pound object
c) tell the new pound object to start accepting all connections
d) 30 seconds later, reap and deallocate the old pound object on the 
theory that it would have
handled all outstanding work and spawned any necessary threads by then.

This approach will finish all outstanding requests because it doesn't 
kill any threads and keeps all live sessions alive.

It is bad because the single pound instance has to be able to read its 
own config file (no big deal... just put the config file in jail with 
pound) and because it requires that the C code be restructured a bit 
more than one might want.  The flip side is that global variables are 
evil in the first instance anyway.

Robert Segall wrote:
[...]
[...]

I figured this out.

For posterity...

As I suspected, the problem was DNS related.  Looking at the source code 
reveals that pound does, indeed, do a DNS lookup (gethostbyname) when 
doing redirectRewriting.

Meaning that if your local DNS server is not setup to properly resolve 
(as ours was not) then redirectRewrites won't work properly in all cases.

More problematically if the DNS server is not setup properly (pointed to 
a dead DNS server, say) then pound will "hang" on ALL "Redirect" 
response types while it waits for the resolver to timeout.  That's not a 
huge "hang time" but it can be on the order of a second or two. 
Certainly noticable.

One might ask why we didn't have our DNS resolver setup properly in the 
first place.  I can only offer that this is a linux firewall box running 
an absolutely minimal configuration, basically just iptables and pound. 
  iptables doesn't need the resolver and we had just assumed pound 
wouldn't either.  That's the problem with assumptions: makes an ass out 
of you and umptions. :-)

Might I suggest that we modify the docs to indicate that pound relies on 
a properly configured DNS resolver?

Even better (but undoubtably waaay more work), might I suggest that 
pound be modified to *not* rely on the resolver?

It seems, at first blush, that it would be possible for pound to collect 
enough info in the back-end config to eliminate the need to do 
gethostbyname calls.  Like having the back-end config-info contain both 
an IP address *and* the DNS name, for instance.  This would allow the 
"is_be" routine to do a match against returned LOCATION headers without 
having to do a lookup on the LOCATION name.  You'd already have 
everything you need for the comparison without having to do that lookup.

One might also make the argument that eliminating dns lookups would 
speed up pound.  The Apache web-server, after-all, has a disable-lookups 
feature for exactly this reason (performance).

But looking through the code it appears that dns-lookups are only done 
when dealing with redirects, when processing the config file, and at 
initial startup (when binding the listeners).  In all cases, not code 
that gets executed very often, relatively speaking.

So maybe just a note in the docs indicating that pound relies on a 
properly config'd dns resolver is enough.

- Gary


glists(at)greywether.com wrote:[...]

On Fri, 02 Sep 2005 18:02:04 -0700 glists(at)greywether.com wrote:
[...]

Pound needs to look names up. This is unavoidable if you have virtual
hosts - there is no other way to check if you need to change (rewrite)
the Location header.

You don't need a full DNS, but make sure your names are resolvable. A
simple entry in /etc/hosts with the address and alternate names will do
fine. On newer Linux systems (with resolv+) you can modify
/etc/host.conf to only look for the file, on older systems you can do
that in /etc/resolv.conf.

Thanks for the tip re. docs - we'll do that.[...]

On Fri, 16 Sep 2005 13:47:26 +0100 Will Tatam <wtatam(at)premierit.com>
wrote:
[...]

That depends pretty much on which syslog you use (Linux and Solaris are
different beasts) and how you compiled Pound.

The way we usually do it:

1. compile Pound so it uses LOCAL7 for logging:

./configure --with-log=LOG_LOCAL7 ...

compile, install, configure.

2. define where the messages for local7 should go. In /etc/syslog.conf
(assuming Linux-style syslog):

local7.=info                    /var/log/pound.log
local7.notice                   /var/log/pound.err

so that normal traffic goes to /var/log/pound.log and everything else
goes to /var/log/pound.err

3. make sure it goes nowhere else. For example

*.*;auth,authpriv,local7.none   -/var/log/syslog

(just look for *.something and add local7.none).

Don't forget to tell syslog about the new configuration.

Hope this helps some.[...]

On Fri, 2005-09-16 at 17:36 +0200, Robert Segall wrote:
[...]


Thanks for the tip Robert but that doesn't work, on redhat distros
local7 is already in use for boot.log

I tried --with-log=LOG_POUND but that throws

pound.c:397: `LOG_POUND' undeclared (first use in this function)

On Mon, 19 Sep 2005 13:47:34 +0100 Will Tatam <wtatam(at)premierit.com>
wrote:
[...]

Look in /usr/include/sys/syslog.h for available facilities. You can't
just make one up - you need to use an existing facility such as
LOG_DAEMON, LOG_LOCAL2 or whatever you want. Normally LOG_LOCAL0 to
LOG_LOCAL7 are reserved for private applications, so at least one of
them should be free.[...]

This is not really a pound problem - set "open files" to a higher value, e.g. 
I''ve set to 8000.
 ulimit -n 8000
 (to see the current value)
ulimit -a
 /David

 On 9/20/05, Alexander Meis <am(at)simoon.de> wrote: [...]

On Tue, 20 Sep 2005 12:50:31 +0300 david walters <dwalters1(at)gmail.com>
wrote:
[...]

8000 is probably overkill, though it doesn't really hurt. Most systems
come with a limit of 1K on the possible number of threads, and you use
two descriptors per thread, so 2K (+ a few extra) descriptors would
really be enough.[...]

Maybe it's a pound problem. We started to get the same error under heavy 
load
after the update to the latest pound version (not happend before).
Also the ulimit -n will increase the max number of open file for the current 
shell
session. But what if pound is started from the boot scripts? Maybe adding 
ulimit -n to the startup script will help. Still i think there is some file 
not closed problem
with the latest pound.

On 9/20/05, david walters <dwalters1(at)gmail.com> wrote:[...]

david walters wrote:[...]

Thanks for helping, i set them to 2048.

Alex

On Wed, 21 Sep 2005 11:15:52 +0900 Stoyan Zhekov <zhekov(at)gmail.com>
wrote:
[...]

Which "latest" do you mean? 1.9 to 1.9.1? 1.8 to 1.9?
[...]

It certainly will - the Pound process inherits whatever limits the
parent had - so the shell running the script or the shell you use
interactively are the same.
[...]

More evidence and details, please.[...]

> -----Original Message-----[...]

You can also use the softlimit program, from the daemontools package by
DJB.  (http://cr.yp.to/daemontools.html)

I run all my pound instances through supervise, with softlimit to limit
open files, data segment size, processes and core files, and with
libsafe.so.2 in my LD_PRELOAD.  Pound runs as a foreground process so
supervise can restart it if it dies.

Works well for me!
Joe Gooch
K12 Systems, Inc.

Dear Joe,

Today i try all patch from you ..,
and there are no error when patching and configure
But when i try to 'make' i got the error like this :

$make
cc -DF_CONF=\"/usr/local/etc/pound.cfg\" -DLINUXCAPS -g -O2 -pthread 
-DAEMON -DUPER -DNEED_STACK  -D_REENTRANT -D_THREAD_SAFE 
-Wstrict-prototypes -pipe   -c -o pound.o pound.c
pound.c:224:28: sys/capability.h: No such file or directory
pound.c: In function `main':
pound.c:885: error: `cap_t' undeclared (first use in this function)
pound.c:885: error: (Each undeclared identifier is reported only once
pound.c:885: error: for each function it appears in.)
pound.c:885: error: syntax error before "caps"
pound.c:886: error: `caps' undeclared (first use in this function)
make: *** [pound.o] Error 1

---
sys/capability.h <--- ???
Currently i install the pound under Mandriva 10.2 ( 2.6.11-6mdk ),

Do i have install SysVinit-2.78-19Au.i386.rpm On my box?

Erwien.


Joe Gooch wrote:[...]

Which "latest" do you mean? 1.9 to 1.9.1? 1.8 to 1.9?


Gentoo 2005.0, pound-1.9.1

More evidence and details, please.


I put a fragment from the log file on:
[ http://rafb.net/paste/results/DXlAfX42.html
]
For bad luck the LogLevel was 0, so not enough information for the requests
:/
Recently I started to think that maybe it's not the pound problem, but we
was just
DDoS-ed/overloaded. Still maybe the log will be interesting for somebody
like
an example of pound under heavy load.

You'll need the libcap utilities.

urpmi libcap1 libcap1-devel libcap-utils


Joe
[...]
of[...]
for[...]
could[...]
own[...]
there[...]
need to[...]
head[...]
Here[...]
process.[...]
(daemontools)[...]
set[...]
enabled[...]
then[...]
some[...]
0,[...]
process[...]
is[...]
It'll[...]
track[...]
the[...]
in[...]
session[...]
track[...]
the[...]
include[...]
Text[...]
sessions,[...]
really[...]
not[...]
current[...]
but[...]
way, if[...]
reboot,[...]
can[...]
feedback,[...]
code.[...]
pound(at)apsis.ch.[...]

On Thu, 22 Sep 2005 14:09:35 +0900 Stoyan Zhekov <zhekov(at)gmail.com>
wrote:
[...]

Certainly interesting. You definitely have an overloaded system - but it
degrades nicely: your back-ends get overloaded before Pound. Please
notice you start having back-end time-outs, followed by "out of file
descriptors" (as you have more and more connections waiting you start
running out of them).

You may want to try:

- increasing the file-descriptor limit, as described in another post.
- adding more back-ends, so that each of them is faster to respond.
- increasing the Server time-out, so that Pound waits longer for a
response.
- decreasing the Client so that at least some connections get killed
early.

Have fun.[...]