Bump.

A scenario this might occur is: Say webdata are served over NFS, but if the NFS
server becomes unresponsive, either it's a local(backend host) problem or 
the NFS server. As a result requests coming in for apache all stall until 
they time out. 

The bottomline is, I think the condition to resurrect a backend must be
stricter.

Anyone can provide more insight in this matter?

Kind regards,
Xiwen

On Mon, Apr 06, 2009 at 03:13:19PM +0200, Xiwen Cheng wrote:[...]
[...]

Pound will only mark a backend dead if the TCP connection to the backend fails.
 (For instance, I'll add an iptables rule on the backend to REJECT connections
to the http port when doing maintenance)  Similarly, resurrect checks for a TCP
connection to the backend.

What you're talking about would happen if the TCP connection succeeded and the
httpd could not return data.  This could also happen if a backend process were
running and generating content, but took a long time to complete.  (This
happens a lot in my situation)  I wouldn't want my backend to be marked dead
because someone ran a large report.

Which is why the checks for life are so rudimentary in pound.  But it's also
why there's a HAPort directive.  You can craft a simple perl script that
listens on a different port, tries to read a dummy file from NFS on connect
attempts, and runs an accept() call if the check succeeds.  If it doesn't, the
backend will be marked dead and stay dead until that check succeeds.

This question comes up a lot. I'm sure there are plenty of examples in the list
archives.

It is interesting however that kill_be does not log that it is killing a
backend... That should likely happen.

Take care!
Joe
[...]

Thanks for the information, and apologies in advance for the
top-posting.  It sounds like Pound might not be the best solution for
what I am trying to do.  I will investigate Squid as a possible reverse
proxy solution.  It's not that I'm opposed to handling the SSL certs on
the reverse proxy, as it does make logical sense to do so.  It's just
that we are talking about 30-40 different servers all with unique certs
and different expiration times... managing the certs is a full-time job
in and of itself, and one that I'm not prepared to take on at this
point.

Thanks again for the information.  I also plan on doing IP based virtual
hosting for this very reason.  1-1 NAT for internal and external means I
don't have to decrypt the HTTP GET requests (which would be impossible
anyway) to determine what back-end server they go to.

Regards,
Luke

-----Original Message-----
From: Dave Steinberg [mailto:dave(at)redterror.net] 
Sent: Friday, May 01, 2009 2:05 PM
To: pound(at)apsis.ch
Subject: Re: [Pound Mailing List] Quick SSL questions

Youngblood, Luke wrote:[...]

Not with pound.  Pound expects to do the SSL work and pass vanilla HTTP 
to your backends.
[...]

Its not just pound, its a function of the SSL protocol.

 > If I use IP based[...]
is[...]
decrypt[...]

What you're talking about is a simple 1-1 NAT, and your router/firewall 
can most likely do that for you.  What people typically want to do is 
host multiple SSL *name based* vhosts on a single IP, which the SSL 
protocol does not support.

That said, I've found it very convenient to offload SSL at the pound 
box, with each site getting its own IP.  Then *since pound speaks plain 
http to the backend*, you can direct all requests to a single backend IP

and do name-based vhosting there.  This makes SSL a configuration 
setting in pound, a DNS change, but the backend doesn't need to be
touched.

Bottom line, whatever your SSL endpoint is must use an ip-based vhosting

scheme.

Good luck,[...]

On Mon, May 04, 2009 at 09:49:35AM -0400, Joe Gooch wrote:[...]
I understand the need for the HAPort directive. I actually considered it at
some point. But if applied, the number of connections made to the backend
will increase drastically, I would even say unnecessary. More connection means
more activity. So in the end more overhead. I think it'd be cheaper, in
terms of resources to use the information gathered from ongoing requests to
determine the status of a backend. I didn't look through the source code
to determine the precise behaviour of having HAPort defined. So if I'm wrong,
please correct me.

After all, in this case we're only interested in availablity of the backend
as in: handle incoming HTTP requests. Using an external program to determine
the
availability of the data source still doesn't imply the availability of the
backend itself. Such a situation is if the backend is under heavy load
(your example) or in case the webserving daemon ends up in a race-condition
situation.

I don't think ignoring backend timeouts, as this seems to be the current
behaviour in pound is desireable. Or the weak condition to resurrect a backend.

Sure someone may be generating a large report which renders the server 
unresponsive for a limited period of time. But that doesn't change the fact 
the backend isn't responding. So it's logical to mark it either as dead or 
give a status to avoid requests being forwarded to it. And _only_ resurrect
it when it answers to HTTP requests. 

An efficient solution is to have the ability to define a URL for
Availability check, which is _only_ used in case a backend has been marked
dead/unavailable. This mechanism looks similar to any other monitoring
solution like Nagios. The difference, which makes this superior to the
existing solution, is the backend won't be unnecessarily bombarded by 
Availability request checks.
[...]
Couldn't find them. Maybe I should look harder.
[...]
Indeed. Maybe someone else can shed some lights on this kind of bevahiour?

Xiwen
[...]

> -----Original Message-----[...]

The man page documents the behavior under "HIGH-AVAILABILITY".... Basically it
polls that port every "Alive" seconds.

But the idea is that the port you check with HAPort is not the same port as the
HTTP daemon.
 [...]
True, but if your data source isn't available, your backend isn't going to be
able to serve the data, right?

In my case, one request running a long report may mean that one request takes a
while to complete, but my other request threads are behaving as normal.  Unless
all of my request threads are full, which Pound isn't going to know, because
the vast majority of the requests are succeeding.  Plus, since I'm using
session affinity, I would want Pound to be *very* sure that none of the
requests are going to succeed to that backend before breaking sessions.

It's entirely possible that dynamic scaling might help you, as that tries to
use timeout values to determine the better/best backend for any given
connection.

It's also possible that your HAPort script doesn't just check NFS, or
datasources.  It could also run a simple HTTP request against the backend to
verify it responds.  I think the flexibility of the system was the reason it
was done this way...  Since all applications/backends are very specific to
their use, it's hard to implement a solution that would work for everyone.

[...]

I don't really see any reason this couldn't be done.  It just means
thr_resurect() in svc.c will need some additional code, such that if the
connect succeeds, it sends a GET request to a URL (if defined in the config),
and then it would need to know success conditions. (HTTP status code of 200? 
Response in n seconds or less?)  Might be worth it if Robert weighed in.

Then again, this was suggested 11/05/2005.
http://www.apsis.ch/pound/pound_list/archive/2005/2005-11/1131177343000

I think the difference in your case is that it would only check the Alive URL
to resurrect, not regularly.
[...]

http://osdir.com/ml/web.pound.general/2006-03/msg00055.html
http://www.apsis.ch/pound/pound_list/archive/2006/2006-06/1151100017000/index_html?fullMode=1
http://www.apsis.ch/pound/pound_list/archive/2006/2006-12/1165505787000
[...]

I've put a patch on my site that should add additional log messages.

http://users.k12system.com/mrwizard/pound/pound24.html


Joe

Thanks for the links and insights. From what I've read, I'm not quite
satisfied with the proposed solutions. That's just me of course, it's true
one cannot satisfy everybody's desires. 

For now I'm focussing on solving the problem why the data source becomes
unavailable (bad autofs with wildcard mappings!) on an irregular basis.
Really painful to troubleshoot a problem one cannot reproduce. 

After that I think I'll write a patch that implements the "efficient
monitor". For now the global idea is to ommit unnecessary URI checks to
backends to determine the liveness of a backend. 

Anyone still have this patch:
http://www.apsis.ch/pound/pound_list/archive/2005/2005-11/1131177343000
The source is gone :(

I decided to read the pound manpage more thoroughly and came across this 
snippet in HIGH-AVAILABILITY section:[...]
Does this mean: if the HAport check failed the backend is marked as dead.
All clients (say using session type IP) on that backend will receive 
error-503. This only accounts for ongoing(STATE_ESTABLISHED) connections 
of these clients? New connections made by these clients will be forwarded
to other backends in the pool.

Best regards,
Xiwen

On Wed, May 06, 2009 at 09:54:16AM -0400, Joe Gooch wrote:[...]
[...]
[...]


[...]

> -----Original Message-----[...]

I understand...  That's why I have 8 patches on my pound page. :)
 [...]

Sounds like a good plan!
[...]

I don't have 1.9.4, but I do have 1.9.1.  I believe the patch will apply
against that.
The patch itself downloaded from the archive page for me, no problem.

http://users.k12system.com/mrwizard/pound/Pound-1.9.1.tgz

For 2.4, you'd likely want to move the AliveURI check into the SERVICE
structure.
[...]

All that logic is in svc.c in get_backend()

Looks to me like:
1) If there are no available backends, it uses the Emergency backend, or sends
a 503.
2) If you have session affinity, it pulls the existing session. If none exists,
it creates one with a random backend.
3) If not using session affinity it chooses a backend at random from the list
of alive backends.

If the selected backend fails, it calls kill_be, which in addition to marking
the backend dead, will ALSO clear all sessions using that backend.  Thus, those
sessions would be recreated in the next pass.  Since that's in a loop, the
failure is trapped and another backend is tried immediately.

Also in svc.c, in do_resurect (which runs every Alive seconds) the first thing
it does is check all alive servers to make sure they're still alive, using the
HAPort.  If it fails, the backend is marked dead.  If there's no HA port, it
skips that check.

Then it checks already dead backends to see if they should now be alive, using
the haport if available, backend port if not.

So, in short, if HAPort fails, the backend is marked dead.  If HAPort never
succeeds, the backend stays dead.  If a client is part of a new session, it
gets a random backend.  When a backend is marked dead, sessions using the old
backend are deleted, causing them to get a new backend.

So clients shouldn't ever make it to a dead backend.

Take care!
Joe

Last week we installed pound on a new server.  We found that our regular 
expressions, which worked with an older verion of PCRE, were no longer 
working.  On our old box, with PCRE version (4.5), the regular 
expressions were doing case-insensitive checks.  The new box, with PCRE 
6.6, is doing case-sensitive checks.

We use RedHat linux here, and I don't remember if PCRE was built 
locally, or a package was used, but whatever the case, our URL matches 
were no longer working correctly.  We use windows boxes behind pound, so 
we never really want to do case-sensitive checks, and I guess were lucky 
that we didn't have this problem before.

To make our pound.cfg work on the new box, I had to put "(?i)" in each 
of our checks (around 50 in all).  I'd like to see pound support an 
global directive or compile-time flag for case-sensitivity checks.  Any 
thoughts?



Sebastiaan van Erk wrote:[...][...][...]

> -----Original Message-----[...]

Welcome back!
[...]

Agreed.  Though I think the suggestion of checking a page for status *only when
deciding whether to resurrect or not* does have merit.  When backends are
loaded down, frequently they'll be erratic before they're stable again.  (At
least in my experience)

http://users.k12system.com/mrwizard/pound/pound24.html
has a LogBackendDeath patch if interested.
 [...]

(Since these go together)

I wasn't aware of (?i) at all until it was posted here.  It could be added to
the man page (if it's not there already) since this seems to be a common thing.

I'm still running Pound 1.9 in places... (yeah. Old)  But the point is I
believe up through 2.2, pound was case insensitive with url matching.  So at
the least, this is a paradigm shift.

In addition, if compiled --without-pcre, would this (?i) option still be
available?  Or is it PCRE only?

Seems like it might be worth having a consistent option, regardless of your
library dependencies.

http://users.k12system.com/mrwizard/pound/pound24.html
has "NoCase" versions of URL and CheckURL directives that provide an alternate
solution to a compile time option, with minimal code impact.



For that matter, the more of those patches that go main-stream, the fewer I
have to maintain. :)

Thanks for your insight!

Joe

I changed this fix code in pound 2.4.4 and after maked the binaries but the
proplem persist

This my configuration file

###############################
User            "pound"
Group           "pound"
LogLevel        3
Alive           10


ListenHTTP
        Address 10.234.18.28
        Port    80
        RewriteLocation 1
        #RewriteDestination 1


        Service
                HeadRequire "Host: www.mysite.com.*"

                Session
                    Type Cookie
                    ID   "MYSITESESSIONID"
                    TTL  300
                End

                BackEnd
                     Address 192.168.1.100
                     Port    10000
                End
                BackEnd
                    Address 192.168.1.101
                    Port    10000
                End
                BackEnd
                    Address 192.168.1.102
                    Port    10000
                End
                BackEnd
                    Address 192.168.1.103
                    Port    10000
                End
                BackEnd
                    Address 192.168.1.104
                    Port    10000
                End

        End


End
######################################

when i get a http://www.mysite.com/default where
default is a real
directory, pound receive a redirect from backend sending to the client the
port of backend.

for example in a client pc

telnet 10.234.18.28 80
Trying 10.234.18.28...
Connected to 10.234.18.28 (10.234.18.28).
Escape character is '^]'.
GET /default HTTP/1.0
host: www.mysite.com

HTTP/1.1 302 Object Moved
Location: http://www.mysite.com:10000/default/
Server: Microsoft-IIS/5.0
Content-Type: text/html
Content-Length: 173

in Location i think i havenīt to see the port of backend because i use
"RewriteLocation 1" directive

how can i solve this problem?
thanks!



2009/4/13 Emilio Campos <emilio.campos.martin(at)gmail.com>
[...]

[...]

awesome - thanks.

>- session tracking on parameter (Ron Bloom): in the path /a;b/c the
element "a;b" is considered a single path element, with the "b" part
usually reserved for OS-specific stuff (such as file version). Have a
look at RFC 2396 for all the gory details. I think we should stick to
the accepted standards and use "?" as a separator between the path and
the query parameters.

OK. Also found a other way to solve this, just put in http.conf
CookieTracking On and CookieName {name} and did session tracking on this
cookie instead of the java sessionid.

Regards,

Ron Blom.


DISCLAIMER:

Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet
de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u
verzocht dat aan de afzender te melden en het bericht te verwijderen. Het
College voor zorgverzekeringen aanvaardt geen aansprakelijkheid voor schade,
van welke aard dan ook, die verband houdt met risico's verbonden aan het
elektronisch verzenden van berichten.

This message may contain information that is not intended for you. If you are
not the addressee or if this message was sent to you by mistake, you are
requested to inform the sender and delete the message. The Health Care
Insurance Board accepts no liability for damage of any kind resulting from the
risks inherent in the electronic transmission of messages.

Jeff,

Do the cameras send video packets using UDP? Since the camera connection
from Lynx says that it needs Java, I am thinking that it delivers a java
applet, and then starts a UDP feed on the camera to send back to the java
applet, which then displays UDP video packets. That's a very common LAN
video design.

Try connecting to the camera through apache proxy and see if you get the
same result.

I do not believe that pound can redirect UDP traffic.
[...]

I used TCPVIEW on a windows box and made a direct connection to the
camera.  I noticed right off the bat, it makes about 30 TCP connections
to the camera; not entirely sure.  After a short time, all but one of
these established connections goes TIME_WAIT.

I wonder if pound maybe seeing this as a possible attack or something,
not sure though.

PF isn't turned on on the BSD box; I'm still testing internally.


-----Original Message-----
From: Jacob Anderson [mailto:jwa(at)beyond-ordinary.com] 
Sent: Friday, May 22, 2009 2:58 PM
To: pound(at)apsis.ch
Subject: RE: [Pound Mailing List] Axis IP Cameras

Jeff,

Do the cameras send video packets using UDP? Since the camera connection
from Lynx says that it needs Java, I am thinking that it delivers a java
applet, and then starts a UDP feed on the camera to send back to the
java applet, which then displays UDP video packets. That's a very common
LAN video design.

Try connecting to the camera through apache proxy and see if you get the
same result.

I do not believe that pound can redirect UDP traffic.
[...]

Here's the URL of the camera directly.
http://outsidecamera/view/view.shtml?id=44192&imagePath=/mjpg/video.mjpg
&size=1 
Within the URL address bar of the webbrowser, it only shows
http://outsidecamera/view/view.shtml

I'm trying to configure Pound to use this "?id=xxxxx" portion of the URL
for it's session tracking, but this too doesn't work.  When I try this,
I get a "The service is not available. Please try again later." error.

I've tried "pound -c -v -f /etc/pound/pound.cfg" and it reports the
config file is ok.  So I start pound and ps is reporting it started.
Yet, there's nothing showing up within the messages logs and I get the
error above.

Any ideas?

Pound.cfg
User	"proxy"
Group	"proxy"
LogLevel	3
Alive	10

ListenHTTP
	Address	xxx.xxx.xxx.xxx
	Port	80
End
Service
	URL	".*id=.*"
	HeadRequire	"Host:.*outsidecamera.*"
	BackEnd
		Address	xxx.xxx.xxx.xxx
		Port	80
		Timeout	300
	End
	Session
		Type	URL
		ID	"id"
		TTL	300
	End
End



-----Original Message-----
From: Maze, Jeffrey S. [mailto:JMaze(at)CO.GEAUGA.OH.US] 
Sent: Friday, May 22, 2009 2:32 PM
To: pound(at)apsis.ch
Subject: [Pound Mailing List] Axis IP Cameras

Hello,
	I installed Pound onto an openBSD 4.5 box and it works fine when
connecting to any internal websites.  It was amazingly easy to setup and
get running.  Thanks for a great product.
	Anyway, the true purpose to getting Pound up and running was so
we could connect to our internal IP security cameras from externally on
the internet.  They're Axis cameras and can be brought up via a web
browser.
	When I connect to our pound box, the web browser title and tab
will show the name of the camera (eg. Live view - AXIS 215 PTZ Network
Camera) and then it sits there until the time-out period.  I then get
the "service is unavailable."
	If I do a netstat of the BSD box, it shows my IP connecting into
Pound, but the IP address of the camera shows a SYN_SENT.  I can lynx to
the camera's IP from the box fine (it reports I need a browser with
Java, etc) and can ping and everything fine.
	Anyone try a configuration such as this or have something I can
try to get this working.
	Thanks.. -Jeff

--
To unsubscribe send an email with subject unsubscribe to pound(at)apsis.ch.
Please contact roseg(at)apsis.ch for questions.