Hello,

is there a possibility to get the requests per second generated from 
pound? I've deactivated logging on my backendservers, so I can't get 
these value from them ...

With poundctl I can only monitor the active sessions, but not the active 
requests/minute - right? The logfile grows to fast to get therefrom any 
infos with grep, wc or such things :(

I'm very interested in any suggestions.

Thanks in advance,
Robert

I wrote a munin plugin for it. Could probably be done much better but
it works for me. Feel free to hack it apart or improve it.


  #!/bin/bash

if [ "$1" = "autoconf" ]; then
 echo yes
 exit 0
fi

if [ "$1" = "config" ]; then

 echo 'graph_title Pound Requests per Second'
 echo 'graph_args --base 1000 -l 0 '
 echo 'graph_vlabel number of requests'
 echo 'graph_category Pound'
 echo 'graph_info This graph shows the number of Pound Requests per second.'
 echo 'pound.label Requests Per Second'
 echo 'pound.draw LINE2'
 echo 'pound.info The number of Pound requests per second.'
 #echo 'poundavg.label Avg Time for Requests (Sec)'
 #echo 'poundavg.draw LINE2'
 #echo 'poundavg.info Average Request Time (Sec).'
 exit 0
fi

# START CODE
function cleanup {
  unset FILE
  unset TMPFILE
  unset TMPFILE2
  unset TMPFILE3
  unset LOGFILE
  unset MONTH
  unset DAY
  unset HOUR
  unset MINUTE
  unset SECOND
  unset LASTMESSAGE_TOTAL
  unset LASTLINE
  unset ORIGLINE
  unset LASTMESSAGE
  unset LASTMESSAGES
  unset CHECK
  unset FOUND
  unset LINES
  unset RANGE
  unset RPS
  unset TOTAL
  unset COUNTSEC
  unset COUNTMIN
  unset COUNTHOUR
  unset COUNT
  unset LAST
  unset FINISHED
}

cleanup
FILE="/var/log/syslog"
TMPFILE="/dev/shm/munin_pound"
TMPFILE2="/dev/shm/munin_pound2"
TMPFILE3="/dev/shm/munin_pound3"
LOGFILE="/root/munin_pound.log"
rm $TMPFILE $TMPFILE2 $TMPFILE3 &> /dev/null
touch $TMPFILE $TMPFILE2 $TMPFILE3
date >> $LOGFILE
MONTH=`date +%b --date="5 minutes ago"`
DAY=`date +%d --date="5 minutes ago" | sed s/^0//g`
HOUR=`date +%H --date="5 minutes ago"`
MINUTE=`date +%M --date="5 minutes ago"`
SECOND=`date +%S --date="5 minutes ago"`
LASTMESSAGE_TOTAL=0
COUNT=0
function last_message {
  LASTLINE="`tail -n1 $TMPFILE`"
  ORIGLINE=`grep -n -e "$LASTLINE" $FILE | head -n1 | cut -d: -f1`
  tail -n "+$ORIGLINE" $FILE > $TMPFILE2
  LASTMESSAGES=`grep -n -e "last message repeated [0-9]\+ times" $TMPFILE2 |
cut -d: -f1`
  echo $LASTMESSAGES >> $TMPFILE3
  echo starting >> $TMPFILE3
  for LASTMESSAGE in $LASTMESSAGES; do
    LASTMESSAGE=`echo $LASTMESSAGE - 1 | bc`
    CHECK=`perl -nwe "$LASTMESSAGE..$LASTMESSAGE and print" $TMPFILE2`
    FOUND=`echo $CHECK | grep -c -e "pound: [1-9]\+."`

    if [ $FOUND -eq 1 ]; then
      FOUND=0
      LASTMESSAGE=`echo $LASTMESSAGE + 1 | bc`
      CHECK=`perl -nwe "$LASTMESSAGE..$LASTMESSAGE and print" $TMPFILE2`
      FOUND=`echo $CHECK | sed s/".*last message repeated"//g | sed s/"[a-zA-Z
]"//g`
      LASTMESSAGE_TOTAL=`echo $LASTMESSAGE_TOTAL + $FOUND | bc`
    fi
  done
}
function finish {
  if [ $FINISHED -eq 1 ]; then
    last_message
    LINES=`cat $TMPFILE | wc -l`
    LINES=`echo $LINES + $LASTMESSAGE_TOTAL | bc`

    #cat $TMPFILE | grep -o -e "[0-9]\+\.[0-9]\+\ sec" | cut -d\  -f1 >
$TMPFILE2
    #TIME=`xargs echo < $TMPFILE2 | sed s/\ /\+/g | bc`
    #echo $TIME $LINES
    #AVGTIME=`echo "scale=3; $LINES / $TIME" | bc`
    #echo poundavg.value $AVGTIME


    RANGE=`echo "60 * 5" | bc`
    RPS=`echo "scale=2; $LINES / $RANGE" | bc`
    echo "pound.value $RPS"
    cleanup
    exit
  fi
}

function cut_old {
  TOTAL=`cat $TMPFILE | wc -l`
  if [ ${LAST:=0} -lt $TOTAL ]; then
    head -n $LAST $TMPFILE > $TMPFILE2
    rm $TMPFILE && mv $TMPFILE2 $TMPFILE
    unset LAST
  fi

  finish
}
function roll_time {
  SECOND=`echo $SECOND | sed s/^0//g`
  MINUTE=`echo $MINUTE | sed s/^0//g`
  HOUR=`echo $HOUR | sed s/^0//g`
  if [ $SECOND -eq 60 ]; then
    SECOND=0
    let "MINUTE += 1"
  fi
  if [ $MINUTE -eq 60 ]; then
    MINUTE=0
    let "HOUR += 1"
  fi
  if [ $HOUR -eq 24 ]; then
    HOUR=0
    let "DAY += 1"
  fi
  COUNTSEC=`echo $SECOND | wc -c`
  COUNTMIN=`echo $MINUTE | wc -c`
  COUNTHOUR=`echo $HOUR | wc -c`
  if [ $COUNTSEC -eq 2 ]; then
    SECOND=`echo $SECOND | sed s/^/0/g`
  fi
  if [ $COUNTMIN -eq 2 ]; then
    MINUTE=`echo $MINUTE | sed s/^/0/g`
  fi
  if [ $COUNTHOUR -eq 2 ]; then
    HOUR=`echo $HOUR | sed s/^/0/g`
  fi
}
tac $FILE | grep -e "pound: [1-9]\+." > $TMPFILE
LAST=`nice -n19 grep -n -e "^$MONTH[ ]\+$DAY[ ]\+" $TMPFILE | tail -n1 | cut
-d: -f1`
FINISHED=`cat $TMPFILE | wc -l | grep -c -e "^0$"`
cut_old
LAST=`nice -n19 grep -n -e "^$MONTH[ ]\+$DAY[ ]\+$HOUR" $TMPFILE | tail -n1 |
cut -d: -f1`
FINISHED=`cat $TMPFILE | wc -l | grep -c -e "^0$"`
cut_old
LAST=`nice -n19 grep -n -e "^$MONTH[ ]\+$DAY[ ]\+$HOUR:$MINUTE" $TMPFILE | tail
-n1 | cut -d: -f1`
FINISHED=`cat $TMPFILE | wc -l | grep -c -e "^0$"`
cut_old
# FIX 3 to 300
while [ ${LAST:=0} -eq 0 ] && [ ${COUNT:=0} -lt 300 ]; do
  LAST=`nice -n19 grep -n -e "^$MONTH[ ]\+$DAY[ ]\+$HOUR:$MINUTE:$SECOND"
$TMPFILE | tail -n1 | cut -d: -f1`
  SECOND=`echo $SECOND + 1 | bc`
  COUNT=`echo $COUNT + 1 | bc`
  roll_time
done
echo $COUNT >> $LOGFILE
if [ $COUNT -eq 300 ]; then
  echo "pound.value 0"
  cleanup
  exit
fi
FINISHED=1
cut_old

  



It take a bit of CPU time while there aren't many logs but the more
there are, the faster it goes.

 
 



Regards,
Jon Higgs


Robert Klikics wrote:
Hello,
  
  
is there a possibility to get the requests per second generated from
pound? I've deactivated logging on my backendservers, so I can't get
these value from them ...
  
  
With poundctl I can only monitor the active sessions, but not the
active requests/minute - right? The logfile grows to fast to get
therefrom any infos with grep, wc or such things :(
  
  
I'm very interested in any suggestions.
  
  
Thanks in advance,
  
Robert
  
  
--
  
To unsubscribe send an email with subject unsubscribe to
pound(at)apsis.ch.
  
Please contact roseg(at)apsis.ch for questions.
  
  




Attention: This e-mail is 
privileged and confidential. If you are not theintended recipient please 
delete the message and notify the sender.Any views or opinions presented are 
solely those of the author.

This e-mail message has been scanned and cleared by MailMarshal -
www.marshalsoftware.com and Sophos Anti-virus 




 

Hi,

thanks, but this looks very complicated for me ;)

I've just written a small script in PHP with shell-commands for the 
usage with cacti and it seems to work fine:

<?php
$tstamp = date("H:i", strtotime("-1 minute"));
$call   = "tail -n 200000 /var/log/daemon.log|grep $tstamp|wc -l";
$num    = `$call`;
$num    = $num / 60;

echo round($num);
?>

The CPU usage is near zero here, and cacti is only polling every 5 
minutes in my config.

But it would be fine if pound gives out these data in an further release 
...

Regards,
Robert


Jon Higgs schrieb:[...]

hehe. yeah it was a little complicated. It started out being quite 
simple but needed to be expanded upon (many times) to make it work 
reliably and accurately.

If it helps you to create your own, this is roughly how I made it work:

Dump the log to a temp file.
Pull out the crap.
Find the first line logged after 5 minutes ago.
Count the lines between then and now.
Add the "this message was repeted x times" to the total.
Divide the total by (60*5) to get hits per second.


The reason mine is slow when using a small log file is because say the 
times that are being searched are between 5:00:00 and 5:05:00, it will 
grep for 5:00:00 and if not found then grep 5:00:01 and so on in a loop. 
If logs are big then it will match first go and break out, otherwise it 
could check for every second for 5 minutes (300 times).

At the time I couldn't think of a better way to do it although I'm sure 
there is a better way.


Robert Klikics wrote:[...][...][...]
######################################################################
Attention: 
This e-mail message is privileged and confidential. If you are not the 
intended recipient please delete the message and notify the sender. 
Any views or opinions presented are solely those of the author.

This e-mail has been scanned and cleared by MailMarshal and 
Sophos Anti-Virus

######################################################################

Sorry, I don't have any suggestion, I just want to show support :) Let's 
formulate it as a feature request:

Pound will have an internal counter(s) for each backend, this counter 
would be incremented on each request. There would ideally be two 
counters, one for successful requests, one for failed.

Also, there would be a poundctl functionality to print these counters, 
something like:

Timestamp,Listener,Service,Backend,Success,Failure

ie.
...
1189587846,1,2,3,5136844684,5
1189587846,1,2,4,5136849871,2

And, of course, a switch to reset these counter.

This printout would be easily parseable from Zabbix, MRTG or whatever.

Robert, what do you think? I suppose that should't be too difficult.

MT.


Robert Klikics napsal(a):[...]

[...]

That or having a switch to get an Apache format log output.
Then we could use any of a gazillion tools made for Apache...
[...]

Robert Klikics wrote:[...]


Hi, correct me if I am wrong, but this script will really only roughly
report how many requests were given during the minute before the script
was called.  So if you are calling it every 5 minutes, you are only
getting the count for 1/5th of the day.

In my opinion, you could get a better count by either calling it every
minute (not preferred), or by calling the script every hour, and
switching your search to look for any entry within the previous hour.

so:

  $tstamp = date("M n H:", strtotime("-1 hour"));

Hi,

because I wrote it for the usage with MRTG/Cacti it works fine. I just 
wanted a value to print a graph for average requests/sec. - and that it 
what it does. Or not?

1 hour isn't possible, because 'grep' needs to long for this. We've got 
~400 reqs/sec, so the log grows too fast ...

Regards,
Robert


Kenneth Burgener schrieb:[...]

Hi,

Robert Klikics wrote:[...][...][...]

I knocked off a quick Pound hit rate counter in perl tonight. The source
and documentation are at:

http://apthorpe.cynistar.net/code/pound/

It uses File::Tail to efficiently tail the logfile (/var/log/syslog by
default) and Parse::Syslog to simplify the code and periodically writes
results to /var/tmp/poundcounter.log. You can change both of those files
via command-line options. The other potentially weird module is
Getopt::Euclid which is used to keep the command-line options consistent
with the POD documentation.

The only known bugs are cosmetic problems with Getopt::Euclid not
properly parsing svn version numbers, and possible spurious hit rates at
low volumes (if there's less than one new syslog entry per minute its
possible for the counter to not write to the stats file, leading to
stale values "flatlining" until there's more syslog activity.)

poundcounter is fully documented and passes cleanly through both
perltidy and perlcritic (in pedantic mode, no less...) so it should be a
reasonable starting point for a logwatching daemon.

hth,
[...]

Francois Rejete napsal(a):[...]

Well, for example, for us, that would be an overkill. We get in peaks 
600 req/sec. Now, pound can handle that, no problem. But if we turn on 
logging, that's a whole different story. We'd have to use a server with 
some I/O muscle as the logs grow a hefty 500MB an hour.

We do apache-style logging on the backends, so the only thing we lack at 
the moment are those really nice MRTG graphs showing how many req/sec we 
handle during the day :).
[...]

Hi,

Michal Taborsky - Internet Mall wrote:[...][...][...]

Or buy a beefy loghost and trade the I/O load for network load on the
webserver (drop in another fast NIC and set up and admin network,
etc...) You should be able to direct the pound syslog traffic to a
remote box by poking at /etc/syslog.conf.

Of course, this is trading one problem for another... :)
[...]

Michal Taborsky - Internet Mall schrieb:[...]

Not with my minimal script a few postings ago ;-)
Should also work for you and does not cause any load.

Or do you have pound's logging disabled completely?
We're running with LogLevel 1 without any problems (logfile is 3-5 
GBytes a day) ...

Regards,
Robert

Hi,

just tested, works fine! It brings out the same value's as my solution, 
so both scripts are probably calculating correct values ...

Thanks,
Robert


Bob Apthorpe schrieb:[...]

On Wed, 2007-09-12 at 16:04 +0200, Michal Taborsky - Internet Mall
wrote:[...]

Surprisingly enough, it is difficult: on machines with small word size
you can easily run into integer overflow problems. Even with 32 bit
integers you'll get an overflow after about one month, assuming 400
requests per second.

The mechanism currently implemented in Pound to support dynamic
rescaling keeps a history of sorts, but the values are periodically
rescaled. See the code in svc.c, in upd_be().

This means we can have a kind of average response time (which is printed
by poundctl), but not the total number of requests.[...]

On Thu, 2007-09-13 at 10:04 +0900, Francois Rejete wrote:[...]

If told to do so, Pound writes its log in Apache format. You still have
to choose which of the "gazillion" tools you want to use...[...]

Mrtg has some preetty slick methods for dealing with counter overflow.
Basically, it diagnoses a catastrophic decrease in a counter that is
supposed to only increase.  It then adjusts the delta to get the right
result.  The final results are pretty nice.


On 9/13/07 9:15 AM, "Robert Segall" <roseg(at)apsis.ch> wrote:
[...][...][...]