I'm trying to decide whether to run our web logfile statistics program
(analog/reportmagic) off of the pound generated logs, or continue to run
them off the webservers logfiles. I can see some pros and cons for each
source, but was wondering if anyone here with high volume setups can suggest
reasons for doing one or the other that I may not be thinking of. Bear in
mind we will be having the logs written 'across the wire' via spreadlog or
via syslog to a central logging machine, so no, we're not talking about
running reportmagic/analog on the pound (or web) servers in any case.

Our webservers get around the neighborhood of 10 million page views per
month, with a moderate amount of SSL. Given that traffic and that the pound
server will be handling the SSL and possibly writing logfiles to the network
(not hard drive), can someone suggest a processor to use in the pound
machine, or at least suggest single vs. dual?

Any guidance appreciated!

Jay West


---
[This E-mail scanned for viruses by Declude Virus]

On Tuesday 01 June 2004 20.17, Jay West wrote:[...]

If I read your numbers correctly this is about 4 pages/second, which is rather 
low traffic. Just about anything (from a Pentium II and up) would support 
this - certainly any modern processor (one of our clients does a comfortable 
30-40 reqs/second AVERAGE on a single Duron 1GHz, and that is far from 
saturating the CPU).

Using the back-end log files: you may have to modify the logging method as all 
requests appear to come from the Pound server. Use X-Forwarded-for instead.

Make sure your network supports your volume. This is probably more important 
than the CPU as you need to move both your contents and the syslog. For very 
large "pages" (music/video) you may want to consider Gigabit.[...]

On Tue, Jun 01, 2004 at 01:17:42PM -0500, Jay West wrote:[...]

I highly recommend mod_log_sql for aggregating apache logs from multiple
back-end servers.  If you centralize your logs on a database server, it
eliminates the need for slow, awkward "log merge" processes for analysis
packages that require log lines in chronological order.

There are certainly some gotchas, and it does require learning a bunch
of new stuff, but IMHO, the benefits far outweigh these issues.

See http://www.outoforder.cc/projects/apache/mod_log_sql/
for details.
[...]

Perhaps you missed Robert's response to my email on this yesterday, a
few hours before your post.  In answer to my query re 8-12 million hits
per day, he said:

| That is not a very high load (average under 150 requests/second, thus
| probably under 400 reqs/sec peak). A single machine (possibly dual
| CPU) would deal easily with that, provided you really don't need SSL.
| If you do: investing in a hardware accelerator would be a good idea.

Your 10 million hits per month is 4 requests per second.  I've got one
pound installation doing almost that much traffic on a Cyrix 6x86 (486
class), albeit without SSL.  Load average remains under 0.2 as long as I
don't ssh in to the box.  ;-)
[...]