Free log analyzers software for linux

IPTables log analizer displays Linux 2.4 iptables logs (rejected, acepted, masqueraded packets...) in a nice HTML page (it support rough netfilter logs but also Shorewall and Suse Firewall logs).

This page shall be easy to read and understand to reduce the manual analysis time.

This page containts statistics on packets and links to more detailled information on a given host, port, domain and so on.

To convice you, here is a typical syslog entry for iptables :

[IPTABLES DROP] : IN=ppp0 OUT= MAC= SRC=172.186.2.157 DST=193.253.186.217 LEN=36 TOS=0x00 PREC=0x00 TTL=115 ID=4775 PROTO=ICMP TYPE=8 CODE=0 ID=512 SEQ=3663

How does it work ?

A small deamon is launched by a user which can read iptables logs files. Each time a new packet is logged, the daemon insert a new row in the database.

The statistics and so on are elaborated by the PHP page itself.

Funnel Web Analyzer provides essential Web site visitor and traffic analysis.
Every person who visits your companys Web site leaves a trail behind. Your Web server logs their every action and you can use this information to help improve your business. Funnel Web Analyzer provides essential Web site visitor and traffic analysis. It measures everything from server load and referrals to visitor demographics and marketing ROI.
Funnel Web Analyzer helps you optimize your web site by allowing you to analyze how users interact with your site and helps you make informed decisions about what changes you can make to improve their experience.
- What are the demographics of my customers?
- How many unique customers do I have, and how many of these are repeat visitors?
- How long does a visitor spend on each page?
- Which streaming media files are most popular?
- What are my most popular pages?
- How successful was my banner advertisement in attracting visitors to my site?
- What are the peak periods of activity on my site?
- What downloads are most popular?
- What search terms are people using to find my site?
Comprehensive Visitor Analysis
Give your executives and managers unique insight into what your customers are really interested in. Analyzer generates more than 50 different reports and graphs in 12 different languages. Reports can even be customized to reflect your corporate identity and published in a variety of formats including PDF, HTML, Word and Excel.
Identify where users come from before they arrive at your site with the referrals report. This is a great way to measure the effectiveness of promotional campaigns, press releases and affiliate/partner programs. You can even generate separate advertising reports to measure the click-through rates of individual banner ad campaigns.
Funnel Web Analyzer is freeware and provides everything you need to quickly analyze the traffic on your Web site. With virtually no setup, you get an extensive set of valuable reports right from the start.

Practical Query Analyzer produces HTML reports on the most frequent queries, slowest queries, queries by type (select/insert/update/delete), and more for both PostgreSQL and MySQL log files.

Log Mine is a tool that produces reports on usage patterns on your Web site.
Web server log files are not just hit counters. They contain valuable information about the usage patterns of your website. Unforunately many web log analysis tools lay emphasis on telling you how many hits your site had or how many pages were seen and how many bytes were transferred.
A more usefull statistic would be which percentage of users came to your site went to a product information page, and which percentage of those users hit the checkout button, and which percentage actually completed their order. The trouble is the very nature of the web makes it nearly impossible to get accurate figures for such statistics.
However over periods of time, the errors present average out and it is possible to get a good indication of these ratios by properly mining the log file. That brings us back to square one, how do we get this information with traditional log analysers?
Traditional log analysers will produce weekly, monthly or daily charts for the usage of your site, but rarely do they allow you to create such charts for individual pages or referrrs - something very usefull if you run advertising campaigns on other sites.
Enter Log Mine. This new web log analyser / Mining tool will allow you to create just about any kind of report from the contents of your log file. Log Mine is not concerned about speed and it will be very greedy when it comes to taking up space on your hard disk/database but it will let you change your reporting without having to process gigabytes of log files each time.
Enhancements:
- Importing of Web server log files into the database was simplified.
- Multiple log files can now be processed at once.
- A bug in the monthly report was fixed.

Log::Localized is a Perl module to localize your logging.

SYNOPSIS

What you most probably want to do is something like:
package Foo;
use Log::Localized;

sub bar {
# this message will be displayed if method bars verbosity is >= 1
llog(1,"running bar()");
}

# this message will be displayed if package Foos verbosity is >= 3
llog(3,"loaded package Foo");
Then paste the following local verbosity rules in a file called verbosity.conf, in the same directory as your program:
# log everything from wherever inside Foo and its subclasses, up to level 3
Foo:: = 3
# except for function Foo::foo who shall have verbosity 0
Foo::bar = 0

SYNOPSIS - ADVANCED

In a program accepting command line arguments, you may want to do:
use Getopt::Long;
use Log::Localized log => 1;

GetOptions("verbose|v+" => sub { $Log::Localized::VERBOSITY++; } );

llog(1,"you used -v");
llog(2,"you used -v -v");
You may alter local verbosity from within the running code:
package Foo;
use Log::Localized log => 1;

# verbosity level is 0 by default

{
# set verbosity locally in this block
local $Log::Debug::VERBOSITY = 5;
llog(5,"this will be logged");
}

debug(5,"but this wont");
If you want to import llog under another name in the calling module:
package Foo;
use Log::Localized rename => "my_log";

# call Log::Localized::llog()
my_log(1,"renamed llog()");
See the examples directory in the module distribution for more real life examples.

Log::Localized provides you with an interface for defining dynamically exactly which part of your code should log messages and with which verbosity.

Log::Localized addresses one issue of traditional logging: in very large systems, a slight increase in logging verbosity usually generates insane amounts of logs. Hence the need of being able to turn on verbosity selectively in some areas of code only, in a localized way.

Log::Localized is based on the concept of local verbosity. Each package and each function in a package has its own local verbosity, set to 0 by default. With Log::Localized you can change the local verbosity in just a function, just a package or just a class hierarchy via a so called verbosity rule. Verbosity rules are passed to Log::Localized either via a configuration file or via an import parameter. By changing verbosity rules according to the needs of the moment, you can alter your programs logging flow in a very fine-grained way, and get logs from only the code areas you are interested in.

Log::Localized comes with default settings that make it usable out of the box, but its configuration options will let you redefine pretty much everything in its behavior.
The actual logging in Log::Localized is handled by Log::Dispatch.

Log::Dispatch::File::Alerts is a Perl object for logging to alert files.

SYNOPSIS

use Log::Dispatch::File::Alerts;

my $file = Log::Dispatch::File::Alerts->new(
name => file1,
min_level => emerg,
filename => Somefile%d{yyyy!!!!}.log,
mode => append );

$file->log( level => emerg,
message => "Ive fallen and I cant get upn" );

ABSTRACT

This module provides an object for logging to files under the Log::Dispatch::* system.

This module subclasses Log::Dispatch::File for logging to date/time stamped files. See Log::Dispatch::File for instructions on usage. This module differs only on the following three points:

alert files

This module will use a seperate file for every log message.

multitasking-safe

This module uses flock() to lock the file while writing to it.

stamped filenames

This module supports a special tag in the filename that will expand to the current date/time/pid.

It is the same tag Log::Log4perl::Layout::PatternLayout uses, see Log::Log4perl::Layout::PatternLayout, chapter "Fine-tune the date". In short: Include a "%d{...}" in the filename where "..." is a format string according to the SimpleDateFormat in the Java World (http://java.sun.com/j2se/1.3/docs/api/java/text/SimpleDateFormat.html). See also Log::Log4perl::DateFormat for information about further restrictions.
In addition to the format provided by Log::Log4perl::DateFormat this module also supports $ for inserting the PID and ! for inserting a uniq number. Repeat the character to define how many character wide the field should be.
A note on the !: The module first tries to find a fresh filename with this set to 1. If there is already a file with that name then it is increased until either a free filename has been found or it reaches 9999. In the later case the module dies.

ADMLogger is a log analyzing engine based on fireparse. After I had written fireparse, I had found myself writing little scripts based on it. Like one that told me what my FTP server did that day. And then more things like - were unauthorized machines trying to get IP addresses from my DHCP server? So I decided to rip out the engine of fireparse to create a generic log analyzer system. Using this core, I could easily build upon it with plugins. This also made it easier on my systems since there was now only one entry for cron, and I didnt have different code everywhere - a bug fix in fireparse had me searching the rest of my systems looking to see if I still used that line.
For the average Linux user, ADMLogger will probably only be a means to an end - that end being able to run fireparse. But with very little Perl programming knowledge, it may become a powerful tool in a System Administrators toolbox.
ADMLogger creates e-mail reports that can be formatted plain text or full HTML, which is up to the plugin designers to support. The main system has a HTML preference, so if your plugin ignores it, so be it. ADMLogger will also remove all filtered entries from the main syslog file into a second file so your other entries are more noticable.
Enhancements:
- Improved non-HTML output support for ALL plugins.

Analog is an application that will help you measure the usage on your web server. It tells you which pages are most popular, which countries people are visiting from, which sites they tried to follow broken links from, and all sorts of other useful information.
Main features:
- is very fast
- can handle big logfiles
- output in any of 33 different languages
- very configurable
- easy to install