Free log files software for linux

LogMiner is a powerful log analysis package for Apache (or other web servers using the combined log format).
LogMiner can extract and present several reports, about visits, hits, traffic, requests, navigation paths, browsers and OSs used by users and so on.
Data is stored in a PostgreSQL database, using a schema which has been optimized to reduce redundancy at minimum.
LogMiner has been inspired by the popular Webalizer package, but it differs in certain core points:
data is stored in a DBMS backend and reports are generated on-the-fly, while Webalizer generates plain html files. A DBMS allows to extract and aggregate data in many ways, whenever you need. A drawback is that you wont have the processing speed of Webalizer when parsing log files.
Webalizer only indexes the last 12 months of data: previous reports are still on disk, but arent accessible from any link. This is the most annoying thing of Webalizer, in my opinion: Im sick of having hundreds of unused files around
LogMiner allows to navigate to previous months easily.
Webalizer reports are hardcoded in the program. LogMiner implements reports in a more extensible way. Each report is in fact a simple PHP class, usually supported by a PL/pgSQL function (although youre free to insert your SQL queries in the PHP code if you like).
LogMiner offers more reports than Webalizer: for instance, the OS charts and the navigation graphs.
Depending on your needs, you might prefer LogMiner over Webalizer, especially if you like having a central SQL repository for your data which enables you to extract the data you need at any time, or to add a kind of report which wasnt planned from the start and apply it to older data.
If you dont have a PostgreSQL database or you expect to process huge log files in seconds, go for Webalizer.
Enhancements:
- This release adds support for libpqxx 2.6.9.
- SQL errors are now logged and dont interrupt the parsing of log files.

Regexp::Log is a Perl base class for log files regexp builders.

SYNOPSIS

my $foo = Regexp::Log::Foo->new(
format => custom %a %b %c/%d,
capture => [qw( host code )],
);

# the format() and capture() methods can be used to set or get
$foo->format(custom %g %e %a %w/%s %c);
$foo->capture(qw( host code ));

# this is necessary to know in which order
# we will receive the captured fields from the regexp
my @fields = $foo->capture;

# the all-powerful capturing regexp :-)
my $re = $foo->regexp;

while () {
my %data;
@data{@fields} = /$re/; # no need for /o, its a compiled regexp

# now munge the fields
...
}

LogMon project will monitor one or more log files, updating when more data is available ala tail -f , within a common terminal window via a "split window". User can scroll up/down/left/right through all the windows.
A header displays each frames file name, and number of lines in the file. Very useful when watching several log files at the same time. Ive tested this under Linux and FreeBSD (as of 0.3.3) and I have had reports of it working under MacOS X 10.4.7, but it should work under other Unices. Please let me know if you get it working on another OS.
Enhancements:
- Fixed a scroll-back bug when using page-up
- Added individual frame selection

Weblogmon is a program that shows current users and other usage information of a web site by monitoring the server log files. Users are identified by client IP address, HTTP username or usertrack cookie, depending on what information is available in the log.
This project is written entirely in Perl, and released under GNU General Public License.
Enhancements:
- No longer shows error messages on screen if resize cant be run< BR >
- Renamed option --numeric as --no-resolve

GNU-Monitor is a transactional monitor that allows client/server applications to be developed with minimal effort.
GNU-Monitor project consists of transaction routing modules that ensure data integrity and recovery on abnormal termination.
Enhancements:
- New services .get_server_list and .get_service_list to get configuration info using the gm_cfgview tool.
- A new service .log-level to dynamically change levels of information on log files.
- A final solution for some transaction bugs.

Log::Dispatch::Config is a Log4j for Perl.

SYNOPSIS

use Log::Dispatch::Config;
Log::Dispatch::Config->configure(/path/to/log.conf);

my $dispatcher = Log::Dispatch::Config->instance;
$dispatcher->debug(this is debug message);
$dispatcher->emergency(something *bad* happened!);

# automatic reloading conf file, when modified
Log::Dispatch::Config->configure_and_watch(/path/to/log.conf);

# or if you write your own config parser:
use Log::Dispatch::Configurator::XMLSimple;

my $config = Log::Dispatch::Configurator::XMLSimple->new(log.xml);
Log::Dispatch::Config->configure($config);

Log::Dispatch::Config is a subclass of Log::Dispatch and provides a way to configure Log::Dispatch object with configulation file (default, in AppConfig format). I mean, this is log4j for Perl, not with all API compatibility though.

METHOD

This module has a class method configure which parses config file for later creation of the Log::Dispatch::Config singleton instance. (Actual construction of the object is done in the first instance call).

So, what you should do is call configure method once in somewhere (like startup.pl in mod_perl), then you can get configured dispatcher instance via Log::Dispatch::Config->instance.

Log::Log4perl::Layout::XMLLayout is an XML Layout.

SYNOPSIS

use Log::Log4perl::Layout::XMLLayout;

my $app = Log::Log4perl::Appender->new("Log::Log4perl::Appender::File");

my $logger = Log::Log4perl->get_logger("abc.def.ghi");
$logger->add_appender($app);

# Log with LocationInfo
my $layout = Log::Log4perl::Layout::XMLLayout->new(
{ LocationInfo => { value => TRUE },
Encoding => { value => iso8859-1}});

$app->layout($layout);
$logger->debug("Thats the message");

########################### Log4perl Config File entries for XMLLayout
log4perl.appender.A1.layout = Log::Log4perl::Layout::XMLLayout
log4perl.appender.A1.layout.LocationInfo = TRUE
log4perl.appender.A1.layout.Encoding =iso8859-1
###########################

Creates a XML layout according to http://jakarta.apache.org/log4j/docs/api/org/apache/log4j/xml/XMLLayout.html
Logfiles generated based on XMLLayout can be viewed and filtered within the log4j chainsaw graphical user interface (see example section below). chainsaw is part of the JAVA based log4j package and can be downloaded from http://jakarta.apache.org/

The output of the XMLLayout consists of a series of log4j:event elements as defined in the log4j.dtd. It does not output a complete well-formed XML file. The output is designed to be included as an external entity in a separate file to form a correct XML file.

For example, if abc is the name of the file where the XMLLayout ouput goes, then a well-formed XML file would be:

< ?xml version="1.0" ? >
< !DOCTYPE log4j:eventSet SYSTEM "log4j.dtd" [< !ENTITY data SYSTEM "abc" >] >

< log4j:eventSet version="1.2" xmlns:log4j="http://jakarta.apache.org/log4j/" > &data; < /log4j:eventSet >

This approach enforces the independence of the XMLLayout and the appender where it is embedded.

The version attribute helps components to correctly intrepret output generated by XMLLayout. The value of this attribute should be "1.1" for output generated by log4j versions prior to log4j 1.2 (final release) and "1.2" for relase 1.2 and later.