Free using log files software for linux

Log Mine is a tool that produces reports on usage patterns on your Web site.
Web server log files are not just hit counters. They contain valuable information about the usage patterns of your website. Unforunately many web log analysis tools lay emphasis on telling you how many hits your site had or how many pages were seen and how many bytes were transferred.
A more usefull statistic would be which percentage of users came to your site went to a product information page, and which percentage of those users hit the checkout button, and which percentage actually completed their order. The trouble is the very nature of the web makes it nearly impossible to get accurate figures for such statistics.
However over periods of time, the errors present average out and it is possible to get a good indication of these ratios by properly mining the log file. That brings us back to square one, how do we get this information with traditional log analysers?
Traditional log analysers will produce weekly, monthly or daily charts for the usage of your site, but rarely do they allow you to create such charts for individual pages or referrrs - something very usefull if you run advertising campaigns on other sites.
Enter Log Mine. This new web log analyser / Mining tool will allow you to create just about any kind of report from the contents of your log file. Log Mine is not concerned about speed and it will be very greedy when it comes to taking up space on your hard disk/database but it will let you change your reporting without having to process gigabytes of log files each time.
Enhancements:
- Importing of Web server log files into the database was simplified.
- Multiple log files can now be processed at once.
- A bug in the monthly report was fixed.

IPTables log analizer displays Linux 2.4 iptables logs (rejected, acepted, masqueraded packets...) in a nice HTML page (it support rough netfilter logs but also Shorewall and Suse Firewall logs).

This page shall be easy to read and understand to reduce the manual analysis time.

This page containts statistics on packets and links to more detailled information on a given host, port, domain and so on.

To convice you, here is a typical syslog entry for iptables :

[IPTABLES DROP] : IN=ppp0 OUT= MAC= SRC=172.186.2.157 DST=193.253.186.217 LEN=36 TOS=0x00 PREC=0x00 TTL=115 ID=4775 PROTO=ICMP TYPE=8 CODE=0 ID=512 SEQ=3663

How does it work ?

A small deamon is launched by a user which can read iptables logs files. Each time a new packet is logged, the daemon insert a new row in the database.

The statistics and so on are elaborated by the PHP page itself.

klogview is a KDE real-time log file viewer, like tail -f. The main window contains any number of dockable log panels, with an arbitrary number of log sources in each of them.
Every log source can have a separate font color and style. Other features include filters, alerts, different encodings, and tray support.
Main features:
- Any number of dockable log panels
- Any number of log sources per log panel
- "File" log source
- "Process output" log source
- Configurable fonts and colors
- Filters and alerts
- Tray icon
Enhancements:
- Removed forced autoscroll
- Ability to reopen log files
- Ability to enable/disable log sources stop filter flag

mergelog provides a fast tool to merge HTTP log files by date.
mergelog is a small and fast C program, which merges HTTP log files by date in Common Log Format (Apache default log format) from Web servers, behind round-robin DNS.
It has been designed to easily process huge logs from highly stressed servers, and can manage gzipped files.
Enhancements:
- dont abort anymore on corrupted log lines
- set BUFFER_SIZE value to 32Ko
- fixed a major bug on a broken month initialization
- corrections on manpages
- fix in configure.in to abort if zlib is not present
- fixed a potential segmentation fault on malformed log lines
- autoconf compliant thanks to Igor Genibel

Logger::Simple is an implementation of the Simran-Log-Log and Simran-Error-Error modules.

SYNOPSIS

use Logger::Simple;
my $log=Logger::Simple->new(LOG=>"/tmp/program.log");
my $x=5;my $y=4;

if($x>$y){
$log->write("$x is greater than $y");
}

new

my $log=Logger::Simple->new(LOG=>"/tmp/logfile");

The new method creates the Logger::Simple object as an inside-out object. The Log parameter is a mandatory one that must be passed to the object at creation, or the object will fail. Upon creation, this method will also call the open_log method which opens the log file.

write

$log->write("This is an error message");

This method will write a message to the logfile, and will update the internal HISTORY array.

retrieve_history

my @history = $log->retrieve_history; my $msg = $log->retrieve_history;
When called in scalar context, it will return the last message written to the HISTORY array. When called in a list context, it will return the entire HISTORY array

clear_history

$log->clear_history;

This method will clear the internal HISTORY array

This module is based on the Simran::Log::Log and Simran::Error::Error modules. I liked the principle behind them, but felt that the interface could be a bit better.

My thanks also goes out once again to Damian Conway for Object Oriented Perl, and also to Sam Tregar, for his book "Writing Perl Modules for CPAN". Both were invaluable references for me.

I would also like to thank Jerry Heden for his Object::InsideOut module, which I used to create this module.

LogMiner is a powerful log analysis package for Apache (or other web servers using the combined log format).
LogMiner can extract and present several reports, about visits, hits, traffic, requests, navigation paths, browsers and OSs used by users and so on.
Data is stored in a PostgreSQL database, using a schema which has been optimized to reduce redundancy at minimum.
LogMiner has been inspired by the popular Webalizer package, but it differs in certain core points:
data is stored in a DBMS backend and reports are generated on-the-fly, while Webalizer generates plain html files. A DBMS allows to extract and aggregate data in many ways, whenever you need. A drawback is that you wont have the processing speed of Webalizer when parsing log files.
Webalizer only indexes the last 12 months of data: previous reports are still on disk, but arent accessible from any link. This is the most annoying thing of Webalizer, in my opinion: Im sick of having hundreds of unused files around
LogMiner allows to navigate to previous months easily.
Webalizer reports are hardcoded in the program. LogMiner implements reports in a more extensible way. Each report is in fact a simple PHP class, usually supported by a PL/pgSQL function (although youre free to insert your SQL queries in the PHP code if you like).
LogMiner offers more reports than Webalizer: for instance, the OS charts and the navigation graphs.
Depending on your needs, you might prefer LogMiner over Webalizer, especially if you like having a central SQL repository for your data which enables you to extract the data you need at any time, or to add a kind of report which wasnt planned from the start and apply it to older data.
If you dont have a PostgreSQL database or you expect to process huge log files in seconds, go for Webalizer.
Enhancements:
- This release adds support for libpqxx 2.6.9.
- SQL errors are now logged and dont interrupt the parsing of log files.

DKP Log Parser (DKPLP) is a tool designed to help administrate time-based and/or event-based DKP reward system in conjunction with EQDKP (or any other DKP software implementing a required interface). The project is configurable to be compatible with any game that produces any kind of log. There are currently predefined pattern sets for EverQuest, EverQuest2 and World of Warcraft (with CT RaidTracker). If you are playing another game then you can either write new patterns for it yourself, or ask the the forum (be sure to include a sample log though).
DKP Log Parsers goal is to reduce the administrative overhead of running complex DKP reward system, by parsing the logs, calculating the DKP per person and sending the results (the loot, raids, participants and DKP) directly to EQDKP (or other DKP software). Its widely configurable, allowing it to be used by the many variations of zero-sum and time-based DKP around.
Overview of how DKP Log Parser is used
DKP Log Parser is a software program run on a users desktop (Windows or Unix). It reads in the contents of a game log once a raid has been completed, parses all the information and displays the data for administrators to edit. Once happy with the information, the administrator uploads the data from their PC to the DKP Log Parser plugin on an EQDKP webserver (or any other server implementing the required DKPLP interface), where it can be viewed by all members of the DKP system.
Main features:
- Inbuilt support for Everquest, Everquest II, World of Warcrafts CT RaidTracker plugin, and Ventrilo.
- Extensible enough to be able to handle most logs.
- Time based dkp:
- Accrue DKP by minute, by interval, by event or by a combination of these.
- Zero-sum DKP can be enabled (by minute or by interval) or disabled.
- Tag certain intervals as being more or less important by using weightings to adjust the DKP.
- Data export:
- Upload data to for example EQDKP, select from a number of different data representations.
- Export to plain text
- Export to BB forum markup
- Export to wiki markup
- Export to XML
- Member list and alt character list synchronises with server, making it easy to share setups with other users.
- Optional autocompletion database to remember item names and values between sessions.
- Preprocessors available to perform other operations on the log file. The skys the limit!