Free log files in linux software for linux

Log Mine is a tool that produces reports on usage patterns on your Web site.
Web server log files are not just hit counters. They contain valuable information about the usage patterns of your website. Unforunately many web log analysis tools lay emphasis on telling you how many hits your site had or how many pages were seen and how many bytes were transferred.
A more usefull statistic would be which percentage of users came to your site went to a product information page, and which percentage of those users hit the checkout button, and which percentage actually completed their order. The trouble is the very nature of the web makes it nearly impossible to get accurate figures for such statistics.
However over periods of time, the errors present average out and it is possible to get a good indication of these ratios by properly mining the log file. That brings us back to square one, how do we get this information with traditional log analysers?
Traditional log analysers will produce weekly, monthly or daily charts for the usage of your site, but rarely do they allow you to create such charts for individual pages or referrrs - something very usefull if you run advertising campaigns on other sites.
Enter Log Mine. This new web log analyser / Mining tool will allow you to create just about any kind of report from the contents of your log file. Log Mine is not concerned about speed and it will be very greedy when it comes to taking up space on your hard disk/database but it will let you change your reporting without having to process gigabytes of log files each time.
Enhancements:
- Importing of Web server log files into the database was simplified.
- Multiple log files can now be processed at once.
- A bug in the monthly report was fixed.

Trinacria Linux is a new Italian distribution based on KNOPPIX.

Trinacria Linuxs primary target markets are the Italian office users and Linux beginners.

loggerfs is the virtual file system that allows you to store logs in a database. I just released the first version of it (checkout the News) and will now be focusing on supporting more logging formats. If you dont yet know what loggerfs is, then heres a simple explanation for when you need it:

- Are you running cron scripts periodically to parse log files and put them in a database?
- Do you need to store log files remotely because you have limited space?
- Would you like a more efficient solution than simply polling the log files and waiting for new data?

If you answer yes to any of the above questions, please take a look at loggerfs. It will allow you to create virtual files to which you can then direct syslog/ apache/ squid/ etc. to log to. For example, instead of having syslog store authentication information in /var/log/auth.log, you could create a virtual file in /var/loggerfs/auth.log and then tell syslog to log to that file. Now instead of storing the information in the auth.log file, the data is actually sent to a database server that you defined in the logs.xml file. Every time new information is sent to the file, it is immediately sent to the database server, which means that:

- It doesnt poll for information, it listens, which makes it a lot more efficient.
- It automatically creates the tables in the database and makes sure that all the information can be stored.
- Youll be able to easily sort/ analyze the log files once theyre in the database.

I encourage you to checkout the CVS on the sourceforge project page, and let me know if you have any questions/ problems/ suggestions.

IPTables log analizer displays Linux 2.4 iptables logs (rejected, acepted, masqueraded packets...) in a nice HTML page (it support rough netfilter logs but also Shorewall and Suse Firewall logs).

This page shall be easy to read and understand to reduce the manual analysis time.

This page containts statistics on packets and links to more detailled information on a given host, port, domain and so on.

To convice you, here is a typical syslog entry for iptables :

[IPTABLES DROP] : IN=ppp0 OUT= MAC= SRC=172.186.2.157 DST=193.253.186.217 LEN=36 TOS=0x00 PREC=0x00 TTL=115 ID=4775 PROTO=ICMP TYPE=8 CODE=0 ID=512 SEQ=3663

How does it work ?

A small deamon is launched by a user which can read iptables logs files. Each time a new packet is logged, the daemon insert a new row in the database.

The statistics and so on are elaborated by the PHP page itself.

Log::Dispatch::File::Alerts is a Perl object for logging to alert files.

SYNOPSIS

use Log::Dispatch::File::Alerts;

my $file = Log::Dispatch::File::Alerts->new(
name => file1,
min_level => emerg,
filename => Somefile%d{yyyy!!!!}.log,
mode => append );

$file->log( level => emerg,
message => "Ive fallen and I cant get upn" );

ABSTRACT

This module provides an object for logging to files under the Log::Dispatch::* system.

This module subclasses Log::Dispatch::File for logging to date/time stamped files. See Log::Dispatch::File for instructions on usage. This module differs only on the following three points:

alert files

This module will use a seperate file for every log message.

multitasking-safe

This module uses flock() to lock the file while writing to it.

stamped filenames

This module supports a special tag in the filename that will expand to the current date/time/pid.

It is the same tag Log::Log4perl::Layout::PatternLayout uses, see Log::Log4perl::Layout::PatternLayout, chapter "Fine-tune the date". In short: Include a "%d{...}" in the filename where "..." is a format string according to the SimpleDateFormat in the Java World (http://java.sun.com/j2se/1.3/docs/api/java/text/SimpleDateFormat.html). See also Log::Log4perl::DateFormat for information about further restrictions.
In addition to the format provided by Log::Log4perl::DateFormat this module also supports $ for inserting the PID and ! for inserting a uniq number. Repeat the character to define how many character wide the field should be.
A note on the !: The module first tries to find a fresh filename with this set to 1. If there is already a file with that name then it is increased until either a free filename has been found or it reaches 9999. In the later case the module dies.

Wflogs is a firewall log analysis tool. It can be used to produce a log summary report in plain text, HTML and XML, or to monitor firewalling logs in real-time.
This project is part of the WallFire project, but can be used independently.
Usage examples:
wflogs -i netfilter -o html netfilter.log > logs.html
converts the given netfilter log file into a HTML report.
wflogs --sort=protocol,-time -i netfilter -o text netfilter.log > logs.txt
converts the given netfilter log file into a sorted (by protocol number, then reverse time) text report.
wflogs -f $start_time >= [this 3 days ago] && $start_time < [this 2 days ago] && $chainlabel =~ /(DROP|REJECT)/ && $sipaddr == 10.0.0.0/8 && $protocol == tcp && ($dport == ssh || $dport == telnet) && ($tcpflags & SYN) -i netfilter -o text --summary=no
shows log entries (without summary) which match the given expression (refused connection attempts that occured 3 days ago to ssh and telnet ports coming from internal network 10.0.0.0/8).
wflogs -i netfilter -o text --resolve=0 --whois=0 netfilter.log
converts the given netfilter log file into a text report (default mode), disabling IP address reverse lookups and whois lookups.
wflogs -i netfilter -o xml netfilter.log > logs.xml
exports netfilter logs in XML.
wflogs -i ipchains -o netfilter ipchains.log > netfilter.log
converts ipchains logs into netfilter log format. So you may process them with your favorite netfilter log analyser, for example (even if the latter may not be better than wflogs itself.
wflogs -i ipfilter -o human --datalen=yes ipfilter.log
produces a report about ipfilter logfile in natural language on stdout, displaying packet length (datalen option) which is not showed by default.
wflogs -R -I
monitors logs in real-time in an interactive shell, waiting for logs in the default system logfile, in guessed format (according to the local firewalling tool).
Supported systems
WallFire is intended to work on real systems such as Unix, especially Linux and *BSD.
Current wflogs input modules are:
- netfilter (Linux 2.4 and 2.6 firewall logs)
- ipchains (Linux 2.2 firewall logs)
- ipfilter (NetBSD, FreeBSD, OpenBSD, Solaris, SunOS 4, IRIX and HP-UX running ipfilter firewall logs).
- cisco_pix (Cisco PIX filter logs)
- cisco_ios (Cisco IOS filter logs)
- snort (Snort ACLs logs)
Please note that input modules are available on any architecture on which wflogs can run (for example, you can perfectly parse Cisco PIX logs on a Linux box).
Enhancements:
- Improved matching of netfilter and ipfilter input modules.
- Added support for Cisco FWSM (PIX).
- Improved netfilter parsing.
- Compilation fixes for *BSD.
- Added wflogs.dtd.
- Added wfchkintegrity tool, which enables to monitor changes in the firewalling configuration.
- Fixed buffer sizes for some input modules.
- Fixed parsing with recent flex versions.

LogMon project will monitor one or more log files, updating when more data is available ala tail -f , within a common terminal window via a "split window". User can scroll up/down/left/right through all the windows.
A header displays each frames file name, and number of lines in the file. Very useful when watching several log files at the same time. Ive tested this under Linux and FreeBSD (as of 0.3.3) and I have had reports of it working under MacOS X 10.4.7, but it should work under other Unices. Please let me know if you get it working on another OS.
Enhancements:
- Fixed a scroll-back bug when using page-up
- Added individual frame selection