Free log analysis software for linux

log_analysis is a log file analysis engine that extracts relevant data for any of the recognised log messages and produces a summary that is much easier to read.
Main features:
- Logs contain lots of extraneous stuff that I want to be logged, but that I dont want to sift through when I review logs (ie. routine, error-free daemon operation.)
- Logs contain a lot of repetition, which drowns out the interesting entries.
- Noting repetition can be tricky because each entry usually has extra features to make it unique, such as a date, maybe a PID (ie. for syslog), and maybe application-specific information (ie. sendmail queue IDs.)
- One needs to remember to review them. :)
- One needs to be root to looks at logs for some OSs.
- On most systems, looking at the logs for just one day can be a pain.
- If I attack each box I deal with and write a separate script to do all this, Ill waste a lot of time duplicating effort.
- Writing patterns is a pain even if you know regular expressions.
log_analysis is my solution to these problems. It goes through several different kinds of logs (currently syslog, wtmp, and sulog), over some period (defaults to yesterday). It strips out the date and PID, and throws away certain entries. Then it tries each entry against a list of perl regular expressions. Each perl regular expression is associated with a category name and a rule for extracting data. When theres a match, the data-extracting rule is applied, and filed under the category.
If a log entry is unknown, its filed under a special category for unknowns. Identical entries for a given category are sorted and counted. Theres an option to mail the output, so you can just run it out of cron. You can also save a local copy of the output. If you prefer to PGP-mail yourself the output, you can do this, too. The whole thing is designed to be easily extended, complete with an easy plug-in interface. The default mode is for reporting, but it also "real" and "gui" modes for continuous monitoring, complete with action support. Oh, and you can edit patterns in a GUI that helps write regular expressions quickly and easily.
Security
The program needs to run with permissions to read your log files in order to be useful, which usually means root. It does not default to SUID root, and I recommend not making it SUID, so just run it as root (ie. manually or out of cron). Ive tried to avoid temp files everywhere that I can, and in the one case where I do use a temp file, I make sure to use the POSIX tmpnam function instead of trying to make up my own temp file algorithm. The default umask is 077. If you use action commands, there is nothing to stop you from using parts of the log message in insecure ways, so for goodness sake, be careful.
Local extensions
log_analysis already has lots of rules, but chances are that you have log entries that arent already covered. So, log_analysis can easily be extended via a local config file, as documented in the log_analysis manpage. Theres even an easy way to do modular plug-ins.
Enhancements:
- This release includes a "find" feature in the GUI, various bugfixes, and assorted minor features.

Sequence Analysis project is a collage of coding projects which I have written over the past several years for various clients in my work as a bioinformatics consultant.

These clients have graciously allowed me to release these works into the public domain as freeware for Macintosh OS X in order to promote the platform and to encourage migration from Classic.

The upper window panel can hold several sequences, which are both editable and selectable. The tabs in the lower analysis panel try to keep up with the current sequence selection to provide immediate feedback. The selection is used in some modules as only the portion being analyzed for other modules i.e. Digest is used to determine if enzymes cut in the in or outside of the selection.

Most commonly available sequence formats have been reverse engineered. You can also access a sequences from the NCBI via its GID or UID. This currently cannot be done from behind a firewall.

Most of the analyses are simple enough that they are obvious to use, Composition, pI. Others could stand some documenation i.e. Pairwise and Primer Design. The Publish tab uses a string to control the layout. Click on the Legend button for some help.

IPTables log analizer displays Linux 2.4 iptables logs (rejected, acepted, masqueraded packets...) in a nice HTML page (it support rough netfilter logs but also Shorewall and Suse Firewall logs).

This page shall be easy to read and understand to reduce the manual analysis time.

This page containts statistics on packets and links to more detailled information on a given host, port, domain and so on.

To convice you, here is a typical syslog entry for iptables :

[IPTABLES DROP] : IN=ppp0 OUT= MAC= SRC=172.186.2.157 DST=193.253.186.217 LEN=36 TOS=0x00 PREC=0x00 TTL=115 ID=4775 PROTO=ICMP TYPE=8 CODE=0 ID=512 SEQ=3663

How does it work ?

A small deamon is launched by a user which can read iptables logs files. Each time a new packet is logged, the daemon insert a new row in the database.

The statistics and so on are elaborated by the PHP page itself.

Visitors is a very fast web log analyzer for Linux, Windows, and other Unix-like operating systems. It takes as input a web server log file, and outputs statistics in form of different reports. The design principles are very different compared to other software of the same type:
No installation required, can process up to 150,000 lines of log entries per second in fast computers (20MB/s with my log files average length).
Designed to be executed by the command line, output html and text reports. The text report can be used in pipe to less to check web stats from ssh.
Support for real time statistics with the Visitors Stream Mode introduced with version 0.3.
To specify the log format is not needed at all. Works out of box with apache and most other web servers with a standard log format (see the documentation for more information on the format).
Its a portable C program, can be compiled on many different systems. Binaries for Windows systems are in the Download section of this page.
The produced html report doesnt contain images or external CSS, is self-contained, you can send it by email to users.
Visitors is free software (and of course, freeware), under the terms of the GPL license. You dont need to pay to use it. Visitors is supported, if you want a custom version made directly by the original author for a modest price, contact me at antirez (at) invece.org. ISPs may take advantage of the high processing speed.
Main features:
- Requested pages.
- Requested images.
- Referers by hits and age.
- Unique visitors in each day.
- Page views per visit.
- Pages accessed by the Google crawler (and the date of googles last access on every page).
- Percentage of visits originated from Google searches for every day.
- Users navigation patterns (web trails).
- Keyphrases used in Google searches.
- User agents.
- Weekdays and Hours distributions of accesses.
- Weekdays/Hours combined bidimentional map.
- Month/Year combined bidimentional map.
- Visual path analysis with Graphviz.
- Operating systems, browsers and domains popularity.
- 404 errors.
Enhancements:
- This release adds an important bugfix in the unique visitors algorithm.
- The output is now nearer to reality (though unique visitors stats are always a guess without the use of a cookie).

Log Mine is a tool that produces reports on usage patterns on your Web site.
Web server log files are not just hit counters. They contain valuable information about the usage patterns of your website. Unforunately many web log analysis tools lay emphasis on telling you how many hits your site had or how many pages were seen and how many bytes were transferred.
A more usefull statistic would be which percentage of users came to your site went to a product information page, and which percentage of those users hit the checkout button, and which percentage actually completed their order. The trouble is the very nature of the web makes it nearly impossible to get accurate figures for such statistics.
However over periods of time, the errors present average out and it is possible to get a good indication of these ratios by properly mining the log file. That brings us back to square one, how do we get this information with traditional log analysers?
Traditional log analysers will produce weekly, monthly or daily charts for the usage of your site, but rarely do they allow you to create such charts for individual pages or referrrs - something very usefull if you run advertising campaigns on other sites.
Enter Log Mine. This new web log analyser / Mining tool will allow you to create just about any kind of report from the contents of your log file. Log Mine is not concerned about speed and it will be very greedy when it comes to taking up space on your hard disk/database but it will let you change your reporting without having to process gigabytes of log files each time.
Enhancements:
- Importing of Web server log files into the database was simplified.
- Multiple log files can now be processed at once.
- A bug in the monthly report was fixed.

Log::Localized is a Perl module to localize your logging.

SYNOPSIS

What you most probably want to do is something like:
package Foo;
use Log::Localized;

sub bar {
# this message will be displayed if method bars verbosity is >= 1
llog(1,"running bar()");
}

# this message will be displayed if package Foos verbosity is >= 3
llog(3,"loaded package Foo");
Then paste the following local verbosity rules in a file called verbosity.conf, in the same directory as your program:
# log everything from wherever inside Foo and its subclasses, up to level 3
Foo:: = 3
# except for function Foo::foo who shall have verbosity 0
Foo::bar = 0

SYNOPSIS - ADVANCED

In a program accepting command line arguments, you may want to do:
use Getopt::Long;
use Log::Localized log => 1;

GetOptions("verbose|v+" => sub { $Log::Localized::VERBOSITY++; } );

llog(1,"you used -v");
llog(2,"you used -v -v");
You may alter local verbosity from within the running code:
package Foo;
use Log::Localized log => 1;

# verbosity level is 0 by default

{
# set verbosity locally in this block
local $Log::Debug::VERBOSITY = 5;
llog(5,"this will be logged");
}

debug(5,"but this wont");
If you want to import llog under another name in the calling module:
package Foo;
use Log::Localized rename => "my_log";

# call Log::Localized::llog()
my_log(1,"renamed llog()");
See the examples directory in the module distribution for more real life examples.

Log::Localized provides you with an interface for defining dynamically exactly which part of your code should log messages and with which verbosity.

Log::Localized addresses one issue of traditional logging: in very large systems, a slight increase in logging verbosity usually generates insane amounts of logs. Hence the need of being able to turn on verbosity selectively in some areas of code only, in a localized way.

Log::Localized is based on the concept of local verbosity. Each package and each function in a package has its own local verbosity, set to 0 by default. With Log::Localized you can change the local verbosity in just a function, just a package or just a class hierarchy via a so called verbosity rule. Verbosity rules are passed to Log::Localized either via a configuration file or via an import parameter. By changing verbosity rules according to the needs of the moment, you can alter your programs logging flow in a very fine-grained way, and get logs from only the code areas you are interested in.

Log::Localized comes with default settings that make it usable out of the box, but its configuration options will let you redefine pretty much everything in its behavior.
The actual logging in Log::Localized is handled by Log::Dispatch.

BASE is the Basic Analysis and Security Engine. It is based on the code from the Analysis Console for Intrusion Databases (ACID) project.
This application provides a web front-end to query and analyze the alerts coming from a SNORT IDS system.
BASE is a web interface to perform analysis of intrusions that snort has detected on your network. It uses a user authentication and role-base system, so that you as the security admin can decide what and how much information each user can see. It also has a simple to use, web-based setup program for people not comfortable with editing files directly.
BASE is supported by a group of volunteers. They are available to answer any questions you may have or help you out in setting up your system. They are also skilled in intrusion detection systems and make use of that knowledge in the development of BASE.
Enhancements:
- This release fixes a number of bugs with PHP 5.
- It also adds a number of new features.

Plucene::Analysis::PorterStemFilter - Porter stemming on the token stream.

SYNOPSIS

# isa Plucene::Analysis:::TokenFilter

my $token = $porter_stem_filter->next;

This class transforms the token stream as per the Porter stemming algorithm.
Note: the input to the stemming filter must already be in lower case, so you will need to use LowerCaseFilter or LowerCaseTokenizer farther down the Tokenizer chain in order for this to work properly!

The Porter Stemmer implements Porter Algorithm for normalization of English words by stripping their extensions and is used to generalize the searches. For example, the Porter algorithm maps both search and searching (as well as searchnessing) to search such that a query for search will also match documents that contains the word searching.

Note that the Porter algorithm is specific to the English language and may give unpredictable results for other languages. Also, make sure to use the same analyzer during the indexing and the searching.

You can find more information on the Porter algorithm at www.tartarus.org/~martin/PorterStemmer.

A nice online demonstration of the Porter algorithm is available at www.scs.carleton.ca/~dquesnel/java/stuff/PorterApplet.html.

METHODS

next

my $token = $porter_stem_filter->next;

Returns the next input token, after being stemmed.