Free log file analysis program software for linux

log_analysis is a log file analysis engine that extracts relevant data for any of the recognised log messages and produces a summary that is much easier to read.
Main features:
- Logs contain lots of extraneous stuff that I want to be logged, but that I dont want to sift through when I review logs (ie. routine, error-free daemon operation.)
- Logs contain a lot of repetition, which drowns out the interesting entries.
- Noting repetition can be tricky because each entry usually has extra features to make it unique, such as a date, maybe a PID (ie. for syslog), and maybe application-specific information (ie. sendmail queue IDs.)
- One needs to remember to review them. :)
- One needs to be root to looks at logs for some OSs.
- On most systems, looking at the logs for just one day can be a pain.
- If I attack each box I deal with and write a separate script to do all this, Ill waste a lot of time duplicating effort.
- Writing patterns is a pain even if you know regular expressions.
log_analysis is my solution to these problems. It goes through several different kinds of logs (currently syslog, wtmp, and sulog), over some period (defaults to yesterday). It strips out the date and PID, and throws away certain entries. Then it tries each entry against a list of perl regular expressions. Each perl regular expression is associated with a category name and a rule for extracting data. When theres a match, the data-extracting rule is applied, and filed under the category.
If a log entry is unknown, its filed under a special category for unknowns. Identical entries for a given category are sorted and counted. Theres an option to mail the output, so you can just run it out of cron. You can also save a local copy of the output. If you prefer to PGP-mail yourself the output, you can do this, too. The whole thing is designed to be easily extended, complete with an easy plug-in interface. The default mode is for reporting, but it also "real" and "gui" modes for continuous monitoring, complete with action support. Oh, and you can edit patterns in a GUI that helps write regular expressions quickly and easily.
Security
The program needs to run with permissions to read your log files in order to be useful, which usually means root. It does not default to SUID root, and I recommend not making it SUID, so just run it as root (ie. manually or out of cron). Ive tried to avoid temp files everywhere that I can, and in the one case where I do use a temp file, I make sure to use the POSIX tmpnam function instead of trying to make up my own temp file algorithm. The default umask is 077. If you use action commands, there is nothing to stop you from using parts of the log message in insecure ways, so for goodness sake, be careful.
Local extensions
log_analysis already has lots of rules, but chances are that you have log entries that arent already covered. So, log_analysis can easily be extended via a local config file, as documented in the log_analysis manpage. Theres even an easy way to do modular plug-ins.
Enhancements:
- This release includes a "find" feature in the GUI, various bugfixes, and assorted minor features.

Net::Analysis are modules for analysing network traffic.

SYNOPSIS

Using an existing analyser:

$ perl -MNet::Analysis -e main help
$ perl -MNet::Analysis -e main TCP,v=1 dump.tcp - basic TCP info
$ perl -MNet::Analysis -e main HTTP,v=1 dump.tcp - HTTP stuff
$ perl -MNet::Analysis -e main Example2,regex=img dump.tcp - run an example

Writing your own analyser:

package MyExample;
use base qw(Net::Analysis::Listener::Base);
# Listen to events from other modules
sub tcp_monologue {
my ($self, $args) = @_;
my ($mono) = $args->{monologue};
my $t = $mono->t_elapsed()->as_number();
my $l = $mono->length();
# Emit your own event
$self->emit(name => example_event,
args => { kb_sec => ($t) ? $l/($t*1024) : N/A }
);
}
# Process your own event
sub example_event {
my ($self, $args) = @_;
printf "Bandwidth: %10.2f KB/secn", $args->{kb_sec};
}
1;
__top

ABSTRACT

Net::Analysis is a suite of modules that parse tcpdump files, reconstruct TCP sessions from the packets, and provide a very lightweight framework for writing protocol anaylsers.
__top

I wanted a batch version of Ethereal in Perl, so I could:

- sift through parsed protocols with structured filters
- write custom reports that mixed events from multiple protocols

So here it is. Net::Analysis is a stack of protocol handlers that emit, and listen for, events.

TA-Lib provides common functions for the technical analysis of stock/future/commodity market data.
TA-Lib can be reused by trading software developers using Excel, .NET, Java, Perl or C/C++.
Main features:
- More than 120 technical analysis indicators such as ADX, MACD, RSI, Stochastic, Bollinger Bands etc...
- bullet Includes candlestick pattern recognition.
- bullet Optional abstract interface allowing your code to support new technical analysis functions without any code change!
Enhancements:
New Features
- New Functions: BETA, MINMAX, MINMAXINDEX, MININDEX, MAXINDEX
- Debian and RPM packaging available.
- Java JAR packaging available.
- New TA_FunctionDescription() returns XML description of API.
- New ta_func_api.xml file generated in root directory of the package.
- Support for unmanaged static libraries with Visual Studio 2005.
Fixes
- #1526632 : Fix bug in LINEARREG_ANGLE
- #1544555 : Now do proper divide by zero detection in TA_ADX
Other Changes
- Better Java/.NET naming convention.
- ta_func_list.txt moved in root directory of the package.
- Removed dependencies on trio and Mersenne Twister functions.
- Volume and Open Interest are now double instead of integers.
- Add license specific to Excel users.

Market Analysis System (MAS) is an open-source software application that provides tools for analysis of financial markets using technical analysis.
Market Analysis System provides facilities for stock charting and futures charting, including price, volume, and a wide range of technical analysis indicators. Market Analysis System also allows automated processing of market data - applying technical analysis indicators with user-selected criteria to market data to automatically generate trading signals - and can be used as the main component of a sophisticated trading system.
Main features:
- Includes basic technical analysis indicators, such as Simple Moving Average, Exponential Moving Average, Stochastic, MACD, RSI, On Balance Volume, and Momentum.
- Includes more advanced indicators, such as Standard Deviation, Slope of EMA of Volume, Slope of MACD Signal Line, Bollinger Bands, and Parabolic SAR.
- User can create new technical analysis indicators, including complex indicators based on existing indicators.
- User can configure criteria for automated trading-signal generation.
- Creation of weekly, monthly, quarterly, and yearly data from daily data.
- Handles intraday data.
- Handles stock and futures data.
- Accepts input data from files, from a database, or from the web. (Includes a configuration for obtaining end-of-day data from yahoo.com.)
- Can be configured and run as a server that provides services for several clients at a time running on remote machines.

Logfilter is a tool for performing ad hoc analysis on Apache log files.
Logfilter is especially useful when you wish to find records matching very specific criteria, since you can specify exactly which fields to match.
The tool also plays well in a Unix environment by accepting input either through files or via stdin.
Main features:
- Match only the records you are interested in, ignore everything else
- Fast matching...the code is written in C for maximum portability & performance
- Released under the Apache Licence so you can customise & extend as you wish
Enhancements:
- The command line options were changed to be more in line with Unix philosophy.
- An autoconf script was added to make building logfilter easier.
- A bug in the case sensitivity matching was fixed.

Network Security Analysis Tool is a fast, stable bulk security scanner designed to audit remote network services and check for versions, security problems, gather information about the servers and the machine, and much more.

A manpage providing extensive information on NSAT has been included in the distribution. It is available after a make install, or just by typing man doc/nsat.8 from this dir. It is suggested that you inform yourself at least about the -v (scan verbosity) option and edit the configuration file. To learn about changes in this version, please consult doc/CHANGES.

New to this version is support for distributed scanning. The manpage describes how to do a distributed scan. Note that distributed scanning in this version is just a preliminary, proof-of-concept, implementation with no guarantees for its security, reliability, or performance.

Check for updated vulnerability lists, config files, etc. from
http://nsat.sourceforge.net

Currently, these are lists of vulnerabilities:

nsat.cgi (CGI scripts)
nsat.conf (configuration)
src/mod/snmp.h (SNMP community names)

Log::Dispatch::File::Alerts is a Perl object for logging to alert files.

SYNOPSIS

use Log::Dispatch::File::Alerts;

my $file = Log::Dispatch::File::Alerts->new(
name => file1,
min_level => emerg,
filename => Somefile%d{yyyy!!!!}.log,
mode => append );

$file->log( level => emerg,
message => "Ive fallen and I cant get upn" );

ABSTRACT

This module provides an object for logging to files under the Log::Dispatch::* system.

This module subclasses Log::Dispatch::File for logging to date/time stamped files. See Log::Dispatch::File for instructions on usage. This module differs only on the following three points:

alert files

This module will use a seperate file for every log message.

multitasking-safe

This module uses flock() to lock the file while writing to it.

stamped filenames

This module supports a special tag in the filename that will expand to the current date/time/pid.

It is the same tag Log::Log4perl::Layout::PatternLayout uses, see Log::Log4perl::Layout::PatternLayout, chapter "Fine-tune the date". In short: Include a "%d{...}" in the filename where "..." is a format string according to the SimpleDateFormat in the Java World (http://java.sun.com/j2se/1.3/docs/api/java/text/SimpleDateFormat.html). See also Log::Log4perl::DateFormat for information about further restrictions.
In addition to the format provided by Log::Log4perl::DateFormat this module also supports $ for inserting the PID and ! for inserting a uniq number. Repeat the character to define how many character wide the field should be.
A note on the !: The module first tries to find a fresh filename with this set to 1. If there is already a file with that name then it is increased until either a free filename has been found or it reaches 9999. In the later case the module dies.

Log Mine is a tool that produces reports on usage patterns on your Web site.
Web server log files are not just hit counters. They contain valuable information about the usage patterns of your website. Unforunately many web log analysis tools lay emphasis on telling you how many hits your site had or how many pages were seen and how many bytes were transferred.
A more usefull statistic would be which percentage of users came to your site went to a product information page, and which percentage of those users hit the checkout button, and which percentage actually completed their order. The trouble is the very nature of the web makes it nearly impossible to get accurate figures for such statistics.
However over periods of time, the errors present average out and it is possible to get a good indication of these ratios by properly mining the log file. That brings us back to square one, how do we get this information with traditional log analysers?
Traditional log analysers will produce weekly, monthly or daily charts for the usage of your site, but rarely do they allow you to create such charts for individual pages or referrrs - something very usefull if you run advertising campaigns on other sites.
Enter Log Mine. This new web log analyser / Mining tool will allow you to create just about any kind of report from the contents of your log file. Log Mine is not concerned about speed and it will be very greedy when it comes to taking up space on your hard disk/database but it will let you change your reporting without having to process gigabytes of log files each time.
Enhancements:
- Importing of Web server log files into the database was simplified.
- Multiple log files can now be processed at once.
- A bug in the monthly report was fixed.