Free logfiles software for linux

check_logfiles is a plugin for Nagios which checks logfiles for defined patterns. check_logfiles project is capable of detecting logfile rotation.

If you tell it how the rotated archives look, it will also examine these files. Unlike check_logfiles, traditional logfile plugins were not aware of the gap which could occur, so under some circumstances they ignored what had happened between their checks.

A configuration file is used to specify where to search, what to search, and what to do if a matching line is found.

Installation:

1. `cd to the directory containing the packages source code and type `./configure to configure the package for your system. If youre using `csh on an old version of System V, you might need to type `sh ./configure instead to prevent `csh from trying to execute `configure itself.

Running `configure takes awhile. While running, it prints some messages telling which features it is checking for.

2. Type `make to compile the package.

3. Optionally, type `make check to run any self-tests that come with the package.

4. Type `make install to install the programs and any data files and documentation.

5. You can remove the program binaries and object files from the source code directory by typing `make clean. To also remove the files that `configure created (so you can compile the package for a different kind of computer), type `make distclean. There is also a `make maintainer-clean target, but that is intended mainly for the packages developers. If you use it, you may have to get all sorts of other programs in order to regenerate files that came with the distribution.

Logrep is a secure multi-platform framework for the collection, extraction, and presentation of information from various log files.
It features HTML reports, multi dimensional analysis, overview pages, SSH communication, and graphs, and supports over 30 popular systems including Snort, Squid, Postfix, Apache, Sendmail, syslog, ipchains, NT event logs, Firewall-1, wtmp, iptables, xferlog, Oracle listener and Pix.
Main features:
- Supports multiple platforms and logfile formats
- Keeps compressed copies of logfiles on a central location
- Secure SSH communication between client and server
- Capable of multi dimensional analysis
- Quick access to favorite graphs, reports and all nodes
- Binary installation packages
- Client setup with a very small footprint
- Open source and highly customizable !
Enhancements:
- A totally re-engineered CGI-based user interface was introduced.
- Support for log files from Checkpoint Firewall-1 NGX and GTA Firewall appliances was added.

LoFiMo is used to monitor logfiles in real time. The output is presented via a web interface and optionally on the console.
Using the web interface it is possible to monitor log files from a remote machine. LoFiMo can be used to colorize the log entries using filters.
Filters can also be used to reformat log entries, hide log entries or play sounds or execute commands when certain log entries are read.
LoFiMo uses inputs to read log entries from different file formats. It can for instance parse files created by the syslog daemon or the apache web server.
Main features:
- Support for apache style log files
- Support for syslog style log files
- Can process iptables log and reformat it. Can detect most ICMP types and convert them to cleat text.
- For example
- Nov 23 16:53:32 esme kernel: FW_ACCEPT: IN=eth1 OUT=eth0 SRC=x.x.x.x DST=x.x.x.x LEN=84 TOS=0x00 PREC=0x00 TTL=53 ID=12419 PROTO=ICMP TYPE=0 CODE=0 ID=7220 SEQ=7
- can be transformed into:
- Nov 23 16:53:32 esme kernel: FW_ACCEPT: ICMP / eth1->eth0 / x.x.x.x->x.x.x.x / len=84 / tos=0x00 / Echo Reply / ttl=53
- Can process postfix mail entries and output them as single line per mail.
- For example these seven lines
- Nov 23 17:56:01 magrat postfix/smtpd[13644]: connect from mx02.domain.com[x.x.x.x]
- Nov 23 17:56:01 magrat postfix/smtpd[13644]: A297015B603: client=mx02.domain.com[x.x.x.x]
- Nov 23 17:56:01 magrat postfix/cleanup[13648]: A297015B603: message-id=< 002d01c5f04e$c44f16c0$0100a8c0@eva >
- Nov 23 17:56:02 magrat postfix/qmgr[2764]: A297015B603: from=< send@domain.com >, size=170113, nrcpt=1 (queue active)
- Nov 23 17:56:10 magrat postfix/smtp[13649]: A297015B603: to=< rec@domain.com >, relay=localhost[127.0.0.1], delay=9, status=sent (250 2.6.0 Ok, id=12734-02, from MTA: 250 Ok: queued as 1508615B633)
- Nov 23 17:56:10 magrat postfix/qmgr[2764]: A297015B603: removed
- Nov 23 17:56:32 magrat postfix/smtpd[13644]: disconnect from mx02.domain.com[x.x.x.x]
- can be transformed into a single line:
- 2005-11-23 17:56:10 magrat postfix send@domain.com -> rec@domain.com / mx02.domain.com[x.x.x.x] -> localhost[127.0.0.1] / 9 sec / 170113 bytes / sent (250 2.6.0 Ok, id=12734-02, from MTA: 250 Ok: queued as 1508615B633)
- Can remove unwanted log entries from the output.
- Can set the style for each field of a log entry using css allowing multiple fonts and colours in the same line.
- Can play a sound whenever a log entry matching certain criteria is read (e.g. when mail from a certain sender arrived).
- Can execute a command whenever a log entry matching certain criteria is read (e.g. to send an email or play sound using a custom sound player).
- Filters use regular expressions to match certain fields or the entire log entries.
- Can process log files of any format.
- Access via web browser from remote machines. View the log entries as they are read in real time or browse and refresh the output in the interval you configure.
Enhancements:
- The character set of monitored files can now be specified.
- Filenames in the execute property of filters can now contain blank characters.

Analog is an application that will help you measure the usage on your web server. It tells you which pages are most popular, which countries people are visiting from, which sites they tried to follow broken links from, and all sorts of other useful information.
Main features:
- is very fast
- can handle big logfiles
- output in any of 33 different languages
- very configurable
- easy to install

Rxlogd is a receive-only syslog server (collector) that can coexist with sysklogd. The project features simplicity, ease of use and a built-in dns cache for high performance.
An example logrotate configuration file is provided.
The intended audience is users of enterprise systems stuck with sysklogd or administrators who dont need more complex syslog server solutions.
Written in Python, rxlogd is architecture-independent. It was tested on i386 and x86_64.
No extra libraries are required.
Usage:
- Install the RPM or the sources from the download page on the machine intended to store syslog for others.
- Red Hat users can simply install the RPM and enjoy the defaults: logs will be stored in /var/log/remote (which will be created for you) and the daemon will run as user nobody. Also, logrotate will rotate 5 copies of your logfiles in that directory.
- Optionally, edit /etc/sysconfig/rxlogd for username to run as (examples are provided). Edit /etc/logrotate.d/rxlogd to change log rotation/retention parameters and to update changes made to the config file.
- Refer to the syslog documentation on how to configure the clients (the machines which will send syslog to your server).
Enhancements:
- Logrotate now uses a separate directory to store old logs in, fixing a bug.
- Minor cosmetic changes to init script.

httplog is a replacement for Apaches rotatelogs and Andrew Fords chronolog. It allows you to specify a logfile using strftime paramaters in the filename to act as a template.
This means that the logs in your logfiles will also be sorted according to the filename. For example, if you specify a logfile of /var/log/http%Y%m%d.log, a new log file would be generated each day, with content for only that one day.
It also supports compression of logfiles using gzip, and many other useful functions.
It takes one argument at the command line, the path to a logfile as a template.
The template pathname can use any valid special characters permitted in strftime(3). Other optional arguments can be appended to the command line.
If you want to have httplog run as a different user and/or group (instead of running as root - possible security risks), you can use the `-u and `-g to have httplog run as that user and/or group.
You can also specify `-b to have httplog use a custom buffer size for memory storage of log data, rather than the default value of just line buffering mode (flushes data per newline).
If you want to have your logfiles gzipped because of size constraints, you can now use the `-z option to have httplog gzip all logfiles once its done writing all data to them (when a rollover period hits and a new logfile is to be created).
You may have a symlink to the currently active logfile by using the `-s option and specifying a filename to be used as the symlink.
This program makes Apache act similar to Microsofts IIS webserver in its logging style (as opposed to every logfile entry in one single huge logfile). This allows you to easily maintain your webserver logfiles for statistics packages in an easily organizable manner without user intervention.
Because of the buffer option added since v1.5, httplog may also be usefull for extremely large sites that get thousands of hits per second, and typically disable logging to save their hard drives from dying.
The buffer size option can be used to create an extremely large buffer in ram so that logfiles would not be flushed as often as if it were in line buffering mode, and in essence, could prolong the life of your valuable drives by decreasing their activity.
Enhancements:
- Removed a couple of lines of code that could cause segfaults if errors were imminent while opening new a file or allocating the buffer memory

ETTERLOG is a utility for Network Lan (SWITCH) for file .ECI and .ECP.
(Ettercap program).

Etterlog is the log analyzer for logfiles created by ettercap. It can handle both compressed (created with -Lc) or uncompressed logfiles. With this tool you can manipulate binary files as you like and you can print data in different ways all the times you want (in contrast with the previous logging system which was used to dump in a single static manner).

You will be able to dump traffic from only one connection of your choice, from only one or more hosts, print data in hex, ascii, binary etc... TIP: All unuseful messages are printed to stderr, so you can save the output from etterlog with the following command:

etterlog [options] logfile > outfile Thus you can dump for example a binary file from an ftp connection if you print the data in binary mode, without headers and selecting only the ftp server as the source of the communication.

GENERAL OPTIONS

-a, --analyze Analyze a log file and display some interesting statistics.

-c, --connections Parse the log file and print a table of unique connections (port to port). This option can be used only on LOG_PACKET logfiles. On LOG_INFO logfiles it is useless. TIP: you can search for a particular host by using the following command: etterlog -c logfile.ecp | grep 10.0.0.1

-f, --filter < TARGET > Print only packets coming from or going to TARGET. The TARGET specification is the same as in ettercap.
TARGET is in the form MAC/IPs/PORTs. Omitting one or more of its parts will be equivalent to set them to ANY. If the log type is LOG_INFO the target is used to display hosts matching the mac, ip and having the specified port(s) open. For example the target //80 will display only information about hosts with a running web server.

-r, --reverse Reverse the matching in the TARGET selection. It means not(TARGET). All but the selected TARGET.

-t, --proto < PROTO > Sniff only PROTO packets (default is TCP + UDP). This option is only useful in "simple" mode. If you start ettercap in interactive mode both TCP and UDP are sniffed.
PROTO can be "tcp", "udp" or "all" for both.

-F, --filcon < CONNECTION > Print packets belonging only to this CONNECTION.
CONNECTION is in the form PROTO:SOURCE:DEST. SOURCE and DEST are in the form IP:PORT. example: etterlog -F TCP:10.0.0.23:3318:198.182.196.56:80

-s, --only-source Display only packets that are sent by the source of the selected CONNECTION. This option makes sense only in conjunction with the -F option. TIP: if you want to save a file transferred in an HTTP or FTP connection, you can use the following command: etterlog -B -s -n -F TCP:10.0.0.1:20:10.0.0.2:35426 logfile.ecp > example.tar.gz

-d, --only-dest Same as --only-source but it filters on the destination host.

-n, --no-headers Do not print the header of each packet. This option is useful if you want to save a file in binary format (-B option). Without the headers you can redirect the output to a file and you will get the original stream. NOTE: the time stamp in the header is in the form: Thu Mar 27 23:03:31 2003 [169396], the value in the square brackets is expressed in microseconds

-m, --show-mac In the headers show also the mac addresses corresponding to the ip addresses.

-k, --color If used in conjunction with -F it displays the source and dest of the connection using different colors. If used with a LOG_INFO file it prints LAN hosts in green, REMOTE hosts in blue and GATEWAYS in red.

-l, --only-local Used displaying an INFO file, it displays information only about local hosts.

-L, --only-remote Used displaying an INFO file, it displays information only about remote hosts.

SEARCH OPTIONS

-e, --regex < REGEX > Display only packets matching the regex < REGEX >.
If this option is used agains a LOG_PACKET logfile, the regex is executed on the payload of the packet. If the type is LOG_INFO, the regex is executed on all the fields of the host profile (OS, banners, service and ethernet adapter).
NOTE: the regex is compiled with the REG_ICASE flag (case insensitive).

-u, --user < USER > Display information about this user. The search is performed over all the user/pass couples collected across all hosts.

-p, --passwords Print only the collected account information for each host. This prevents the huge profile output. It can be used in conjunction with the -u option to filter the users. An asterisk * used in front of an account represents a failed login attempt.

-i, --show-client Show the client ip address when displaying the collected users and passwords. It may be useful when ACLs are in place.

-I, --client < IP > Show passwords only coming from a specific < IP >. This is useful to view all the usernames and passwords of a client.


EDITING OPTIONS

-C, --concat Use this option to concatenate two (or more) files into one single file. This is useful if you have collected ettercap log files from multiple sources and want to have an unified report. The output file must be specified with the -o option and the input files are listed as normal arguments. example:
etterlog -C -o outfile input1 input2 input3

-o, --outfile < FILE > specifies the output file for a concatenation.


VISUALIZATION METHOD


-B, --binary Print data as they are, in binary form. Useful to dump binary data to a file (as described above).

-X, --hex Print the packets in hex format. example: the string "HTTP/1.1 304 Not Modified" becomes: 0000: 4854 5450 2f31 2e31 2033 3034 204e 6f74 HTTP/1.1 304 Not
0010: 204d 6f64 6966 6965 64 Modified

-A, --ascii Print only "printable" characters, the others are displayed as dots .

-T, --text Print only the "printable" characters and skip the others.

-E, --ebcdic Convert an EBCDIC text to ASCII.

-H, --html Strip all html tags from the text. A tag is every string between < and >. example: < title >This is the title< /title >, but the following < string > will not be displayed. This is the title, but the following will not be displayed.

-U, --utf8 < encoding > Print the packets in UTF-8 format. The < encoding > parameter specifies the encoding to be used while performing the conversion. Use the `iconv --list` command to obtain a list of all supported encodings.

-Z, --zero Print always the void string. i.e. print only header information and no packet content will be printed.

-x, --xml Print the host information in xml form, so you can parse it with your favourite program. The DTD associated with the xml output is in share/etterlog.dtd

STANDARD OPTIONS

-v, --version Print the version and exit.

-h, --help Print the help screen with a short summary of the available options.


EXAMPLES

Here are some examples of using etterlog.

etterlog -k -l dump.eci
Displays information about local hosts in different colors.

etterlog -X dump.ecp
Prints packets in HEX mode with full headers.

etterlog -c dump.ecp
Displays the list of connections logged in the file.

etterlog -Akn -F TCP:10.0.0.1:13423:213.203.143.52:6666 dump.ecp
Displays the IRC traffic made by 10.0.0.1 in ASCII mode, without headers information and in colored mode.

etterlog -H -t tcp -f //80 dump.ecp
Dumps all HTTP traffic and strips html tags.

etterlog -Z -r -f /10.0.0.2/22 dump.ecp
Displays only the headers of all connections except ssh on host 10.0.0.2

etterlog -A -e user -f //110 dump.ecp
Displays only POP packets containing the user regexp (case insensitive).

etterlog -u root dump.eci
Displays information about all the accounts of the user root.

etterlog -e Apache dump.eci
Displays information about all the hosts running Apache.

etterlog -e Linux dump.eci
Displays information about all the hosts with the Linux operating system.

etterlog -t tcp -f //110 dump.eci
Displays information about all the hosts with the tcp port 110 open.

etterlog -t udp dump.eci
Displays information about all the hosts with at least one UDP port open.

etterlog -B -s -n -F TCP:10.0.0.1:20:10.0.0.2:35426 logfile.ecp > example.tar.gz
Dumps in binary form the data sent by 10.0.0.1 over the data port of FTP. Since the headers are omitted, you will get the file as it was.

NewSyslog is a highly configurable program for managing and archiving log files.
Main features:
- It is more portable (using GNU Autoconf) and it can be compiled and installed on most any modern Unix or Unix-like system.
- It has support for fixed time-of-day daily archiving with a command-line option to identify the daily roll-over invocation (which may be at midnight, or at any other regular daily time).
- It supports the FreeBSD feature that allows specification of the log roll-over time as a daily, weekly, or monthly interval (with optional time-of-day specification for the last two). [The other overly flexible, ISO 8601 interpretation of the interval "@" option is not supported -- it is too generic and not meaningful enough in the context of log file management.]
- It supports optional PID files so that non-standard daemons can be told to re-open their logfiles after archiving has taken place. (Including /dev/null which disables signalling of any daemon when the specified log file is rolled over.)
- It can send a signal other than SIGHUP to the daemon associated with a given log file.
- It can leave the most recently archived log file uncompressed, which is necessary for daemons like httpd and smail because they continue to write to the current log file until their current jobs have completed. (This also makes it much easier to review recent log data with normal Unix tools.) [NetBSD now has this feature.]
- It supports the FreeBSD feature of being able to restrict processing to just those log files specified on the command line.
- Unlike the NetBSD version it first parses the config file before taking any action, meaning that if any errors are encountered it will report them and quit without doing anything.
- Unlike the FreeBSD version, it will roll a log file if either the interval or size limits have been reached (FreeBSDs version makes it too easy to have a rapidly growing log file overflow the filesystem).
- Unlike the NetBSD version it always creates any missing log file (though this can be disabled on a per-file basis).
- It uses an advisory lock on the current configuration file to prevent multiple invocations from tripping over each other.
- The documentation is far better!