Free diagnostics for all software for linux

mod_diagnostics is a debugging and diagnostic tool for application developers - particularly filter modules.

mod_diagnostics.c can be inserted anywhere in the Apache filter chain, and logs traffic (buckets and brigades) passing through.

It is a purely passive watcher, and will never touch the traffic passing through.

Examples

Probably the best way to explain mod_diagnistics is by example.

Strange delays in some browsers

In an update to mod_xml, a new bug was introduced. It was not immediately obvious, but in some browsers the request would hang and then timeout. The effect was only observed when using the XSLT output filter with Xalan-C, and only happened with HTTP/1.1 browser, not with HTTP/1.0. Furthermore, hitting "cancel" before the timeout in an HTTP/1.1 browser would cause the page to display!
Inserting mod_diagnostics before and after the offending filter, the bug was immediately obvious. The module was simply failing to pass an EOS bucket down the chain. A trivial fix!

Obscure bug in a third-party library

A user of mod_proxy_html reported serious performance problems when parsing an 8Mb HTML file. He had profiled the problem, and found the entire processing time was in the final call to htmlParseChunk in libxml2.

I investigated this by inserting mod_diagnostics before and after mod_proxy_html, and running it with the largest HTML document I had available (the MySQL manual, about 2.6Mb). I was able to confirm that nothing was passed down the chain until the final call, so not only was it slow, but it had also broken Apache pipelining.

To refine the diagnosis, I added a flush in each call to the filter in mod_proxy_html. Now mod_diagnostics showed a small amount of data (under 1Kb) coming through in the first call to the filter, but nothing else until the end. Further investigation showed that the data stopped coming when the first HTML comment was encountered in the source.

At this point I ran it under gdb, looking for the comment handling. I found that it was failing to find the end of the comment. The problem was resolved only in the last call to htmlParseChunk, which didnt go through the buggy code. I disabled the buggy code, and found it was now working correctly, with approximately the same amount of input and output data in each call to the mod_proxy_html filter - so pipelining was now fixed. My correspondent reported total processing Time for his 8Mb file reduced from 30 minutes to 9 seconds (on five-year-old hardware).

The bug was reported to the libxml team, who have now fixed it.

This is a Linux Live CD based on KNOPPIX technology using mostly pure Debian/sid.
Most tools follow the GNU GENERAL PUBLIC LICENSE. If you are in doubt, look at the specific docs.
Main features:
- Kernel 2.6.11.11 with many patches
- ACPI and DMA enabled by default (can be disabled with acpi=off and nodma respectively)
- i586 optimization - not for use with older CPUs!
- 128 MB RAM required, 256 MB RAM recommended
- Unionfs support (with unionfs cheat)
- AVM Fritz!Card DSL support
- Eagle USB DSL support
- KDE 3.4.1
- Captive 1.1.5
- OpenOffice 1.9.104
- ALSA 1.0.8
- GRUB boot loader for CD start - ideal for rescue in command line mode
- Memtest86+ - Advanced Memory Diagnostic Tool in the extra menu of the boot loader
Simply boot from CD and enjoy Linux. Some tools may request a root password. As none is set, you have to set one using "sudo passwd" or use "su" in konsole.
Whenever you execute something with root permissions, you should know what you are doing! For web surfing over LAN no root access is required. I am sure you can discover many things to do with it :)
Dont try to write to NTFS partitions using the standard NTFS driver included with the kernel, as its support for writing is very lacking - thus it could destroy the partition and the data in it.
There is a new Captive NTFS driver that does a better job at writing to NTFS, so use it instead. For FAT partitions you can enable write-access with the context menu (right mouse button).
You may not have the rights to modify Linux partitions, use the root mode in konsole if needed. Some links are not working in the menu - that is not my fault - the packages from Debian/sid are very new and may have some little bugs, but you can always use new releases.

NetWhistler 2.6 is a easy to use Network monitoring software that offers integrated fault and performance management functionality.
Mila NetWhistler project auto-detects networks and presents them on graphical maps, which reflect up-to-the-minute device and host status. It includes powerful diagnostic tools including ping, main services (ftp, ssh, smtp, pop3, http) monitoring, portscan, email alerts, network discovery, SNMP tools (MIB browser and Trap Console for incoming SNMP traps from any SNMP-enabled device in the network ).
NetWhistler 2.6 can automatically scan a network in search of non-SNMP and SNMP-enabled devices that you want to manage in your network. Scanning parameters ( network address, network mask, SNMP community ) are fully configurable.
Once the discovery operation has been completed, the program displays a map of all devices along with basic information about them such as address, DNS name, MAC address and description. You can find or delete a node on the map, or change node properties. The map can be stored in a local file or in a MySQL database.
Enhancements:
- New features include a Web console and the ability to import and export the database.
- The firewall builder was removed.
- Custom ping, mac-detect, and snmp-results scripts were added.
- Custom node types were added.
- Some bugfixes were made.

ndisc6 consists of two small command line tools (ndisc6 and rdisc6) that perform ICMPv6 Neighbor Discovery and ICMPv6 Router Discovery respectively.
NDisc6 is primarily meant for IPv6 networking diagnostics or to detect rogue IPv6 nodes or routers on an Ethernet segment.
Enhancements:
- traceroute has been parallelized for faster execution, and handles IPv6 extension headers and more ICMPv6 errors, UDP-Lite probes.
- rdisc6 supports RFC4191 (router preference and specific routes) and RDNSS.

Test::More is yet another framework for writing test scripts.

SYNOPSIS

use Test::More tests => 23;
# or
use Test::More qw(no_plan);
# or
use Test::More skip_all => $reason;

BEGIN { use_ok( Some::Module ); }
require_ok( Some::Module );

# Various ways to say "ok"
ok($got eq $expected, $test_name);

is ($got, $exptected, $test_name);
isnt($got, $expected, $test_name);

# Rather than print STDERR "# heres what went wrongn"
diag("heres what went wrong");

like ($got, qr/expected/, $test_name);
unlike($got, qr/expected/, $test_name);

cmp_ok($got, ==, $expected, $test_name);

is_deeply($got_complex_structure, $expected_complex_structure, $test_name);

SKIP: {
skip $why, $how_many unless $have_some_feature;

ok( foo(), $test_name );
is( foo(42), 23, $test_name );
};

TODO: {
local $TODO = $why;

ok( foo(), $test_name );
is( foo(42), 23, $test_name );
};

can_ok($module, @methods);
isa_ok($object, $class);

pass($test_name);
fail($test_name);

BAIL_OUT($why);

# UNIMPLEMENTED!!!
my @status = Test::More::status;

STOP! If youre just getting started writing tests, have a look at Test::Simple first. This is a drop in replacement for Test::Simple which you can switch to once you get the hang of basic testing.

The purpose of this module is to provide a wide range of testing utilities. Various ways to say "ok" with better diagnostics, facilities to skip tests, test future features and compare complicated data structures. While you can do almost anything with a simple ok() function, it doesnt provide good diagnostic output.

Devel::RingBuffer is a Perl module with shared memory ring buffers for Perl scripts diagnosis/debug.

SYNOPSIS

#
# create ringbuffer
#
use Devel::RingBuffer;
use Devel::RingBuffer::TieInt;

my $ringbuf = Devel::RingBuffer->new(
File => somefile.trace,
Rings => 20,
Slots => 20,
SlotSize => 300,
MessageSize => 256,
GlobalSize => 24 * 1024,
StopOnCreate => 0,
TraceOnCreate => 1) || die "Cant create a ring buffer.";

my $ring = $ringbuf->allocate();

Provides shared memory structures (using memory mapped files via IPC::Mmap) to be used by diagnostic and debugger applications for Perl scripts (see Devel::STrace). Using XS/C code to maximize performance, creates a set of ringbuffers with a configurable number of slots.

Each slot includes a field for a linenumber, a timestamp, and a fully qualified subroutine name. Each ring buffer also includes additional headers and fields to support diagnostic interfaces, e.g., watched expressions, command/reponse interfaces to the monitored applications, etc.

NAT and blocking all but Port 22 is a simple iptables firewall script.

Sample:

# Internal and External Devices
dev_world=ppp0
dev_int=eth0

# Firewall IP
addr_int=192.168.1.1

# Internal Net
net_int=192.168.1.0/24

###################################################
# Load Modules
insmod ip_tables
insmod ip_conntrack
insmod ip_conntrack_ftp
insmod ipt_state
insmod iptable_nat
insmod ipt_MASQUERADE

###################################################
# Delete all Rules in Filtertable
iptables -F

###################################################
# Define new chains
iptables -N BLOCK
iptables -N EXT-INT
iptables -N INT-EXT
iptables -N ICMP-DENY
iptables -N INT-IF
iptables -N EXT-IF

###################################################

iptables -A BLOCK -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A BLOCK -m state --state NEW -i ! $dev_world -j ACCEPT
iptables -A BLOCK -j DROP

iptables -A INPUT -j BLOCK
iptables -A FORWARD -j BLOCK

###################################################
# Point to chains
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i $dev_int -s $net_int -j INT-IF
iptables -A INPUT -d ! $addr_int -i $dev_world -s ! $net_int -j EXT-IF
iptables -A INPUT -j DROP

iptables -A FORWARD -d ! $net_int -i $dev_world -s $net_int -j INT-EXT
iptables -A FORWARD -d $net_int -i $dev_int -s ! $net_int -j EXT-INT
iptables -A FORWARD -j DROP

iptables -A OUTPUT -j ACCEPT

###################################################
# Chain Rules
iptables -A EXT-INT -j DROP

iptables -A EXT-IF -i ! $dev_world -j DROP
iptables -A EXT-IF -p tcp --dport 22 -j ACCEPT
iptables -A EXT-IF -p tcp --dport 5901 -j ACCEPT
iptables -A EXT-IF -p tcp --dport 1024: -j ACCEPT
iptables -A EXT-IF -p udp --dport 1024: -j ACCEPT
iptables -A EXT-IF -j DROP

iptables -A INT-IF -j ACCEPT

###################################################
# NAT Rules
# Standard Routing
iptables -A POSTROUTING -t nat -o $dev_world -j MASQUERADE -s $net_int

# Port Forwarding
#iptables -A PREROUTING -t nat -p tcp -d 192.168.1.1 --dport 5901 --to 192.168.1.2:5901 -j DNAT

##################################################
# Enable IP-Forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward)

Iterator::IO is a Perl module with filesystem and stream iterators.

SYNOPSIS

use Iterator::IO;

# Return the names of files in a directory (except . and ..)
$iter = idir_listing ($path);

# Return all the files in a directory tree, one at a time.
# Like File::Find, in slow motion.
$iter = idir_walk ($path);

# Return the lines of a file, one at a time.
$iter = ifile ($filename, %options);

# Return the lines of a file, in reverse order
$iter = ifile_reverse ($filename, %options);

This module provides filesystem and stream iterator functions. See the Iterator module for more information about how to use iterators.

FUNCTIONS

idir_listing

$iter = idir_listing ($path);

Iterator that returns the names of the files in the $path directory. If $path is omitted, defaults to the current directory. Does not return the . and .. files (under unix).

Requires IO::Dir and Cwd.

Example:

To return only certain files, combine this with an igrep:

$iter = igrep {-s && -M < 1} idir "/some/path";

(Returns non-empty files modified less than a day ago). (igrep) is defined in the Iterator::Util module).

idir_walk

$iter = idir_walk ($path);

Returns the files in a directory tree, one by one. Its sort of like File::Find in slow motion.

Requires IO::Dir and Cwd.

ifile

$iter = ifile ($filename, %options);

Opens a file, generates an iterator to return the lines of the file.
%options is a reference to a hash of options. Currently, two options are supported:

chomp

chomp => boolean indicates whether lines should be chomped before being returned by the iterator. The default is true.

$/

$/ => value specifies what string to use as the record separator. If not specified, the current value of $/ is used.

"rs" or "input_record_separator" may be used as option names instead of "$/", if you find that to be more readable. See the English module.

Option names are case-insensitive.

ifile requires IO::File.
ifile_reverse

$iter = ifile_reverse ($filename, %options);

Exactly the same as "ifile", but reads the lines of the file backwards.
The input_record_separator option values undef (slurp whole file) and scalar references (fixed-length records) are not currently supported.

INTERFACE CHANGE

In version 0.01 of Iterator::IO, the "ifile" and ifile_reverse functions accepted their options in a different manner. This has now changed to operate via a hash reference of options. The old way will still work, but is deprecated and will be removed in a future release.

EXPORTS

This module exports all function names to the callers namespace by default.

DIAGNOSTICS

Iterator::IO uses Exception::Class objects for throwing exceptions. If youre not familiar with Exception::Class, dont worry; these exception objects work just like $@ does with die and croak, but they are easier to work with if you are trapping errors.

See the Iterator module documentation for more information on how to trap and handle these exception objects.

Parameter Errors

Class: Iterator::X::Parameter_Error

You called an Iterator::IO function with one or more bad parameters. Since this is almost certainly a coding error, there is probably not much use in handling this sort of exception.

As a string, this exception provides a human-readable message about what the problem was.

Exhausted Iterators

Class: Iterator::X::Exhausted

You called value on an iterator that is exhausted; that is, there are no more values in the sequence to return.

As a string, this exception is "Iterator is exhausted."

I/O Errors

Class: Iterator::X::IO_Error

This exception is thrown when any sort of I/O error occurs; this only happens with the filesystem iterators.

This exception has one method, os_error, which returns the original $! that was trapped by the Iterator object.

As a string, this exception provides some human-readable information along with $!.

Internal Errors

Class: Iterator::X::Internal_Error

Something happened that I thought couldnt possibly happen. I would appreciate it if you could send me an email message detailing the circumstances of the error.