Free netflow analysis software for linux

Net::Analysis are modules for analysing network traffic.

SYNOPSIS

Using an existing analyser:

$ perl -MNet::Analysis -e main help
$ perl -MNet::Analysis -e main TCP,v=1 dump.tcp - basic TCP info
$ perl -MNet::Analysis -e main HTTP,v=1 dump.tcp - HTTP stuff
$ perl -MNet::Analysis -e main Example2,regex=img dump.tcp - run an example

Writing your own analyser:

package MyExample;
use base qw(Net::Analysis::Listener::Base);
# Listen to events from other modules
sub tcp_monologue {
my ($self, $args) = @_;
my ($mono) = $args->{monologue};
my $t = $mono->t_elapsed()->as_number();
my $l = $mono->length();
# Emit your own event
$self->emit(name => example_event,
args => { kb_sec => ($t) ? $l/($t*1024) : N/A }
);
}
# Process your own event
sub example_event {
my ($self, $args) = @_;
printf "Bandwidth: %10.2f KB/secn", $args->{kb_sec};
}
1;
__top

ABSTRACT

Net::Analysis is a suite of modules that parse tcpdump files, reconstruct TCP sessions from the packets, and provide a very lightweight framework for writing protocol anaylsers.
__top

I wanted a batch version of Ethereal in Perl, so I could:

- sift through parsed protocols with structured filters
- write custom reports that mixed events from multiple protocols

So here it is. Net::Analysis is a stack of protocol handlers that emit, and listen for, events.

Network Security Analysis Tool is a fast, stable bulk security scanner designed to audit remote network services and check for versions, security problems, gather information about the servers and the machine, and much more.

A manpage providing extensive information on NSAT has been included in the distribution. It is available after a make install, or just by typing man doc/nsat.8 from this dir. It is suggested that you inform yourself at least about the -v (scan verbosity) option and edit the configuration file. To learn about changes in this version, please consult doc/CHANGES.

New to this version is support for distributed scanning. The manpage describes how to do a distributed scan. Note that distributed scanning in this version is just a preliminary, proof-of-concept, implementation with no guarantees for its security, reliability, or performance.

Check for updated vulnerability lists, config files, etc. from
http://nsat.sourceforge.net

Currently, these are lists of vulnerabilities:

nsat.cgi (CGI scripts)
nsat.conf (configuration)
src/mod/snmp.h (SNMP community names)

DNA (Deep Network Analyser) is an open, flexible, and extensible deep network analyzer server and software architecture for passively gathering and analyzing network packets, network sessions, and applications protocols.
Deep Network Analyser project is designed to be used for Internet security, network management, intrustion detection, protocol and network analysis, information gathering, and network monitoring applications.
Main features:
- Extensible Java based network sensor (processing layers 2-7)
Configurable processing and output:
- Packet flows like Ethereal
- IP Flows like CISCO netflow
- Stateful Sessions (client/server flow pairs)
- Application protocol element output
- Configurable and extensible application protocol element parsing.
- Application protocol parsing toolkit APIs allows for new protocol parser to be easily developed and extended
- Targeting based full session capture facility, like a realtime targeted TCPDump.
- Flexible targeting from IPAddr, Port tuple to Application sensitive targeting.
- Configurable and extensible output forwarding (file, DB, Streams, JMS, RMI, etc.)
- Extensible realtime collection portable to many OS/Packet processing environments
Easily adaptable to packet processing environments:
- Specialized linux drivers mechanismon
- Network Appliances
- Network Switches / Routers
- Highly mutithreaded for increased performance over multi processor environments
Enhancements:
- Adoption of OpenAdaptor(tm) as the Output Adapter mechanism.
- Support for local-only administration.
- A new targeted packet capture parser, new run scripts, and a new install mechanism.
- Many bugfixes.

Statistical Traffic Analysis Kit is a set of command-line traffic analysis tools, designed to help a network administrator to see what is happening at a router at the moment.

Unlike tcpdump (1), the stak set uses statistical and stream-oriented methods, and will rarely produce an output stream at a speed beyond human perception. The output is less accurate.

The kit consists of five different utilities, designed to perform the following tasks:
estimating overall traffic rates (stakrate),
determining network nodes generating the highest traffic (stakhosts)
monitoring the amount of traffic exchanged with particular autonomous
systems (stakasta),
extracting strings from packets (stakextract),
determining connections and flows generating the highest traffic
(stakstreams, experimental),

FlowScan is a network analysis and reporting tool.[ COPYRIGHT=1]
Enhancements:
- The CampusIO and SubNetIO reports were enhanced with a new optional configuration directive: TopN. When defined, this directive causes ``Top Talker reports to be produced. These HTML reports contain the most active (i.e. ``top) source and destination addresses.
- The CampusIO and SubNetIO reports were enhanced to record the number of local IP addresses that where active for each network and subnet into the RRD files. This enables users to estimate the number of active hosts hosts over time, detect ``scans which systematically sweep across network address space, and to calculate the average bytes, packets, and flows per host.
- The template Makefile used to produce the graphs was enhanced to allow the inclusion of ``events in the graphs, similarly to what can be done with Cricket. This allows you to label events such as configuration changes and outages to discover correlations with traffic measurement.
- Two new utilities suitable for stand-alone use, are included. ip2hostname converts IP addresses to their respective hostnames. event2vrule adds ``events to rrdtool graphs.
- Added support for LFAP (Lightweight Flow Accouting Protocol) used by Riverstone and Enterasys (formerly Cabletron) routers. This currently requires slate (from http://www.nmops.org) and lfapd by Steven Premeau . lfapd produces time-stamped raw flow files in the same cflowd-defined format that is processed by FlowScan.
- Added the ability for the CampusIO report to identify outbound flows based solely on the flows destination IP address. While this is less trustworthy than using NextHops or OutputIfIndexes, it is now the default and will be useful for environments where the flow nexthop or output ifIndex values are not meaningful.
- The CampusIO report contains a new experimental feature which reads a BGP routing table, and therefore can determine which Autonomous systems source, transit, or sink most of your institutions traffic. The CampusIO report was enhanced with new optional configuration directives: BGPDumpFile, TopN, ReportPrefixFormat. When properly defined, these directives cause CampusIO to create tabular HTML reports named {origin|path}_{in|out}.html under OutputDir after analyzing each raw flow file. These reports show the ``top Autonomous Systems with which your site exchanges traffic.
- A WebProxyIfIndex directive was added to the CampusIO report. This allows one to specify the index of the interface to which HTTP traffic is being transparently redirected. This enables FlowScan to properly count HTTP flows even though NetFlow v5 does not accurately report the nexthop value for flows which are transparently redirected via a Cisco route-map.
- CampusIO now contains a fix for a bug introduced in FlowScan-1.005 which would sometimes cause perl to abort with this message: patricia.c:645: patricia_lookup: Assertion `prefix failed.

TA-Lib provides common functions for the technical analysis of stock/future/commodity market data.
TA-Lib can be reused by trading software developers using Excel, .NET, Java, Perl or C/C++.
Main features:
- More than 120 technical analysis indicators such as ADX, MACD, RSI, Stochastic, Bollinger Bands etc...
- bullet Includes candlestick pattern recognition.
- bullet Optional abstract interface allowing your code to support new technical analysis functions without any code change!
Enhancements:
New Features
- New Functions: BETA, MINMAX, MINMAXINDEX, MININDEX, MAXINDEX
- Debian and RPM packaging available.
- Java JAR packaging available.
- New TA_FunctionDescription() returns XML description of API.
- New ta_func_api.xml file generated in root directory of the package.
- Support for unmanaged static libraries with Visual Studio 2005.
Fixes
- #1526632 : Fix bug in LINEARREG_ANGLE
- #1544555 : Now do proper divide by zero detection in TA_ADX
Other Changes
- Better Java/.NET naming convention.
- ta_func_list.txt moved in root directory of the package.
- Removed dependencies on trio and Mersenne Twister functions.
- Volume and Open Interest are now double instead of integers.
- Add license specific to Excel users.

flow-tools is a set of programs for processing and managing NetFlow exports from Cisco and Juniper routers. The tools included are: flow-capture, flow-cat, flow-dscan, flow-expire, flow-export, flow-fanout, flow-filter, flow-gen, flow-header, flow-import, flow-mask, flow-merge, flow-nfilter, flow-print, flow-receive, flow-report, flow-send, flow-split, flow-stat, flow-tag, and flow-xlate.

Flow data is collected and stored by default in host byte ordera nd the files are portable across every endian architectures.

Commands that utilize the network use a localip/remoteip/port designation for communication. "localip" is the IP address the host will use as a source for sending or bind to when receiving NetFlow PDUs (ie the destination address of the exporter. Configuring the "localip" to 0 will force the kernel to decide what IP address to use for sending and listen on all IP addresses for receiving. "remoteip" is the destination IP address used for sending or the expected address of the source when receiving. If the "remoteip" is 0 then the application will accept flows from any source address. The "port" is the UDP port number used for sending or receiving. When using multicast addresses the localip/remoteip/port is used to represent the source, group, and port respectively.

Flows are exported from a router in a number of different configurable versions. A flow is a collection of key fields and additional data. The flow key is {srcaddr, dstaddr, input, output, srcport, dstport, prot, ToS}. Flow-tools supports one export version per file.

Export versions 1, 5, 6, and 7 all maintain {nexthop, dPkts, dOctets, First, Last, flags}, ie the next-hop IP address, number of packets, number of octets (bytes), start time, end time, and flags such as the TCP header bits. Version 5 adds the additional fields {src_as, dst_as, src_mask, dst_mask}, ie source AS, destination AS, source network mask, and destination network mask. Version 7 which is specific to the Catalyst switches adds in addition to the version 5 fields {router_sc}, which is the Router IP address which populates the flow cache shortcut in the Supervisor. Version 6 which is not officially supported by Cisco adds in addition to the version 5 fields {in_encaps, out_encaps, peer_nexthop}, ie the input and output interface encapsulation size, and the IP address of the next hop within the peer. Version 1 exports do not contain a sequence number and therefore should be avoided, although it is safe to store the data as version 1 if the additional fields are not used.

Version 8 IOS NetFlow is a second level flow cache that reduces the data exported from the router. There are currently 11 formats, all of which provide {dFlows, dOctets, dPkts, First, Last} for the key fields.

8.1 - Source and Destination AS, Input and Output interface
8.2 - Protocol and Port
8.3 - Source Prefix and Input interface
8.4 - Destination Prefix and Output interface
8.5 - Source/Destination Prefix and Input/Output interface
8.9 - 8.1 + ToS
8.10 - 8.2 + ToS
8.11 - 8.3 + ToS
8.12 - 8.5 + ToS
8.13 - 8.2 + ToS
8.14 - 8.3 + ports + ToS

Version 8 CatIOS NetFlow appears to be a less fine grained first level flow cache.

8.6 - Destination IP, ToS, Marked ToS,
8.7 - Source/Destination IP, Input/Output interface, ToS, Marked ToS,
8.8 - Source/Destination IP, Source/Destination Port,
Input/Output interface, ToS, Marked ToS,

The following programs are included in the flow-tools distribution.

flow-capture - Collect, compress, store, and manage disk space for exported flows from a router.
flow-cat - Concatenate flow files. Typically flow files will contain a small window of 5 or 15 minutes of exports. Flow-cat can be used to append files for generating reports that span longer time periods.
flow-fanout - Replicate NetFlow datagrams to unicast or multicast destinations. Flow-fanout is used to facilitate multiple collectors attached to a single router.
flow-report - Generate reports for NetFlow data sets. Reports include source/destination IP pairs, source/destination AS, and top talkers. Over 50 reports are currently supported.
flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to group flows by customer network. The tags can later be used with flow-fanout or flow-report to generate customer based traffic reports.
flow-filter - Filter flows based on any of the export fields. Flow-filter is used in-line with other programs to generate reports based on flows matching filter expressions.
flow-import - Import data from ASCII or cflowd format.
flow-export - Export data to ASCII or cflowd format.
flow-send - Send data over the network using the NetFlow protocol.
flow-receive - Receive exports using the NetFlow protocol without storing to disk like flow-capture.
flow-gen - Generate test data.
flow-dscan - Simple tool for detecting some types of network scanning and Denial of Service attacks.
flow-merge - Merge flow files in chronoligical order.
flow-xlate - Perform translations on some flow fields.
flow-expire - Expire flows using the same policy of flow-capture.
flow-header - Display meta information in flow file.
flow-split - Split flow files into smaller files based on size, time, or tags.

glFlow is a (D)DoS logger written with speed in mind. glFlow detects attacks on high speed links through real-time flow aggregation and analysis.
What do I run it on ?
It was written on FreeBSD and tested on both FreeBSD and Linux. It should work on any OS to which libpcap and OpenSSL were ported. The rest of the code is perfectly portable.
How does it work ?
Cisco Systems have defined the flow as a four value tuplet: {srcaddr, srcport, dstaddr, dstport}. The format evolved over time. The complete structures for various NetFlow versions are available on Ciscos site. Now, lets assume that the attacker floods the victim with packets that keep the same characteristics throughout the duration of the attack. No source spoof, no
source port increments or randomizations. That would lead to a very large packet rate inside that flow. glFlow calculates the average packet rate in every flow and raises an alarm signal if the threshold is hit.
What about spoofed attacks ? How are they detected ? Simple. glFlow keeps a history for every destination host that it sees. When a new flow is created, the flow counter for that host is incremented. The average number of newly created flows corresponding to a specific host in a specific amount of time is calculated, and, as above, an alarm is raised if the threshold is hit.
To prevent attacks that dont hit any of the above thresholds, theres
a new one starting with v0.1, measuring the packet rate for a destination.
Cant other tools, like SNORT, do this ?
We sincereley believe not. Remember, glFlow was written with high
speeds in mind. Weve been using it at over 500Mbps. At that speed, with an
ordinary x86 machine, even with a strong motherboard/NIC combination, you cant
do anything fancy. glFlow was specifically designed for detecting large floods
in real time, or at least something close to that.
How is it that its so fast ?
Well, Andrei did a great job implementing a very fast binary tree. That allowed us to drop the threaded model and choose a single loop design. The new results were stunning. The tests were made on a P4 Xeon/3 GHz, with an Intel GigE NIC. The average traffic rate was about 500Mbps, with an average packet rate of 100kpps. That lead to about 200k active flows. glFlow managed to clean the inactive ones in less than 0.3 seconds. There was no alarm raised
after more than 5 seconds of flooding. glFlow ate ~50% of the CPU, while consuming about 40MB of system memory.
How do I install and run it ?
Run ./configure --help. Youll see two adjustable knobs: --with-hash and --enable-debug. The first one permits you to switch between MD4 and MD5 summing of the flow and host structures kept in the memory. The second lets you run glflow in the foreground, printing some statistics on stdout.
The thresholds are harcoded in defs.h. You shouldnt have any trouble tweaking them. However, weve observed that the best results are obtained when using the same values for flow lifetime and the time between flow cleanups. And they shouldnt be much over 20. The smaller the tree is, the faster it will be cleaned.
Finally, edit your /etc/syslog.conf and write something like this: "local6.*< tabs >/var/log/something". Restart sys[k]logd afterwards.
Fire glFlow up, like this: "./glFlow < interface > < bpf filter >" and watch /var/log/something for changes. You may play with nmap or some DoS programs to test it. The IPs in the syslog will be shown as integers rather than in dotted notation. We decided to leave this job to the log analyzer.
Can it go even faster ?
Sure. There are a few methods which permit you to improve the packet capture. For more info read Luca Deris paper: http://luca.ntop.org/Ring.pdf
Enhancements:
- This is a bugfix release.