Free netflow software for linux

DB based NetFLow Collector aims to collect Cisco NetFlow data and store it to a database.
DB based NetFlow Collector has a plugin interface, which makes it flexible for fitting in particular tasks.
Enhancements:
- First release. post your comments/bug reports.

cflowd is a flow analysis tool currently used for analyzing Ciscos NetFlow enabled switching method.

The current release (described below) includes the collections, storage, and basic analysis modules for cflowd and for arts++ libraries. This analysis package permits data collection and analysis by ISPs and network engineers in support of capacity planning, trends analysis, and characterization of workloads in a network service provider environment. Other areas where cflowd may prove useful are: tracking for Web hosting, accounting and billing, network planning and analysis, network monitoring, developing user profiles, data warehousing and mining, as well as security-related investigations.


cflowd is no longer supported by CAIDA. Instead, please consider the use of flow-tools, which will provide a toolset for working with NetFlow data. flow-tools can also be used (like cflowd) in conjunction with FlowScan, maintained by Dave Plonka at the University of Wisconsin, Madison.

glFlow is a (D)DoS logger written with speed in mind. glFlow detects attacks on high speed links through real-time flow aggregation and analysis.
What do I run it on ?
It was written on FreeBSD and tested on both FreeBSD and Linux. It should work on any OS to which libpcap and OpenSSL were ported. The rest of the code is perfectly portable.
How does it work ?
Cisco Systems have defined the flow as a four value tuplet: {srcaddr, srcport, dstaddr, dstport}. The format evolved over time. The complete structures for various NetFlow versions are available on Ciscos site. Now, lets assume that the attacker floods the victim with packets that keep the same characteristics throughout the duration of the attack. No source spoof, no
source port increments or randomizations. That would lead to a very large packet rate inside that flow. glFlow calculates the average packet rate in every flow and raises an alarm signal if the threshold is hit.
What about spoofed attacks ? How are they detected ? Simple. glFlow keeps a history for every destination host that it sees. When a new flow is created, the flow counter for that host is incremented. The average number of newly created flows corresponding to a specific host in a specific amount of time is calculated, and, as above, an alarm is raised if the threshold is hit.
To prevent attacks that dont hit any of the above thresholds, theres
a new one starting with v0.1, measuring the packet rate for a destination.
Cant other tools, like SNORT, do this ?
We sincereley believe not. Remember, glFlow was written with high
speeds in mind. Weve been using it at over 500Mbps. At that speed, with an
ordinary x86 machine, even with a strong motherboard/NIC combination, you cant
do anything fancy. glFlow was specifically designed for detecting large floods
in real time, or at least something close to that.
How is it that its so fast ?
Well, Andrei did a great job implementing a very fast binary tree. That allowed us to drop the threaded model and choose a single loop design. The new results were stunning. The tests were made on a P4 Xeon/3 GHz, with an Intel GigE NIC. The average traffic rate was about 500Mbps, with an average packet rate of 100kpps. That lead to about 200k active flows. glFlow managed to clean the inactive ones in less than 0.3 seconds. There was no alarm raised
after more than 5 seconds of flooding. glFlow ate ~50% of the CPU, while consuming about 40MB of system memory.
How do I install and run it ?
Run ./configure --help. Youll see two adjustable knobs: --with-hash and --enable-debug. The first one permits you to switch between MD4 and MD5 summing of the flow and host structures kept in the memory. The second lets you run glflow in the foreground, printing some statistics on stdout.
The thresholds are harcoded in defs.h. You shouldnt have any trouble tweaking them. However, weve observed that the best results are obtained when using the same values for flow lifetime and the time between flow cleanups. And they shouldnt be much over 20. The smaller the tree is, the faster it will be cleaned.
Finally, edit your /etc/syslog.conf and write something like this: "local6.*< tabs >/var/log/something". Restart sys[k]logd afterwards.
Fire glFlow up, like this: "./glFlow < interface > < bpf filter >" and watch /var/log/something for changes. You may play with nmap or some DoS programs to test it. The IPs in the syslog will be shown as integers rather than in dotted notation. We decided to leave this job to the log analyzer.
Can it go even faster ?
Sure. There are a few methods which permit you to improve the packet capture. For more info read Luca Deris paper: http://luca.ntop.org/Ring.pdf
Enhancements:
- This is a bugfix release.

FlowScan is a network analysis and reporting tool.[ COPYRIGHT=1]
Enhancements:
- The CampusIO and SubNetIO reports were enhanced with a new optional configuration directive: TopN. When defined, this directive causes ``Top Talker reports to be produced. These HTML reports contain the most active (i.e. ``top) source and destination addresses.
- The CampusIO and SubNetIO reports were enhanced to record the number of local IP addresses that where active for each network and subnet into the RRD files. This enables users to estimate the number of active hosts hosts over time, detect ``scans which systematically sweep across network address space, and to calculate the average bytes, packets, and flows per host.
- The template Makefile used to produce the graphs was enhanced to allow the inclusion of ``events in the graphs, similarly to what can be done with Cricket. This allows you to label events such as configuration changes and outages to discover correlations with traffic measurement.
- Two new utilities suitable for stand-alone use, are included. ip2hostname converts IP addresses to their respective hostnames. event2vrule adds ``events to rrdtool graphs.
- Added support for LFAP (Lightweight Flow Accouting Protocol) used by Riverstone and Enterasys (formerly Cabletron) routers. This currently requires slate (from http://www.nmops.org) and lfapd by Steven Premeau . lfapd produces time-stamped raw flow files in the same cflowd-defined format that is processed by FlowScan.
- Added the ability for the CampusIO report to identify outbound flows based solely on the flows destination IP address. While this is less trustworthy than using NextHops or OutputIfIndexes, it is now the default and will be useful for environments where the flow nexthop or output ifIndex values are not meaningful.
- The CampusIO report contains a new experimental feature which reads a BGP routing table, and therefore can determine which Autonomous systems source, transit, or sink most of your institutions traffic. The CampusIO report was enhanced with new optional configuration directives: BGPDumpFile, TopN, ReportPrefixFormat. When properly defined, these directives cause CampusIO to create tabular HTML reports named {origin|path}_{in|out}.html under OutputDir after analyzing each raw flow file. These reports show the ``top Autonomous Systems with which your site exchanges traffic.
- A WebProxyIfIndex directive was added to the CampusIO report. This allows one to specify the index of the interface to which HTTP traffic is being transparently redirected. This enables FlowScan to properly count HTTP flows even though NetFlow v5 does not accurately report the nexthop value for flows which are transparently redirected via a Cisco route-map.
- CampusIO now contains a fix for a bug introduced in FlowScan-1.005 which would sometimes cause perl to abort with this message: patricia.c:645: patricia_lookup: Assertion `prefix failed.

OSSP flow2rrd is a companion tool to the Flow-Tools toolkit for storing NetFlow network traffic data in an accumulating fixed-size RRDTool Round-Robin-Database (RRD) for visualization purposes.
This file is piece of OSSP flow2rrd, a tool for storing NetFlow data into an RRD which can be found at http://www.ossp.org/pkg/tool/flow2rrd/.
Enhancements:
- Created the initial version of OSSP flow2rrd.

EHNT is a tool which turns streams of Netflow (version 5) data into something useful and human-readable. (Netflow is a UDP-based traffic reporting protocol created by Cisco, generated by Cisco, Juniper, Foundry, and other routers.)
Netflow operates in many ways. It will dump flow records in human-readable form. It will also provide reports on top ASes, IP protocols, and tcp/udp ports. The reports can be generated over various intervals, from 1 minute to 1 day.
Component programs are:
1. ehntserv listens to netflow version 5 UDP packets, and also listens for client TCP connections. When a TCP client connects, the server starts forwarding all the netflow packets it receives (plus the IP address of the originating device) to that client.
ehntserv does not currently do any IP access control. I suggest that you use ipchains or iptables on your linux box, or IP Filter (ipf) (http://coombs.anu.edu.au/ipfilter/) on your Solaris or BSD box. I dont know what the current state of packet filtering is on other Unixes; IP Filter seems to support several.
2. ehnt connects to ehntserv and displays the flows it receives in various ways. It currently has four modes (-m ):
- top mode displays average utilization by top ASes, IP protocols, or tcp/udp ports over a given interval (from 1 minute to 1 day).
Top mode is different when it focuses on a single interface on a single router, because then you get to see summaries of source and destionation for both inbound and outbound traffic. Otherwise, you just get summaries of source and destination.
- dump mode displays individual flows
- shortdump mode display individual flows in a more compact but hard
to read fashion
- colondump mode display individual flows in a machine-readable format.
And yes, I recognize that the name of this mode is unpleasant.
In all three modes, simple (REALLY simple) filtering can be done for AS
number, TCP/UDP port, IP protocol number, device sending the flow record,
and SNMP interface index.
You may think of ehnt in the three dump modes as a brain-dead and incredibly
simple tcpdump for netflow.
ehnt also has the silly and uninspiredly-named big filter, in which it
only displays flows with are bigger (in packets or bytes) than any flow
received before it. This only makes sense in the three dump modes.
Enhancements:
- Added Unix domain support for client connections, enabled by default

System for Internet-Level Knowledge (SiLK) project is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks.
The SiLK tool suite supports the efficient collection, storage and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.
SiLK consists of two sets of tools: a packing system and analysis suite. The packing system receives Netflow V5 PDUs and converts them into a more space efficient format, recording the packed records into service-specific binary flat files. The analysis suite consists of tools which can read these flat files and then perform various query operations, ranging from per-record filtering to statistical analysis of groups of records. The analysis tools interoperate using pipes, allowing a user to develop a relatively sophisticated query from a simple beginning.
The vast majority of the current code-base is implemented in C, Perl, or Python. This code has been tested on Linux, Solaris, OpenBSD, and Mac OS X, but should be usable with little or no change on other Unix platforms.
System for Internet-Level Knowledge software components are released under the GPL.
Enhancements:
- New scan detection system: rwscan and rwscanquery
- rwscan reads SiLK Flow data and uses a hybrid of Threshold Random Walk and Bayesian Logistic Regression to detect scanning activity. rwscan output textual records describing the scan. If these are inserted into a relational database, rwscanquery can be used to query for the scanning activity. rwscanquery can query Oracle, Postgres, or MySQL databases.
- New tools for IPFIX support
- rwsilk2ipfix converts SiLK Flow records to an IPFIX format.
- rwipfix2silk converts IPFIX flow records to the SiLK format.
- These tools can be used in place of the rwp2yaf2silk script.
- Support for these tools requires that libfixbuf-0.6.0 be installed prior to building SiLK.
- New tools for IP storage
- rwipaexport takes IP addresses from an IP Address Association (IPA) catalog and creates a SiLK IPset, Bag, or Prefix Map (pmap).
- rwipaimport enters the IP addresses from a SiLK IPset, Bag, or Prefix Map into an IPA catalog.
- Support for these tools requires that libipa-0.2.0 be installed prior to building SiLK.
- Additional new tools
- rwsplit divides a SiLK Flow file into smaller files based on the number of flows, bytes, packets, or unique IPs. It also provides the ability to sample the input.
- rwsettool provides the functionality of rwsetintersect and rwsetunion and additional functions such as set difference and sampling of an IPset. The rwsetintersect and rwsetunion tools are deprecated.
- rwsetmember determines if a (textual) IP is a member of an IPset. Determinating this in previous releases of SiLK required filtering the output of rwsetcat or creating an IPset containing a single IP.
- rwpmapcat prints the contents of a Prefix Map (pmap) file.
- rwfilter enhancements and bug fixes
- Allow the the parameter to the --flags-all, --flags-init, and --flags-session switches can be a list of HIGH/MASK pairs separated by commas, e.g., --flags-all=S/S,A/A
- Do not print statistics or create output files when the --dry-run switch is specified.
- Fix a file corruption issue that would occur when processing multiple files if the first input file was not successfully opened: the output file would be generated without a SiLK header.
- Exit with a non-zero exit status if the class, type, or sensor values are invalid.
- Fix a bug in processing the --start-date and --end-date switches when local timezone support was enabled and the local timezone was east of UTC.
- rwbag enhancements and bug fixes
- rwbag now supports creating Bags whose key is the sensor ID, next hop IP, input interface or output interface.
- Allow rwbag to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
- Print errors as human readable text, not error codes
- Fix a bug with releasing memory multiple times when rwbag ran out of memory.
- rwrandomizeip enhancement
- Allow the user to restrict the set of IPs that are modified via two command line arguments: --dont-change-set and --only-change-set. Both switches take an IPset; the first switch prevents the IP from being changed; the second causes only the listed IPs to be changed.
- mapsid enhancement
- The --print-classes switch will print the class(es) to which each sensor belongs.
- rwcount enhancement and changes
- Implemented the --output-path switch which directs rwcount to write its output to the specified location.
- Allow rwcount to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
- The column widths have changed slightly
- rwaddrcount enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwcut enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwstats enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwset enhancement
- Implmented the --copy-input switch as described for rwcount.
- rwtotal enhancement
- Implemented the --output-path switch as described for rwcount.
- rwuniq enhancement
- Implemented the --output-path switch as described for rwcount.
- rwsetcat bug fix
- Fix bug where the $PAGER was not being used.
- rwbagcat bug fixes
- Do not print a warning message when attempting to print an empty Bag or when the min/max limits caused no entries to be printed.
- Fix bug where the $PAGER was not being used.
- Print errors as human readable text, not error codes
- rwbagtool bug fix
- Print errors as human readable text, not error codes
- rwcat bug fix
- Modify rwcat so it will always print the SiLK header to a file, even when no records are present
- rwappend enhancement and bug fix
- New --print-statistics switch causes the number of records processed to be printed to the standard error.
- Output change: Modified rwappend so it only prints the number of records processed when --print-statistics is given.
- Fix a problem that occurred when SiLK was compiled with compression enabled by default and the applications were processing SiLK files produced by releases of SiLK prior to 0.10.5: the application would exit with the error message "Operation not permitted on compressed file" and no output would be generated.
- rwswapbytes bug fix
- See compression-related bug fix for rwappend
- rwnetmask bug fix
- See compression-related bug fix for rwappend
- Administration and configuration changes:
- New "silk.conf" file removes the requirement that sensors be defined at compile-time.
- The sensors, classes, and types are now defined at run-time through the use of a "silk.conf" text file. This file should be installed in the SILK_DATA_ROOTDIR directory.
- The run-time configuration allows a single installation of the analysis tools to query multiple data sets; simply set the SILK_DATA_ROOTDIR environment variable to the location of the data.
- The location of this file can also be specified by setting the SILK_CONFIG_FILE environment variable to its location, or by using the --site-config-file switch on most SiLK applications.
- The packer (rwflowpack) still requires certain classes and types to be defined, and it cannot use new classes and types without modifying C code. This restriction will go away in a future release.
- Major changes to the build system.
- The build system now uses all aspects of the GNU Autotools chain including automake and libtool.
- The tools can now be built with shared library support, reducing the size of the binaries and allowing the kernel to use a single copy of libsilk when multiple SiLK tools are running.
- Note that the use of shared libraries means the binaries can no longer easily be relocated; instead you should run "make install" again with the new location.
- The SiLK headers are now copied to the install target directory
- GNU make is no longer required to build the tools.
- New packing rules are used by default.
- The default site has changed from "generic" to "twoway". The twoway site allows flow records to be categorized and stored as internal-to-internal (int2int) and external-to-external (ext2ext). In addition, the "out" type is no longer everything that is not "in". The files created by the generic site are forward compatible with the twoway site; however, if you wish to continue using your current packing rules, run configure with the --enable-silk-site=generic switch. See the SiLK Installation Handbook for details.
- New transfer daemons: rwsender and rwreceiver
- These are meant to replace the direct connectivity between flowcap and rwflowpack. These daemons allow the flowcap files to be sent to multiple rwflowpack processes.
- In addition, they allow rwflowpack to process data on one system and send small files containing SiLK Flow records (called "incremental files") to another system (where the rwflowappend daemon is running) for analysis.
- New packing tool: rwflowappend
- rwflowappend appends SiLK Flow records contained in "incremental files" to hourly files.
- Changes to flowcap and rwflowpack
- The flowcap and rwflowpack tools have been modified to work with the new rwsender and rwreceiver, though they can also be used in legacy mode. With the transport removed from flowcap, flowcap files can now be sent to multiple locations.
- IPFIX flow collection enhancement
- Previous releases of SiLK (rwflowpack and flowcap) could only read IPFIX streams generated by YAF. With this release, SiLK can read flows from any IPFIX-compliant generator.
- Remove zlib requirement in rwflowpack
- Allow rwflowpack to be built even if zlib is not available. However, rwflowpack will not be able to read files of NetFlow PDUs when zlib is not present.
- New packing tool: rwpackchecker
- rwpackchecker performs a basic integrity check of a packed SiLK file.

DNA (Deep Network Analyser) is an open, flexible, and extensible deep network analyzer server and software architecture for passively gathering and analyzing network packets, network sessions, and applications protocols.
Deep Network Analyser project is designed to be used for Internet security, network management, intrustion detection, protocol and network analysis, information gathering, and network monitoring applications.
Main features:
- Extensible Java based network sensor (processing layers 2-7)
Configurable processing and output:
- Packet flows like Ethereal
- IP Flows like CISCO netflow
- Stateful Sessions (client/server flow pairs)
- Application protocol element output
- Configurable and extensible application protocol element parsing.
- Application protocol parsing toolkit APIs allows for new protocol parser to be easily developed and extended
- Targeting based full session capture facility, like a realtime targeted TCPDump.
- Flexible targeting from IPAddr, Port tuple to Application sensitive targeting.
- Configurable and extensible output forwarding (file, DB, Streams, JMS, RMI, etc.)
- Extensible realtime collection portable to many OS/Packet processing environments
Easily adaptable to packet processing environments:
- Specialized linux drivers mechanismon
- Network Appliances
- Network Switches / Routers
- Highly mutithreaded for increased performance over multi processor environments
Enhancements:
- Adoption of OpenAdaptor(tm) as the Output Adapter mechanism.
- Support for local-only administration.
- A new targeted packet capture parser, new run scripts, and a new install mechanism.
- Many bugfixes.