Free netflow software for linux

Stager is a system for aggregating and presenting network statistics. Stager project is generic and can be customized to present and process any kind of network statistics.
The backend collects data and stores reports in a database, automatically handling the aggregation of hourly statistics into days, weeks, and months.
The Web frontend presents data in tables, matrices, or plots. The reports are fully customizable, and their definitions are stored in the database.
Installation:
If you are upgrading from a previous version of Stager, see the UPGRADE file.
This version of Stager only support Netflow. Future releases will also support roundtrip measurements, SNMP and various passive monitoring measurements.
To run Stager you now need PHP4 CLI with SNMP support. Under debian you can just install php4-cli and php4-snmp
Backend:
1. Create a new Postgresql user that is allowed to create new databases
2. Create a new user that runs the Stager backend. The rest of the installation should be done as this user
3. /stager-install.pl --type=backend --backends=netflow --path=/installation/path/
4. Edit /installation/path/etc/netflow.cfg and /installation/path/etc/getRouterInfo.cfg
5. cd /installation/path/bin
6. ./db_install.pl --backend=netflow
7. Default access control is to give full access to admin user and limited access to guest users. Check /installation/path/lib/getRouterInfo.custom.php if you want to change this.
8. ./getRouterInfo.php -v -v -o
9. ./get-netflow.pl -v --no-missing --delete-old Check for error messages
10. Edit /installation/path/bin/getRouterInfo.sh
11. Edit crontab:
30 0-23/2 * * * $HOME/stager/bin/getRouterInfo.sh -v //How often you run this command depends on how dynamic your network is
45 * * * * $HOME/stager/bin/get-netflow.pl --delete-old
50 1 * * * $HOME/stager/bin/aggregate.pl --backend=netflow --interval=1 day --no-distributed --manual-mode
45 2 * * 1 $HOME/stager/bin/aggregate.pl --backend=netflow --interval=1 week --timeformat YYYY-IW --no-cap --no-distributed
50 2 1 * * $HOME/stager/bin/aggregate.pl --backend=netflow --interval=1 month --timeformat YYYY-MM --no-distributed
Frontend:
1. ./stager-install.pl --type=frontend --path=/installation/path
2. edit /installation/path/config/user.config.php
Whats New in 2.0.1 Stable Release:
- This release fixes several bugs in both the frontend and backend.
- The most important fix is that bookmarks with relative times are now working properly.
- This means that if you create a report that shows data, for example, for the last hour with available data, you can now create a bookmark that will always show data for the last hour with available data.
Whats New in 3.0 Beta 1 Development Release:
- The backend has been completely rewritten and is now implemented in PHP.
- The new backend is faster and more generic so that it is easier to add support for other types of statistics.
- The new backends are also more robust against database down time.
- In the web GUI, it is now possible to right click on most data to bring up a context menu where it is possible to add custom links.

System for Internet-Level Knowledge (SiLK) project is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks.
The SiLK tool suite supports the efficient collection, storage and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.
SiLK consists of two sets of tools: a packing system and analysis suite. The packing system receives Netflow V5 PDUs and converts them into a more space efficient format, recording the packed records into service-specific binary flat files. The analysis suite consists of tools which can read these flat files and then perform various query operations, ranging from per-record filtering to statistical analysis of groups of records. The analysis tools interoperate using pipes, allowing a user to develop a relatively sophisticated query from a simple beginning.
The vast majority of the current code-base is implemented in C, Perl, or Python. This code has been tested on Linux, Solaris, OpenBSD, and Mac OS X, but should be usable with little or no change on other Unix platforms.
System for Internet-Level Knowledge software components are released under the GPL.
Enhancements:
- New scan detection system: rwscan and rwscanquery
- rwscan reads SiLK Flow data and uses a hybrid of Threshold Random Walk and Bayesian Logistic Regression to detect scanning activity. rwscan output textual records describing the scan. If these are inserted into a relational database, rwscanquery can be used to query for the scanning activity. rwscanquery can query Oracle, Postgres, or MySQL databases.
- New tools for IPFIX support
- rwsilk2ipfix converts SiLK Flow records to an IPFIX format.
- rwipfix2silk converts IPFIX flow records to the SiLK format.
- These tools can be used in place of the rwp2yaf2silk script.
- Support for these tools requires that libfixbuf-0.6.0 be installed prior to building SiLK.
- New tools for IP storage
- rwipaexport takes IP addresses from an IP Address Association (IPA) catalog and creates a SiLK IPset, Bag, or Prefix Map (pmap).
- rwipaimport enters the IP addresses from a SiLK IPset, Bag, or Prefix Map into an IPA catalog.
- Support for these tools requires that libipa-0.2.0 be installed prior to building SiLK.
- Additional new tools
- rwsplit divides a SiLK Flow file into smaller files based on the number of flows, bytes, packets, or unique IPs. It also provides the ability to sample the input.
- rwsettool provides the functionality of rwsetintersect and rwsetunion and additional functions such as set difference and sampling of an IPset. The rwsetintersect and rwsetunion tools are deprecated.
- rwsetmember determines if a (textual) IP is a member of an IPset. Determinating this in previous releases of SiLK required filtering the output of rwsetcat or creating an IPset containing a single IP.
- rwpmapcat prints the contents of a Prefix Map (pmap) file.
- rwfilter enhancements and bug fixes
- Allow the the parameter to the --flags-all, --flags-init, and --flags-session switches can be a list of HIGH/MASK pairs separated by commas, e.g., --flags-all=S/S,A/A
- Do not print statistics or create output files when the --dry-run switch is specified.
- Fix a file corruption issue that would occur when processing multiple files if the first input file was not successfully opened: the output file would be generated without a SiLK header.
- Exit with a non-zero exit status if the class, type, or sensor values are invalid.
- Fix a bug in processing the --start-date and --end-date switches when local timezone support was enabled and the local timezone was east of UTC.
- rwbag enhancements and bug fixes
- rwbag now supports creating Bags whose key is the sensor ID, next hop IP, input interface or output interface.
- Allow rwbag to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
- Print errors as human readable text, not error codes
- Fix a bug with releasing memory multiple times when rwbag ran out of memory.
- rwrandomizeip enhancement
- Allow the user to restrict the set of IPs that are modified via two command line arguments: --dont-change-set and --only-change-set. Both switches take an IPset; the first switch prevents the IP from being changed; the second causes only the listed IPs to be changed.
- mapsid enhancement
- The --print-classes switch will print the class(es) to which each sensor belongs.
- rwcount enhancement and changes
- Implemented the --output-path switch which directs rwcount to write its output to the specified location.
- Allow rwcount to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
- The column widths have changed slightly
- rwaddrcount enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwcut enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwstats enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwset enhancement
- Implmented the --copy-input switch as described for rwcount.
- rwtotal enhancement
- Implemented the --output-path switch as described for rwcount.
- rwuniq enhancement
- Implemented the --output-path switch as described for rwcount.
- rwsetcat bug fix
- Fix bug where the $PAGER was not being used.
- rwbagcat bug fixes
- Do not print a warning message when attempting to print an empty Bag or when the min/max limits caused no entries to be printed.
- Fix bug where the $PAGER was not being used.
- Print errors as human readable text, not error codes
- rwbagtool bug fix
- Print errors as human readable text, not error codes
- rwcat bug fix
- Modify rwcat so it will always print the SiLK header to a file, even when no records are present
- rwappend enhancement and bug fix
- New --print-statistics switch causes the number of records processed to be printed to the standard error.
- Output change: Modified rwappend so it only prints the number of records processed when --print-statistics is given.
- Fix a problem that occurred when SiLK was compiled with compression enabled by default and the applications were processing SiLK files produced by releases of SiLK prior to 0.10.5: the application would exit with the error message "Operation not permitted on compressed file" and no output would be generated.
- rwswapbytes bug fix
- See compression-related bug fix for rwappend
- rwnetmask bug fix
- See compression-related bug fix for rwappend
- Administration and configuration changes:
- New "silk.conf" file removes the requirement that sensors be defined at compile-time.
- The sensors, classes, and types are now defined at run-time through the use of a "silk.conf" text file. This file should be installed in the SILK_DATA_ROOTDIR directory.
- The run-time configuration allows a single installation of the analysis tools to query multiple data sets; simply set the SILK_DATA_ROOTDIR environment variable to the location of the data.
- The location of this file can also be specified by setting the SILK_CONFIG_FILE environment variable to its location, or by using the --site-config-file switch on most SiLK applications.
- The packer (rwflowpack) still requires certain classes and types to be defined, and it cannot use new classes and types without modifying C code. This restriction will go away in a future release.
- Major changes to the build system.
- The build system now uses all aspects of the GNU Autotools chain including automake and libtool.
- The tools can now be built with shared library support, reducing the size of the binaries and allowing the kernel to use a single copy of libsilk when multiple SiLK tools are running.
- Note that the use of shared libraries means the binaries can no longer easily be relocated; instead you should run "make install" again with the new location.
- The SiLK headers are now copied to the install target directory
- GNU make is no longer required to build the tools.
- New packing rules are used by default.
- The default site has changed from "generic" to "twoway". The twoway site allows flow records to be categorized and stored as internal-to-internal (int2int) and external-to-external (ext2ext). In addition, the "out" type is no longer everything that is not "in". The files created by the generic site are forward compatible with the twoway site; however, if you wish to continue using your current packing rules, run configure with the --enable-silk-site=generic switch. See the SiLK Installation Handbook for details.
- New transfer daemons: rwsender and rwreceiver
- These are meant to replace the direct connectivity between flowcap and rwflowpack. These daemons allow the flowcap files to be sent to multiple rwflowpack processes.
- In addition, they allow rwflowpack to process data on one system and send small files containing SiLK Flow records (called "incremental files") to another system (where the rwflowappend daemon is running) for analysis.
- New packing tool: rwflowappend
- rwflowappend appends SiLK Flow records contained in "incremental files" to hourly files.
- Changes to flowcap and rwflowpack
- The flowcap and rwflowpack tools have been modified to work with the new rwsender and rwreceiver, though they can also be used in legacy mode. With the transport removed from flowcap, flowcap files can now be sent to multiple locations.
- IPFIX flow collection enhancement
- Previous releases of SiLK (rwflowpack and flowcap) could only read IPFIX streams generated by YAF. With this release, SiLK can read flows from any IPFIX-compliant generator.
- Remove zlib requirement in rwflowpack
- Allow rwflowpack to be built even if zlib is not available. However, rwflowpack will not be able to read files of NetFlow PDUs when zlib is not present.
- New packing tool: rwpackchecker
- rwpackchecker performs a basic integrity check of a packed SiLK file.

AsItHappens is a real-time network performance monitor. AsItHappens project collects data from devices over a network and displays them on a graph, optionally storing collected data in a database for later retrieval.
Current collection types include network response, bandwidth, Cisco NBAR, and Cisco NetFlow. AsItHappens polls data in regular intervals, which can be as low as every second, to give immediate feedback on network performance.
Main features:
- Granular collection of data to the point of collecting every second
- Real-time graphing of collected data
- Response data collection via ICMP or TCP/UDP echo
- Inbound and outbound bandwidth data collection via SNMP
- Cisco NBAR (Network-based Application Recognition) Top-N collection via SNMP
- Cisco NetFlow Top-N collection via SNMP with flow grouping and match criteria options
- Optional database storage of collection sessions
- Retrieval of user-defined time intervals within a stored collection session
- Resizable graphing window with automatic scaling of graph data to fit
- Options to define how to aggregate or interpolate data when graphing e.g. to show maximums instead of averages
- The ability to add text labels to the graphing panel to explain desired areas of the graph

glFlow is a (D)DoS logger written with speed in mind. glFlow detects attacks on high speed links through real-time flow aggregation and analysis.
What do I run it on ?
It was written on FreeBSD and tested on both FreeBSD and Linux. It should work on any OS to which libpcap and OpenSSL were ported. The rest of the code is perfectly portable.
How does it work ?
Cisco Systems have defined the flow as a four value tuplet: {srcaddr, srcport, dstaddr, dstport}. The format evolved over time. The complete structures for various NetFlow versions are available on Ciscos site. Now, lets assume that the attacker floods the victim with packets that keep the same characteristics throughout the duration of the attack. No source spoof, no
source port increments or randomizations. That would lead to a very large packet rate inside that flow. glFlow calculates the average packet rate in every flow and raises an alarm signal if the threshold is hit.
What about spoofed attacks ? How are they detected ? Simple. glFlow keeps a history for every destination host that it sees. When a new flow is created, the flow counter for that host is incremented. The average number of newly created flows corresponding to a specific host in a specific amount of time is calculated, and, as above, an alarm is raised if the threshold is hit.
To prevent attacks that dont hit any of the above thresholds, theres
a new one starting with v0.1, measuring the packet rate for a destination.
Cant other tools, like SNORT, do this ?
We sincereley believe not. Remember, glFlow was written with high
speeds in mind. Weve been using it at over 500Mbps. At that speed, with an
ordinary x86 machine, even with a strong motherboard/NIC combination, you cant
do anything fancy. glFlow was specifically designed for detecting large floods
in real time, or at least something close to that.
How is it that its so fast ?
Well, Andrei did a great job implementing a very fast binary tree. That allowed us to drop the threaded model and choose a single loop design. The new results were stunning. The tests were made on a P4 Xeon/3 GHz, with an Intel GigE NIC. The average traffic rate was about 500Mbps, with an average packet rate of 100kpps. That lead to about 200k active flows. glFlow managed to clean the inactive ones in less than 0.3 seconds. There was no alarm raised
after more than 5 seconds of flooding. glFlow ate ~50% of the CPU, while consuming about 40MB of system memory.
How do I install and run it ?
Run ./configure --help. Youll see two adjustable knobs: --with-hash and --enable-debug. The first one permits you to switch between MD4 and MD5 summing of the flow and host structures kept in the memory. The second lets you run glflow in the foreground, printing some statistics on stdout.
The thresholds are harcoded in defs.h. You shouldnt have any trouble tweaking them. However, weve observed that the best results are obtained when using the same values for flow lifetime and the time between flow cleanups. And they shouldnt be much over 20. The smaller the tree is, the faster it will be cleaned.
Finally, edit your /etc/syslog.conf and write something like this: "local6.*< tabs >/var/log/something". Restart sys[k]logd afterwards.
Fire glFlow up, like this: "./glFlow < interface > < bpf filter >" and watch /var/log/something for changes. You may play with nmap or some DoS programs to test it. The IPs in the syslog will be shown as integers rather than in dotted notation. We decided to leave this job to the log analyzer.
Can it go even faster ?
Sure. There are a few methods which permit you to improve the packet capture. For more info read Luca Deris paper: http://luca.ntop.org/Ring.pdf
Enhancements:
- This is a bugfix release.

Monitoring API project is a multi-user programming interface designed to simplify the development of network monitoring software and allows users to express their monitoring needs in a device-independent way.
The main abstraction provided by MAPI is the network flow. Although flows have been used before in network monitoring systems, MAPI gives flows a first-class status. Applications that uses MAPI can specify what flows or flow statistics they are interested in by applying functions to flows.
A MAPI function can be a BPF filter, string search, packet counter or more advanced like a NetFlow generator. These function will automatically run in hardware if there is support for it on the hardware being used.
MAPI currently supports the following hardware:
- Normal NICs through libpcap
- DAG cards without co-processor
- SCAMPI adapter
Enhancements:
- This release includes support for distributed monitoring, several new MAPI functions, demo applications, and a lot of bugfixes.

IPCAD stands for IP Cisco Accounting Daemon. It runs in background, listens traffic on the specified interfaces, and records the traffic for later retrieval and analysis. IPCAD can use raw BPF devices, PCAP library, divert, tee or Linux iptables ULOG & IPQ packet sources to capture the packets.

IPCAD can export gathered information using rsh or NetFlow.


Uses BPF, libpcap divert, tee or Linux ULOG & IPQ for traffic snooping
RSH, NetFlow and console output in Cisco-like fashion
RSH access lists
Address aggregation support for RSH and NetFlow.
UDP/TCP/SCTP ports handling
Dynamic interfaces (PPP, VPN) support

Requires:
At least Berkeley packet filter or libpcap library.

Runs on FreeBSD, OpenBSD, Linux, MacOS X/Darwin, Solaris.

EHNT is a tool which turns streams of Netflow (version 5) data into something useful and human-readable. (Netflow is a UDP-based traffic reporting protocol created by Cisco, generated by Cisco, Juniper, Foundry, and other routers.)
Netflow operates in many ways. It will dump flow records in human-readable form. It will also provide reports on top ASes, IP protocols, and tcp/udp ports. The reports can be generated over various intervals, from 1 minute to 1 day.
Component programs are:
1. ehntserv listens to netflow version 5 UDP packets, and also listens for client TCP connections. When a TCP client connects, the server starts forwarding all the netflow packets it receives (plus the IP address of the originating device) to that client.
ehntserv does not currently do any IP access control. I suggest that you use ipchains or iptables on your linux box, or IP Filter (ipf) (http://coombs.anu.edu.au/ipfilter/) on your Solaris or BSD box. I dont know what the current state of packet filtering is on other Unixes; IP Filter seems to support several.
2. ehnt connects to ehntserv and displays the flows it receives in various ways. It currently has four modes (-m ):
- top mode displays average utilization by top ASes, IP protocols, or tcp/udp ports over a given interval (from 1 minute to 1 day).
Top mode is different when it focuses on a single interface on a single router, because then you get to see summaries of source and destionation for both inbound and outbound traffic. Otherwise, you just get summaries of source and destination.
- dump mode displays individual flows
- shortdump mode display individual flows in a more compact but hard
to read fashion
- colondump mode display individual flows in a machine-readable format.
And yes, I recognize that the name of this mode is unpleasant.
In all three modes, simple (REALLY simple) filtering can be done for AS
number, TCP/UDP port, IP protocol number, device sending the flow record,
and SNMP interface index.
You may think of ehnt in the three dump modes as a brain-dead and incredibly
simple tcpdump for netflow.
ehnt also has the silly and uninspiredly-named big filter, in which it
only displays flows with are bigger (in packets or bytes) than any flow
received before it. This only makes sense in the three dump modes.
Enhancements:
- Added Unix domain support for client connections, enabled by default