DB based NetFLow Collector aims to collect Cisco NetFlow data and store it to a database.

DB based NetFlow Collector has a plugin interface, which makes it flexible for fitting in particular tasks.

Whats New in This Release:

· First release. post your comments/bug reports.

OSSP flow2rrd is a companion tool to the Flow-Tools toolkit for storing NetFlow network traffic data in an accumulating fixed-size RRDTool Round-Robin-Database (RRD) for visualization purposes.

This file is piece of OSSP flow2rrd, a tool for storing NetFlow data into an RRD which can be found at http://www.ossp.org/pkg/tool/flow2rrd/.

Whats New in This Release:

· Created the initial version of OSSP flow2rrd.

JNFA project is a netflow analyzer. It uses a MySQL database to store accounting information.

Filters are used in JNFA to allow very flexible classification any kind of traffic and to store it in the different fields in a database.

IPCAD stands for IP Cisco Accounting Daemon. It runs in background, listens traffic on the specified interfaces, and records the traffic for later retrieval and analysis. IPCAD can use raw BPF devices, PCAP library, divert, tee or Linux iptables ULOG & IPQ packet sources to capture the packets.

IPCAD can export gathered information using rsh or NetFlow.


Uses BPF, libpcap divert, tee or Linux ULOG & IPQ for traffic snooping
RSH, NetFlow and console output in Cisco-like fashion
RSH access lists
Address aggregation support for RSH and NetFlow.
UDP/TCP/SCTP ports handling
Dynamic interfaces (PPP, VPN) support

Requires:
At least Berkeley packet filter or libpcap library.

Runs on FreeBSD, OpenBSD, Linux, MacOS X/Darwin, Solaris.

Softflowd project is flow-based network traffic analyser capable of Cisco NetFlow data export. Softflowd semi-statefully tracks traffic flows recorded by listening on a network interface or by reading a packet capture file.

These flows may be reported via NetFlow to a collecting host or summarised within softflowd itself.

Installation:

./configure
make
make install

Unfortunately some systems like to make life complicated. Things work fine on the systems that I develop and test on (OpenBSD and Linux). There is peliminary support for Solaris 9 (i.e. it compiled), but no testing on this platform has been performed.

Whats New in This Release:

· Manual sending of NetFlow v.9 template refreshes is now supported.
· ICMP type/code is now encoded into port numbers, similar to some Cisco products.
· Lower-resolution tracking of flows (ignoring port or protocol information) was implemented for high-bandwidth users.
· Several small but important bugfixes were made.

bbnfc is useful for debugging netflow exports as produced by Cisco, Juniper, etc. routers. This simple program sits on a user-specified UDP port and displays to stdout all netflow exports that are sent to the machine. Complete program is 300 lines long (C source).

The source should compile under most flavours of UNIX, with little or no modification.
Note: Some web browsers will tend to display .tgz archives on the screen rather than saving them to a file. You may need to hold down the shift key when selecting the link below (or possibly some other brain-dead combination).


In this archive you will see the following files:
readme.bbnfc - This readme file
bbnfc.c - Source code for bbnfc
netflow.h - Header file
bbnfc - Pre-compiled binary for Linux (Intel)

To compile from source the following (or some variation thereof) should work:
gcc -o bbnfc bbnfc.c

Basic usage instructions can be obtained with
./bbnfc -h

Stager is a system for aggregating and presenting network statistics. Stager project is generic and can be customized to present and process any kind of network statistics.

The backend collects data and stores reports in a database, automatically handling the aggregation of hourly statistics into days, weeks, and months.

The Web frontend presents data in tables, matrices, or plots. The reports are fully customizable, and their definitions are stored in the database.

Installation:

If you are upgrading from a previous version of Stager, see the UPGRADE file.

This version of Stager only support Netflow. Future releases will also support roundtrip measurements, SNMP and various passive monitoring measurements.

To run Stager you now need PHP4 CLI with SNMP support. Under debian you can just install php4-cli and php4-snmp

Backend:

1. Create a new Postgresql user that is allowed to create new databases
2. Create a new user that runs the Stager backend. The rest of the installation should be done as this user
3. /stager-install.pl --type=backend --backends=netflow --path=/installation/path/
4. Edit /installation/path/etc/netflow.cfg and /installation/path/etc/getRouterInfo.cfg
5. cd /installation/path/bin
6. ./db_install.pl --backend=netflow
7. Default access control is to give full access to admin user and limited access to guest users. Check /installation/path/lib/getRouterInfo.custom.php if you want to change this.
8. ./getRouterInfo.php -v -v -o
9. ./get-netflow.pl -v --no-missing --delete-old Check for error messages
10. Edit /installation/path/bin/getRouterInfo.sh
11. Edit crontab:
30 0-23/2 * * * $HOME/stager/bin/getRouterInfo.sh -v //How often you run this command depends on how dynamic your network is
45 * * * * $HOME/stager/bin/get-netflow.pl --delete-old
50 1 * * * $HOME/stager/bin/aggregate.pl --backend=netflow --interval=1 day --no-distributed --manual-mode
45 2 * * 1 $HOME/stager/bin/aggregate.pl --backend=netflow --interval=1 week --timeformat YYYY-IW --no-cap --no-distributed
50 2 1 * * $HOME/stager/bin/aggregate.pl --backend=netflow --interval=1 month --timeformat YYYY-MM --no-distributed

Frontend:

1. ./stager-install.pl --type=frontend --path=/installation/path
2. edit /installation/path/config/user.config.php

Whats New in 2.0.1 Stable Release:

· This release fixes several bugs in both the frontend and backend.
· The most important fix is that bookmarks with relative times are now working properly.
· This means that if you create a report that shows data, for example, for the last hour with available data, you can now create a bookmark that will always show data for the last hour with available data.

Whats New in 3.0 Beta 1 Development Release:

· The backend has been completely rewritten and is now implemented in PHP.
· The new backend is faster and more generic so that it is easier to add support for other types of statistics.
· The new backends are also more robust against database down time.
· In the web GUI, it is now possible to right click on most data to bring up a context menu where it is possible to add custom links.

fprobe is a small NetFlow probe which will listen on a network interface. It isusing libpcap, aggregate the traffic and export NetFlow V5 datagram to a remote collector for processing. A flow is identified by ip protocol, source ip, source port, destination ip, destination port.
Right now only ethernet interfaces are supported. Support for more media types (tunnel, ppp etc) will be added in nex versions.

/fprobe -t IP:PORT [ -i interface ] [ -s scan ] [ expression ]
-t IP:PORT NetFlow collector address
-i interface interface to listen for traffic (default eth0)
-s scan interval in seconds between two flow tables scans (Default: 10)
-c file file with MAC definitions
-p dont put the interface in promisc mode
-b go in background (daemon mode)
-l file log file name
expression a bpf expresion to filter traffic (See libpcap/tcpdump)

For example:
./fprobe -i eth2 -t 127.0.0.1:8182

This will sniff the traffic on interface eth2 and will send the NetFlow data to localhost (127.0.0.1) on UDP port 8182.

Internal flow table is parsed every scan seconds for expired flows which are sent to remote collector.
Whats New in This Release:

· can handle IP fragments
· can set the snmp interface ID based on source/destination MAC address
· fixed uptime in exported flows
· new hash function for internal storage
· delay between udp datagrams emited

flowd application is a fast, small and secure NetFlow collector.

Here are some key features of "flowd":

· Understands NetFlow protocol v.1, v.5, v.7 and v.9 (including IPv6 flows)
· Supports both IPv4 and IPv6 transport of flows
· Secure: flowd is privilege separated to limit the impact of any compromise
· Supports filtering and tagging of flows, using a packet filter-like syntax
· Stores recorded flow data in a compact binary format which supports run-time choice over which flow fields are stored
· Ships with both Perl and Python interfaces for reading and parsing the on-disk record format
· Is licensed under a liberal BSD-like license
· Supports reception of flow export datagrams sent to multicast groups (IPv4 and IPv6), thereby allowing the construction of redundant flow collector systems

flowd works with any standard NetFlow exporter, including hardware devices (e.g. routers) or software flow tracking agents, such as my own softflowd and pfflowd. Please refer to the README for more information.

The flowd daemon follows the Unix philosophy of "doing one thing well" - it doesnt try to do anything beyond accepting NetFlow packets and storing them in a standard format on disk. In particular, it does not include support for storing flows in multiple formats or performing data analysis. That sort of thing is left to external tools. The source distribution includes several example tools including a basic reporting script and one to store flows in a SQL database.

Whats New in This Release:

· This release includes major improvements to performance and functionality.
· In particular, the flow format has been modified to store more information and be faster to read, input and output buffering has been improved, new flow filtering options have been added, and the Python API has been rewritten and extended to be many times faster.

cflowd is a flow analysis tool currently used for analyzing Ciscos NetFlow enabled switching method.

The current release (described below) includes the collections, storage, and basic analysis modules for cflowd and for arts++ libraries. This analysis package permits data collection and analysis by ISPs and network engineers in support of capacity planning, trends analysis, and characterization of workloads in a network service provider environment. Other areas where cflowd may prove useful are: tracking for Web hosting, accounting and billing, network planning and analysis, network monitoring, developing user profiles, data warehousing and mining, as well as security-related investigations.


cflowd is no longer supported by CAIDA. Instead, please consider the use of flow-tools, which will provide a toolset for working with NetFlow data. flow-tools can also be used (like cflowd) in conjunction with FlowScan, maintained by Dave Plonka at the University of Wisconsin, Madison.

EHNT is a tool which turns streams of Netflow (version 5) data into something useful and human-readable. (Netflow is a UDP-based traffic reporting protocol created by Cisco, generated by Cisco, Juniper, Foundry, and other routers.)

Netflow operates in many ways. It will dump flow records in human-readable form. It will also provide reports on top ASes, IP protocols, and tcp/udp ports. The reports can be generated over various intervals, from 1 minute to 1 day.

Component programs are:

1. ehntserv listens to netflow version 5 UDP packets, and also listens for client TCP connections. When a TCP client connects, the server starts forwarding all the netflow packets it receives (plus the IP address of the originating device) to that client.

ehntserv does not currently do any IP access control. I suggest that you use ipchains or iptables on your linux box, or IP Filter (ipf) (http://coombs.anu.edu.au/ipfilter/) on your Solaris or BSD box. I dont know what the current state of packet filtering is on other Unixes; IP Filter seems to support several.

2. ehnt connects to ehntserv and displays the flows it receives in various ways. It currently has four modes (-m ):

- top mode displays average utilization by top ASes, IP protocols, or tcp/udp ports over a given interval (from 1 minute to 1 day).

Top mode is different when it focuses on a single interface on a single router, because then you get to see summaries of source and destionation for both inbound and outbound traffic. Otherwise, you just get summaries of source and destination.
- dump mode displays individual flows
- shortdump mode display individual flows in a more compact but hard
to read fashion
- colondump mode display individual flows in a machine-readable format.
And yes, I recognize that the name of this mode is unpleasant.

In all three modes, simple (REALLY simple) filtering can be done for AS
number, TCP/UDP port, IP protocol number, device sending the flow record,
and SNMP interface index.

You may think of ehnt in the three dump modes as a brain-dead and incredibly
simple tcpdump for netflow.

ehnt also has the silly and uninspiredly-named big filter, in which it
only displays flows with are bigger (in packets or bytes) than any flow
received before it. This only makes sense in the three dump modes.

Whats New in This Release:

· Added Unix domain support for client connections, enabled by default

AsItHappens is a real-time network performance monitor. AsItHappens project collects data from devices over a network and displays them on a graph, optionally storing collected data in a database for later retrieval.

Current collection types include network response, bandwidth, Cisco NBAR, and Cisco NetFlow. AsItHappens polls data in regular intervals, which can be as low as every second, to give immediate feedback on network performance.

Here are some key features of "AsItHappens":

· Granular collection of data to the point of collecting every second
· Real-time graphing of collected data
· Response data collection via ICMP or TCP/UDP echo
· Inbound and outbound bandwidth data collection via SNMP
· Cisco NBAR (Network-based Application Recognition) Top-N collection via SNMP
· Cisco NetFlow Top-N collection via SNMP with flow grouping and match criteria options
· Optional database storage of collection sessions
· Retrieval of user-defined time intervals within a stored collection session
· Resizable graphing window with automatic scaling of graph data to fit
· Options to define how to aggregate or interpolate data when graphing e.g. to show maximums instead of averages
· The ability to add text labels to the graphing panel to explain desired areas of the graph

flow-tools is a set of programs for processing and managing NetFlow exports from Cisco and Juniper routers. The tools included are: flow-capture, flow-cat, flow-dscan, flow-expire, flow-export, flow-fanout, flow-filter, flow-gen, flow-header, flow-import, flow-mask, flow-merge, flow-nfilter, flow-print, flow-receive, flow-report, flow-send, flow-split, flow-stat, flow-tag, and flow-xlate.

Flow data is collected and stored by default in host byte ordera nd the files are portable across every endian architectures.

Commands that utilize the network use a localip/remoteip/port designation for communication. "localip" is the IP address the host will use as a source for sending or bind to when receiving NetFlow PDUs (ie the destination address of the exporter. Configuring the "localip" to 0 will force the kernel to decide what IP address to use for sending and listen on all IP addresses for receiving. "remoteip" is the destination IP address used for sending or the expected address of the source when receiving. If the "remoteip" is 0 then the application will accept flows from any source address. The "port" is the UDP port number used for sending or receiving. When using multicast addresses the localip/remoteip/port is used to represent the source, group, and port respectively.

Flows are exported from a router in a number of different configurable versions. A flow is a collection of key fields and additional data. The flow key is {srcaddr, dstaddr, input, output, srcport, dstport, prot, ToS}. Flow-tools supports one export version per file.

Export versions 1, 5, 6, and 7 all maintain {nexthop, dPkts, dOctets, First, Last, flags}, ie the next-hop IP address, number of packets, number of octets (bytes), start time, end time, and flags such as the TCP header bits. Version 5 adds the additional fields {src_as, dst_as, src_mask, dst_mask}, ie source AS, destination AS, source network mask, and destination network mask. Version 7 which is specific to the Catalyst switches adds in addition to the version 5 fields {router_sc}, which is the Router IP address which populates the flow cache shortcut in the Supervisor. Version 6 which is not officially supported by Cisco adds in addition to the version 5 fields {in_encaps, out_encaps, peer_nexthop}, ie the input and output interface encapsulation size, and the IP address of the next hop within the peer. Version 1 exports do not contain a sequence number and therefore should be avoided, although it is safe to store the data as version 1 if the additional fields are not used.

Version 8 IOS NetFlow is a second level flow cache that reduces the data exported from the router. There are currently 11 formats, all of which provide {dFlows, dOctets, dPkts, First, Last} for the key fields.

8.1 - Source and Destination AS, Input and Output interface
8.2 - Protocol and Port
8.3 - Source Prefix and Input interface
8.4 - Destination Prefix and Output interface
8.5 - Source/Destination Prefix and Input/Output interface
8.9 - 8.1 + ToS
8.10 - 8.2 + ToS
8.11 - 8.3 + ToS
8.12 - 8.5 + ToS
8.13 - 8.2 + ToS
8.14 - 8.3 + ports + ToS

Version 8 CatIOS NetFlow appears to be a less fine grained first level flow cache.

8.6 - Destination IP, ToS, Marked ToS,
8.7 - Source/Destination IP, Input/Output interface, ToS, Marked ToS,
8.8 - Source/Destination IP, Source/Destination Port,
Input/Output interface, ToS, Marked ToS,

The following programs are included in the flow-tools distribution.

flow-capture - Collect, compress, store, and manage disk space for exported flows from a router.
flow-cat - Concatenate flow files. Typically flow files will contain a small window of 5 or 15 minutes of exports. Flow-cat can be used to append files for generating reports that span longer time periods.
flow-fanout - Replicate NetFlow datagrams to unicast or multicast destinations. Flow-fanout is used to facilitate multiple collectors attached to a single router.
flow-report - Generate reports for NetFlow data sets. Reports include source/destination IP pairs, source/destination AS, and top talkers. Over 50 reports are currently supported.
flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to group flows by customer network. The tags can later be used with flow-fanout or flow-report to generate customer based traffic reports.
flow-filter - Filter flows based on any of the export fields. Flow-filter is used in-line with other programs to generate reports based on flows matching filter expressions.
flow-import - Import data from ASCII or cflowd format.
flow-export - Export data to ASCII or cflowd format.
flow-send - Send data over the network using the NetFlow protocol.
flow-receive - Receive exports using the NetFlow protocol without storing to disk like flow-capture.
flow-gen - Generate test data.
flow-dscan - Simple tool for detecting some types of network scanning and Denial of Service attacks.
flow-merge - Merge flow files in chronoligical order.
flow-xlate - Perform translations on some flow fields.
flow-expire - Expire flows using the same policy of flow-capture.
flow-header - Display meta information in flow file.
flow-split - Split flow files into smaller files based on size, time, or tags.

Monitoring API project is a multi-user programming interface designed to simplify the development of network monitoring software and allows users to express their monitoring needs in a device-independent way.

The main abstraction provided by MAPI is the network flow. Although flows have been used before in network monitoring systems, MAPI gives flows a first-class status. Applications that uses MAPI can specify what flows or flow statistics they are interested in by applying functions to flows.

A MAPI function can be a BPF filter, string search, packet counter or more advanced like a NetFlow generator. These function will automatically run in hardware if there is support for it on the hardware being used.

MAPI currently supports the following hardware:

- Normal NICs through libpcap
- DAG cards without co-processor
- SCAMPI adapter

Whats New in This Release:

· This release includes support for distributed monitoring, several new MAPI functions, demo applications, and a lot of bugfixes.

Panoptis plans to create a network security tool (N-IDS) to detect and block DoS and DDoS attacks. The programming language is C++, and the input is being provided by routers.

First, you need a router that exports NetFlow(TM) data. Versions 1, 5 and 8 are supported, although version 8 has not been tested AT ALL. You also need a server for accepting data and processing it.


In order to compile the software you need a C++ compiler (tested only with g++ for the time being) and the CommonC++ library, found at http://www.gnu.org/software/commonc++/CommonC++.html At the moment the software has been linked against and tested with commoncpp2-1.0.9

YOU WILL ALSO NEED g++ VERSION 3.x!!! This is very important! Compiling with g++ 2.95.x or earlier versions causes segmantation faults in some cases. This has to do with CommonC++, not Panoptis.

Before you can use the software, you must also have read SNMP access to your router. That is only needed by the speeds.py script that collects some initial information (the .py extention should already make you think youll need the Python programming language installed -- thats true.

Whats New in This Release:

· Update so that Panoptis compiles and runs on newer systems (GCC 3.3.5, CommonC++2 1.5.3).
· No new features, unfortunately.