Free netflow collector software for linux

It is a package useful to collect statistics from remote nodes. That includes standard server metrics and any other numerical quantity we can use to characterize the node state (as the number of users or processes).

It is built under a distributed agent-manager model. RRDtool is used as database, so representation capabilities are bundled.

Although rrdUtils are not a requisite, they are used in some auxiliar tasks and using both tools is highly recommended.

Installation:

Manager side

gunzip -c rrdUtils-3.1.tar.gz | tar -xf -
cd rrdUtils
./configure --with-rrddir=/stats
make install

gunzip -c stats-station-3.0.tar.gz | tar -xf -
cd stats-station
./configure --with-statsdir=/stats --with-port=666
make install

cd rrdConf
./install_confs.sh
./haz_index.sh -g

The -g flag creates graphs with the html pages, and reduces the number of error messages that haz_index will issue.
The only task that needs to be completed at this point is the periodic graphs generation by a crontab entry that calls the make_graphs.sh script.
Once we start the server using the provided initscript, we get the manager side ready.

Agent side

gunzip -c stats-3.0.tar.gz | tar -xf -
cd stats
./configure --with-host=central.node --with-port=666
--with-statsdir=/usr/local/stats
make install

With a crontab entry to execute the statlaunch script every 15 minutes, the whole system is up and running.

libGarbageCollector is an incremental garbage collector with a tri-color, Baker treadmill, write-barrier implementation.

libGarbageCollector is built from the garbage collector code used in the Io programming language project.

stic bundles a few Linux tools which are intended to support the task of collecting an unreasonable amount of pictures (preferrably in JPEG format).
similar
a program for detecting duplicate or very similar images. It maintains a
database of characteristic color samples which it compares with submitted
pictures. similar depends on libjpeg and ImageMagick s convert (on a
modern Linux desktop system these components should already be present).
Storage medium may be a usual filesystem or a MySQL database. There
also is a MySQL UDF extension to compare image samples within SQL queries.
similar contains the communications module described at sagent.
simv
a core program to perform file management tasks on an image collection.
Its main purpose is to coordinate file movements with the content of
similars database. This applies to importing new files which get tested
against the existent collection, as well as to inform similar about moving
and deleting files within the collection.
simv depends on an external image viewer like ImageMagick s display
(should already be present on a modern Linux desktop system) or John
Bradleys xv (quite a fast one).
simv contains the communications module described at sagent.
sagent
a standalone version of the communications module used in simv and
similar. This software receives input from its start terminal and multiple
clients, distributes several types of output back to them, and is also
able to act itself as such a client.
Since communications mainly use TCP/IP there is an encryption layer
(Blowfish with 128 bit keys) which provides user authentication. Any
single activity of such a user may be particularly permitted or denied.
Secure connections should be possible that way as long as one can defend
the keyfiles and programs on client and server host against foreign
access.
Front-end connection software is available in C, Tcl/Tk and PHP3 to build
custom clients. In the most primitive case even telnet can act as a
client.
The standalone program sagent may be used as communications node in a tree
of clients. Another purpose is to be a shell frontend which sends commands
to a server and receives its replies.
snntpbatch
a command line based NNTP (newsgroups) client. It is mainly intended for
automatic download of images by use of a filter language. Nevertheless it
also downloads the message texts and converts them to HTML code which
includes the downloaded images. Also, it is capable of automatically
posting sets of images to the newsgroups.
The tools are designed to be very independent of the system flavor. On an
average Linux desktop there should be no need to update existing system
components. Actually one could use stic without having display equipment for
graphics.
Any program activity which is possible in dialog may also be performed in
batch runs. Therefore the tools are quite suitable for users who like to get
boring tasks automated and manual tasks simplified.
All tools code is open source and distributed under BSD license.
Example images Credit: U. S. Fish and Wildlife Service (see images/CREDITS)
Enhancements:
- The new encrypted protocol version 0.2 is standard now : SHA-1 seal, 256 bit keys, variable chaining initialization vector.
- The protocol of a connection is chosen by the client and may or may not be accepted by the server.
- See sagent command -security options clientprotocol , serverprotocol.

genlogstatcoll is a generic collector to collect statistic data from log file delivered in via a syslog interface and provided via the AgentX interface through a SNMP daemon.

A syslog daemon (for instance syslog-ng) has to be configured to send relevant data (for instance the messages on the facility MAIL) to the genlogstatcoll too.

Considering the genlogstatcoll input interface is configured on 172.16.1.12, port 9514, this config line for syslog-ng will do the trick:

destination maillog { file("/var/log/mail/$YEAR/$MONTH/$DAY/mail.log"
create_dirs(yes) dir_perm(0755) owner("root") group("mail")
perm(0640)); udp("172.16.1.12" port(9514)); };

An SNMP daemon (for instance net-snmp 5.2.3) has to be configured to open the AgentX interface.

Considering the SNMP daemon should open the interface on 172.16.1.12, port 9161, these line in the snmpd.conf will do it:

master agentx
AgentXSocket 172.16.1.12:9161

Now, you can configure a facility to filter the input data, a SNMP oid to mount on and a couple of regexes, to count their matches in the log data.

flowd application is a fast, small and secure NetFlow collector.
Main features:
- Understands NetFlow protocol v.1, v.5, v.7 and v.9 (including IPv6 flows)
- Supports both IPv4 and IPv6 transport of flows
- Secure: flowd is privilege separated to limit the impact of any compromise
- Supports filtering and tagging of flows, using a packet filter-like syntax
- Stores recorded flow data in a compact binary format which supports run-time choice over which flow fields are stored
- Ships with both Perl and Python interfaces for reading and parsing the on-disk record format
- Is licensed under a liberal BSD-like license
- Supports reception of flow export datagrams sent to multicast groups (IPv4 and IPv6), thereby allowing the construction of redundant flow collector systems
flowd works with any standard NetFlow exporter, including hardware devices (e.g. routers) or software flow tracking agents, such as my own softflowd and pfflowd. Please refer to the README for more information.
The flowd daemon follows the Unix philosophy of "doing one thing well" - it doesnt try to do anything beyond accepting NetFlow packets and storing them in a standard format on disk. In particular, it does not include support for storing flows in multiple formats or performing data analysis. That sort of thing is left to external tools. The source distribution includes several example tools including a basic reporting script and one to store flows in a SQL database.
Enhancements:
- This release includes major improvements to performance and functionality.
- In particular, the flow format has been modified to store more information and be faster to read, input and output buffering has been improved, new flow filtering options have been added, and the Python API has been rewritten and extended to be many times faster.

flow-tools is a set of programs for processing and managing NetFlow exports from Cisco and Juniper routers. The tools included are: flow-capture, flow-cat, flow-dscan, flow-expire, flow-export, flow-fanout, flow-filter, flow-gen, flow-header, flow-import, flow-mask, flow-merge, flow-nfilter, flow-print, flow-receive, flow-report, flow-send, flow-split, flow-stat, flow-tag, and flow-xlate.

Flow data is collected and stored by default in host byte ordera nd the files are portable across every endian architectures.

Commands that utilize the network use a localip/remoteip/port designation for communication. "localip" is the IP address the host will use as a source for sending or bind to when receiving NetFlow PDUs (ie the destination address of the exporter. Configuring the "localip" to 0 will force the kernel to decide what IP address to use for sending and listen on all IP addresses for receiving. "remoteip" is the destination IP address used for sending or the expected address of the source when receiving. If the "remoteip" is 0 then the application will accept flows from any source address. The "port" is the UDP port number used for sending or receiving. When using multicast addresses the localip/remoteip/port is used to represent the source, group, and port respectively.

Flows are exported from a router in a number of different configurable versions. A flow is a collection of key fields and additional data. The flow key is {srcaddr, dstaddr, input, output, srcport, dstport, prot, ToS}. Flow-tools supports one export version per file.

Export versions 1, 5, 6, and 7 all maintain {nexthop, dPkts, dOctets, First, Last, flags}, ie the next-hop IP address, number of packets, number of octets (bytes), start time, end time, and flags such as the TCP header bits. Version 5 adds the additional fields {src_as, dst_as, src_mask, dst_mask}, ie source AS, destination AS, source network mask, and destination network mask. Version 7 which is specific to the Catalyst switches adds in addition to the version 5 fields {router_sc}, which is the Router IP address which populates the flow cache shortcut in the Supervisor. Version 6 which is not officially supported by Cisco adds in addition to the version 5 fields {in_encaps, out_encaps, peer_nexthop}, ie the input and output interface encapsulation size, and the IP address of the next hop within the peer. Version 1 exports do not contain a sequence number and therefore should be avoided, although it is safe to store the data as version 1 if the additional fields are not used.

Version 8 IOS NetFlow is a second level flow cache that reduces the data exported from the router. There are currently 11 formats, all of which provide {dFlows, dOctets, dPkts, First, Last} for the key fields.

8.1 - Source and Destination AS, Input and Output interface
8.2 - Protocol and Port
8.3 - Source Prefix and Input interface
8.4 - Destination Prefix and Output interface
8.5 - Source/Destination Prefix and Input/Output interface
8.9 - 8.1 + ToS
8.10 - 8.2 + ToS
8.11 - 8.3 + ToS
8.12 - 8.5 + ToS
8.13 - 8.2 + ToS
8.14 - 8.3 + ports + ToS

Version 8 CatIOS NetFlow appears to be a less fine grained first level flow cache.

8.6 - Destination IP, ToS, Marked ToS,
8.7 - Source/Destination IP, Input/Output interface, ToS, Marked ToS,
8.8 - Source/Destination IP, Source/Destination Port,
Input/Output interface, ToS, Marked ToS,

The following programs are included in the flow-tools distribution.

flow-capture - Collect, compress, store, and manage disk space for exported flows from a router.
flow-cat - Concatenate flow files. Typically flow files will contain a small window of 5 or 15 minutes of exports. Flow-cat can be used to append files for generating reports that span longer time periods.
flow-fanout - Replicate NetFlow datagrams to unicast or multicast destinations. Flow-fanout is used to facilitate multiple collectors attached to a single router.
flow-report - Generate reports for NetFlow data sets. Reports include source/destination IP pairs, source/destination AS, and top talkers. Over 50 reports are currently supported.
flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to group flows by customer network. The tags can later be used with flow-fanout or flow-report to generate customer based traffic reports.
flow-filter - Filter flows based on any of the export fields. Flow-filter is used in-line with other programs to generate reports based on flows matching filter expressions.
flow-import - Import data from ASCII or cflowd format.
flow-export - Export data to ASCII or cflowd format.
flow-send - Send data over the network using the NetFlow protocol.
flow-receive - Receive exports using the NetFlow protocol without storing to disk like flow-capture.
flow-gen - Generate test data.
flow-dscan - Simple tool for detecting some types of network scanning and Denial of Service attacks.
flow-merge - Merge flow files in chronoligical order.
flow-xlate - Perform translations on some flow fields.
flow-expire - Expire flows using the same policy of flow-capture.
flow-header - Display meta information in flow file.
flow-split - Split flow files into smaller files based on size, time, or tags.

cflowd is a flow analysis tool currently used for analyzing Ciscos NetFlow enabled switching method.

The current release (described below) includes the collections, storage, and basic analysis modules for cflowd and for arts++ libraries. This analysis package permits data collection and analysis by ISPs and network engineers in support of capacity planning, trends analysis, and characterization of workloads in a network service provider environment. Other areas where cflowd may prove useful are: tracking for Web hosting, accounting and billing, network planning and analysis, network monitoring, developing user profiles, data warehousing and mining, as well as security-related investigations.


cflowd is no longer supported by CAIDA. Instead, please consider the use of flow-tools, which will provide a toolset for working with NetFlow data. flow-tools can also be used (like cflowd) in conjunction with FlowScan, maintained by Dave Plonka at the University of Wisconsin, Madison.