Free flows software for linux

flowd application is a fast, small and secure NetFlow collector.
Main features:
- Understands NetFlow protocol v.1, v.5, v.7 and v.9 (including IPv6 flows)
- Supports both IPv4 and IPv6 transport of flows
- Secure: flowd is privilege separated to limit the impact of any compromise
- Supports filtering and tagging of flows, using a packet filter-like syntax
- Stores recorded flow data in a compact binary format which supports run-time choice over which flow fields are stored
- Ships with both Perl and Python interfaces for reading and parsing the on-disk record format
- Is licensed under a liberal BSD-like license
- Supports reception of flow export datagrams sent to multicast groups (IPv4 and IPv6), thereby allowing the construction of redundant flow collector systems
flowd works with any standard NetFlow exporter, including hardware devices (e.g. routers) or software flow tracking agents, such as my own softflowd and pfflowd. Please refer to the README for more information.
The flowd daemon follows the Unix philosophy of "doing one thing well" - it doesnt try to do anything beyond accepting NetFlow packets and storing them in a standard format on disk. In particular, it does not include support for storing flows in multiple formats or performing data analysis. That sort of thing is left to external tools. The source distribution includes several example tools including a basic reporting script and one to store flows in a SQL database.
Enhancements:
- This release includes major improvements to performance and functionality.
- In particular, the flow format has been modified to store more information and be faster to read, input and output buffering has been improved, new flow filtering options have been added, and the Python API has been rewritten and extended to be many times faster.

flow-tools is a set of programs for processing and managing NetFlow exports from Cisco and Juniper routers. The tools included are: flow-capture, flow-cat, flow-dscan, flow-expire, flow-export, flow-fanout, flow-filter, flow-gen, flow-header, flow-import, flow-mask, flow-merge, flow-nfilter, flow-print, flow-receive, flow-report, flow-send, flow-split, flow-stat, flow-tag, and flow-xlate.

Flow data is collected and stored by default in host byte ordera nd the files are portable across every endian architectures.

Commands that utilize the network use a localip/remoteip/port designation for communication. "localip" is the IP address the host will use as a source for sending or bind to when receiving NetFlow PDUs (ie the destination address of the exporter. Configuring the "localip" to 0 will force the kernel to decide what IP address to use for sending and listen on all IP addresses for receiving. "remoteip" is the destination IP address used for sending or the expected address of the source when receiving. If the "remoteip" is 0 then the application will accept flows from any source address. The "port" is the UDP port number used for sending or receiving. When using multicast addresses the localip/remoteip/port is used to represent the source, group, and port respectively.

Flows are exported from a router in a number of different configurable versions. A flow is a collection of key fields and additional data. The flow key is {srcaddr, dstaddr, input, output, srcport, dstport, prot, ToS}. Flow-tools supports one export version per file.

Export versions 1, 5, 6, and 7 all maintain {nexthop, dPkts, dOctets, First, Last, flags}, ie the next-hop IP address, number of packets, number of octets (bytes), start time, end time, and flags such as the TCP header bits. Version 5 adds the additional fields {src_as, dst_as, src_mask, dst_mask}, ie source AS, destination AS, source network mask, and destination network mask. Version 7 which is specific to the Catalyst switches adds in addition to the version 5 fields {router_sc}, which is the Router IP address which populates the flow cache shortcut in the Supervisor. Version 6 which is not officially supported by Cisco adds in addition to the version 5 fields {in_encaps, out_encaps, peer_nexthop}, ie the input and output interface encapsulation size, and the IP address of the next hop within the peer. Version 1 exports do not contain a sequence number and therefore should be avoided, although it is safe to store the data as version 1 if the additional fields are not used.

Version 8 IOS NetFlow is a second level flow cache that reduces the data exported from the router. There are currently 11 formats, all of which provide {dFlows, dOctets, dPkts, First, Last} for the key fields.

8.1 - Source and Destination AS, Input and Output interface
8.2 - Protocol and Port
8.3 - Source Prefix and Input interface
8.4 - Destination Prefix and Output interface
8.5 - Source/Destination Prefix and Input/Output interface
8.9 - 8.1 + ToS
8.10 - 8.2 + ToS
8.11 - 8.3 + ToS
8.12 - 8.5 + ToS
8.13 - 8.2 + ToS
8.14 - 8.3 + ports + ToS

Version 8 CatIOS NetFlow appears to be a less fine grained first level flow cache.

8.6 - Destination IP, ToS, Marked ToS,
8.7 - Source/Destination IP, Input/Output interface, ToS, Marked ToS,
8.8 - Source/Destination IP, Source/Destination Port,
Input/Output interface, ToS, Marked ToS,

The following programs are included in the flow-tools distribution.

flow-capture - Collect, compress, store, and manage disk space for exported flows from a router.
flow-cat - Concatenate flow files. Typically flow files will contain a small window of 5 or 15 minutes of exports. Flow-cat can be used to append files for generating reports that span longer time periods.
flow-fanout - Replicate NetFlow datagrams to unicast or multicast destinations. Flow-fanout is used to facilitate multiple collectors attached to a single router.
flow-report - Generate reports for NetFlow data sets. Reports include source/destination IP pairs, source/destination AS, and top talkers. Over 50 reports are currently supported.
flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to group flows by customer network. The tags can later be used with flow-fanout or flow-report to generate customer based traffic reports.
flow-filter - Filter flows based on any of the export fields. Flow-filter is used in-line with other programs to generate reports based on flows matching filter expressions.
flow-import - Import data from ASCII or cflowd format.
flow-export - Export data to ASCII or cflowd format.
flow-send - Send data over the network using the NetFlow protocol.
flow-receive - Receive exports using the NetFlow protocol without storing to disk like flow-capture.
flow-gen - Generate test data.
flow-dscan - Simple tool for detecting some types of network scanning and Denial of Service attacks.
flow-merge - Merge flow files in chronoligical order.
flow-xlate - Perform translations on some flow fields.
flow-expire - Expire flows using the same policy of flow-capture.
flow-header - Display meta information in flow file.
flow-split - Split flow files into smaller files based on size, time, or tags.

glFlow is a (D)DoS logger written with speed in mind. glFlow detects attacks on high speed links through real-time flow aggregation and analysis.
What do I run it on ?
It was written on FreeBSD and tested on both FreeBSD and Linux. It should work on any OS to which libpcap and OpenSSL were ported. The rest of the code is perfectly portable.
How does it work ?
Cisco Systems have defined the flow as a four value tuplet: {srcaddr, srcport, dstaddr, dstport}. The format evolved over time. The complete structures for various NetFlow versions are available on Ciscos site. Now, lets assume that the attacker floods the victim with packets that keep the same characteristics throughout the duration of the attack. No source spoof, no
source port increments or randomizations. That would lead to a very large packet rate inside that flow. glFlow calculates the average packet rate in every flow and raises an alarm signal if the threshold is hit.
What about spoofed attacks ? How are they detected ? Simple. glFlow keeps a history for every destination host that it sees. When a new flow is created, the flow counter for that host is incremented. The average number of newly created flows corresponding to a specific host in a specific amount of time is calculated, and, as above, an alarm is raised if the threshold is hit.
To prevent attacks that dont hit any of the above thresholds, theres
a new one starting with v0.1, measuring the packet rate for a destination.
Cant other tools, like SNORT, do this ?
We sincereley believe not. Remember, glFlow was written with high
speeds in mind. Weve been using it at over 500Mbps. At that speed, with an
ordinary x86 machine, even with a strong motherboard/NIC combination, you cant
do anything fancy. glFlow was specifically designed for detecting large floods
in real time, or at least something close to that.
How is it that its so fast ?
Well, Andrei did a great job implementing a very fast binary tree. That allowed us to drop the threaded model and choose a single loop design. The new results were stunning. The tests were made on a P4 Xeon/3 GHz, with an Intel GigE NIC. The average traffic rate was about 500Mbps, with an average packet rate of 100kpps. That lead to about 200k active flows. glFlow managed to clean the inactive ones in less than 0.3 seconds. There was no alarm raised
after more than 5 seconds of flooding. glFlow ate ~50% of the CPU, while consuming about 40MB of system memory.
How do I install and run it ?
Run ./configure --help. Youll see two adjustable knobs: --with-hash and --enable-debug. The first one permits you to switch between MD4 and MD5 summing of the flow and host structures kept in the memory. The second lets you run glflow in the foreground, printing some statistics on stdout.
The thresholds are harcoded in defs.h. You shouldnt have any trouble tweaking them. However, weve observed that the best results are obtained when using the same values for flow lifetime and the time between flow cleanups. And they shouldnt be much over 20. The smaller the tree is, the faster it will be cleaned.
Finally, edit your /etc/syslog.conf and write something like this: "local6.*< tabs >/var/log/something". Restart sys[k]logd afterwards.
Fire glFlow up, like this: "./glFlow < interface > < bpf filter >" and watch /var/log/something for changes. You may play with nmap or some DoS programs to test it. The IPs in the syslog will be shown as integers rather than in dotted notation. We decided to leave this job to the log analyzer.
Can it go even faster ?
Sure. There are a few methods which permit you to improve the packet capture. For more info read Luca Deris paper: http://luca.ntop.org/Ring.pdf
Enhancements:
- This is a bugfix release.

Akarmi - Flow is a template-based C++ framework. Akarmi - Flow simplifies creating programs from independent processing elements that are connected by event channels.

It is currently being rewritten to use terms similar to CORBAs event service.

Installation:

cmake .
make all
make install

[testing: optional]
cd test
make test

Gerris project is an Open Source Free Software library for the solution of the partial differential equations describing fluid flow.
Gerris is supported by NIWA (National Institute of Water and Atmospheric research) and by the Marsden Fund of the Royal Society of New Zealand.
The code is written entirely in C and uses both the GLib Library and the GTS Library for geometrical functions and object-oriented programming.
Main features:
- The same code base is compiled with 2D and 3D support.
- Quadtree-based (Octree in 3D) spatial discretisation with automatic and dynamic local refinement.
- Multigrid Poisson solver.
- Second-order Godunov type advection scheme.
- Solves the time-dependent incompressible variable-density Euler, Stokes or Navier-Stokes equations or the 2D shallow-water and 3D hydrostatic oceanic equations.
- Support for complex solid boundaries (automatic locally-refined mesh generation).
- Semi-implicit multigrid diffusion solver with support for complex boundaries and associated boundary conditions in 2D and 3D.
- Semi-implicit multigrid barotropic solver for the oceanic equations.
- Adaptive mesh refinement: the resolution is adapted dynamically to the features of the flow.
- Flexible and powerful specifications of parameters.
- Flexible object-oriented custom specification of initial and boundary conditions, source terms, outputs etc...
- Portable parallel support using the MPI library.
- Volume of Fluid advection scheme for interfacial flows.
Enhancements:
- Bugfixes and significant speedups in the multilevel Poisson solver.
- Support for variable mesh resolution along solid boundaries.
- Improvements have been made to the robustness of very complex solid boundaries.
- Adaptive refinement of VOF-advected tracers has been added, as well as a preliminary implementation of CSF surface tension using Renardy El Ab.
- "proper discretisation".
- Solid boundaries can be refined according to the local curvature.
- Implicit Coriolis terms work with the Navier-Stokes solver.
- There is support for "thin" 3D domains.

Java + information flow (Jif in short) is a security-typed programming language that extends Java with support for information flow control and access control, both at compile time and at run time.

The source code for the Jif compiler and run-time system is now available for download. Jif is written in Java and is built using the Polyglot extensible Java compiler framework.

Static information flow control can protect the confidentiality and integrity of information manipulated by computing systems. The compiler tracks the correspondence between information the policies that restrict its use, enforcing security properties end-to-end within the system. After checking information flow within Jif programs, the Jif compiler translates them to Java programs and uses an ordinary Java compiler to produce secure executable programs.

Jif extends Java by adding labels that express restrictions on how information may be used. For example, the following variable declaration declares not only that the variable x is an int, but also that the information in x is governed by a security policy:

int {Alice→Bob} x;

In this case, the security policy says that the information in x is controlled by the principal Alice, and that Alice permits this information to be seen by the principal Bob. The policy {Alice←Bob} means that information is owned by Alice, and that Alice permits it to be affected by Bob. Based on label annotations like these, the Jif compiler analyzes information flows within programs, to determines whether they enforce the confidentiality and integrity of information.

fprobe is a small NetFlow probe which will listen on a network interface. It isusing libpcap, aggregate the traffic and export NetFlow V5 datagram to a remote collector for processing. A flow is identified by ip protocol, source ip, source port, destination ip, destination port.
Right now only ethernet interfaces are supported. Support for more media types (tunnel, ppp etc) will be added in nex versions.
/fprobe -t IP:PORT [ -i interface ] [ -s scan ] [ expression ]
-t IP:PORT NetFlow collector address
-i interface interface to listen for traffic (default eth0)
-s scan interval in seconds between two flow tables scans (Default: 10)
-c file file with MAC definitions
-p dont put the interface in promisc mode
-b go in background (daemon mode)
-l file log file name
expression a bpf expresion to filter traffic (See libpcap/tcpdump)
For example:
./fprobe -i eth2 -t 127.0.0.1:8182
This will sniff the traffic on interface eth2 and will send the NetFlow data to localhost (127.0.0.1) on UDP port 8182.
Internal flow table is parsed every scan seconds for expired flows which are sent to remote collector.
Enhancements:
- can handle IP fragments
- can set the snmp interface ID based on source/destination MAC address
- fixed uptime in exported flows
- new hash function for internal storage
- delay between udp datagrams emited