System for Internet-Level Knowledge 0.11.1

System for Internet-Level Knowledge (SiLK) project is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks.
The SiLK tool suite supports the efficient collection, storage and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.
SiLK consists of two sets of tools: a packing system and analysis suite. The packing system receives Netflow V5 PDUs and converts them into a more space efficient format, recording the packed records into service-specific binary flat files. The analysis suite consists of tools which can read these flat files and then perform various query operations, ranging from per-record filtering to statistical analysis of groups of records. The analysis tools interoperate using pipes, allowing a user to develop a relatively sophisticated query from a simple beginning.
The vast majority of the current code-base is implemented in C, Perl, or Python. This code has been tested on Linux, Solaris, OpenBSD, and Mac OS X, but should be usable with little or no change on other Unix platforms.
System for Internet-Level Knowledge software components are released under the GPL.
Enhancements:
- New scan detection system: rwscan and rwscanquery
- rwscan reads SiLK Flow data and uses a hybrid of Threshold Random Walk and Bayesian Logistic Regression to detect scanning activity. rwscan output textual records describing the scan. If these are inserted into a relational database, rwscanquery can be used to query for the scanning activity. rwscanquery can query Oracle, Postgres, or MySQL databases.
- New tools for IPFIX support
- rwsilk2ipfix converts SiLK Flow records to an IPFIX format.
- rwipfix2silk converts IPFIX flow records to the SiLK format.
- These tools can be used in place of the rwp2yaf2silk script.
- Support for these tools requires that libfixbuf-0.6.0 be installed prior to building SiLK.
- New tools for IP storage
- rwipaexport takes IP addresses from an IP Address Association (IPA) catalog and creates a SiLK IPset, Bag, or Prefix Map (pmap).
- rwipaimport enters the IP addresses from a SiLK IPset, Bag, or Prefix Map into an IPA catalog.
- Support for these tools requires that libipa-0.2.0 be installed prior to building SiLK.
- Additional new tools
- rwsplit divides a SiLK Flow file into smaller files based on the number of flows, bytes, packets, or unique IPs. It also provides the ability to sample the input.
- rwsettool provides the functionality of rwsetintersect and rwsetunion and additional functions such as set difference and sampling of an IPset. The rwsetintersect and rwsetunion tools are deprecated.
- rwsetmember determines if a (textual) IP is a member of an IPset. Determinating this in previous releases of SiLK required filtering the output of rwsetcat or creating an IPset containing a single IP.
- rwpmapcat prints the contents of a Prefix Map (pmap) file.
- rwfilter enhancements and bug fixes
- Allow the the parameter to the --flags-all, --flags-init, and --flags-session switches can be a list of HIGH/MASK pairs separated by commas, e.g., --flags-all=S/S,A/A
- Do not print statistics or create output files when the --dry-run switch is specified.
- Fix a file corruption issue that would occur when processing multiple files if the first input file was not successfully opened: the output file would be generated without a SiLK header.
- Exit with a non-zero exit status if the class, type, or sensor values are invalid.
- Fix a bug in processing the --start-date and --end-date switches when local timezone support was enabled and the local timezone was east of UTC.
- rwbag enhancements and bug fixes
- rwbag now supports creating Bags whose key is the sensor ID, next hop IP, input interface or output interface.
- Allow rwbag to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
- Print errors as human readable text, not error codes
- Fix a bug with releasing memory multiple times when rwbag ran out of memory.
- rwrandomizeip enhancement
- Allow the user to restrict the set of IPs that are modified via two command line arguments: --dont-change-set and --only-change-set. Both switches take an IPset; the first switch prevents the IP from being changed; the second causes only the listed IPs to be changed.
- mapsid enhancement
- The --print-classes switch will print the class(es) to which each sensor belongs.
- rwcount enhancement and changes
- Implemented the --output-path switch which directs rwcount to write its output to the specified location.
- Allow rwcount to act like UNIX tee(1) by adding the --copy-input switch. This switch sends all SiLK Flow input to the specified file, stream, or named pipe.
- The column widths have changed slightly
- rwaddrcount enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwcut enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwstats enhancement
- Implemented the --output-path and --copy-input switches as described for rwcount.
- rwset enhancement
- Implmented the --copy-input switch as described for rwcount.
- rwtotal enhancement
- Implemented the --output-path switch as described for rwcount.
- rwuniq enhancement
- Implemented the --output-path switch as described for rwcount.
- rwsetcat bug fix
- Fix bug where the $PAGER was not being used.
- rwbagcat bug fixes
- Do not print a warning message when attempting to print an empty Bag or when the min/max limits caused no entries to be printed.
- Fix bug where the $PAGER was not being used.
- Print errors as human readable text, not error codes
- rwbagtool bug fix
- Print errors as human readable text, not error codes
- rwcat bug fix
- Modify rwcat so it will always print the SiLK header to a file, even when no records are present
- rwappend enhancement and bug fix
- New --print-statistics switch causes the number of records processed to be printed to the standard error.
- Output change: Modified rwappend so it only prints the number of records processed when --print-statistics is given.
- Fix a problem that occurred when SiLK was compiled with compression enabled by default and the applications were processing SiLK files produced by releases of SiLK prior to 0.10.5: the application would exit with the error message "Operation not permitted on compressed file" and no output would be generated.
- rwswapbytes bug fix
- See compression-related bug fix for rwappend
- rwnetmask bug fix
- See compression-related bug fix for rwappend
- Administration and configuration changes:
- New "silk.conf" file removes the requirement that sensors be defined at compile-time.
- The sensors, classes, and types are now defined at run-time through the use of a "silk.conf" text file. This file should be installed in the SILK_DATA_ROOTDIR directory.
- The run-time configuration allows a single installation of the analysis tools to query multiple data sets; simply set the SILK_DATA_ROOTDIR environment variable to the location of the data.
- The location of this file can also be specified by setting the SILK_CONFIG_FILE environment variable to its location, or by using the --site-config-file switch on most SiLK applications.
- The packer (rwflowpack) still requires certain classes and types to be defined, and it cannot use new classes and types without modifying C code. This restriction will go away in a future release.
- Major changes to the build system.
- The build system now uses all aspects of the GNU Autotools chain including automake and libtool.
- The tools can now be built with shared library support, reducing the size of the binaries and allowing the kernel to use a single copy of libsilk when multiple SiLK tools are running.
- Note that the use of shared libraries means the binaries can no longer easily be relocated; instead you should run "make install" again with the new location.
- The SiLK headers are now copied to the install target directory
- GNU make is no longer required to build the tools.
- New packing rules are used by default.
- The default site has changed from "generic" to "twoway". The twoway site allows flow records to be categorized and stored as internal-to-internal (int2int) and external-to-external (ext2ext). In addition, the "out" type is no longer everything that is not "in". The files created by the generic site are forward compatible with the twoway site; however, if you wish to continue using your current packing rules, run configure with the --enable-silk-site=generic switch. See the SiLK Installation Handbook for details.
- New transfer daemons: rwsender and rwreceiver
- These are meant to replace the direct connectivity between flowcap and rwflowpack. These daemons allow the flowcap files to be sent to multiple rwflowpack processes.
- In addition, they allow rwflowpack to process data on one system and send small files containing SiLK Flow records (called "incremental files") to another system (where the rwflowappend daemon is running) for analysis.
- New packing tool: rwflowappend
- rwflowappend appends SiLK Flow records contained in "incremental files" to hourly files.
- Changes to flowcap and rwflowpack
- The flowcap and rwflowpack tools have been modified to work with the new rwsender and rwreceiver, though they can also be used in legacy mode. With the transport removed from flowcap, flowcap files can now be sent to multiple locations.
- IPFIX flow collection enhancement
- Previous releases of SiLK (rwflowpack and flowcap) could only read IPFIX streams generated by YAF. With this release, SiLK can read flows from any IPFIX-compliant generator.
- Remove zlib requirement in rwflowpack
- Allow rwflowpack to be built even if zlib is not available. However, rwflowpack will not be able to read files of NetFlow PDUs when zlib is not present.
- New packing tool: rwpackchecker
- rwpackchecker performs a basic integrity check of a packed SiLK file.