Free statistic analyzer software for linux

Sendmail Analyzer is a perl script reporting full HTML and graph sendmail usage reports. The project reports statistics on inbound, outbound and largest messages, senders and recipients, relays, domains, and complete mailboxes usage if your country law allow it.

Statistics are generated per hour, day, month and year. Graphs are in PNG format.

Network Traffic Analyzer analyzes the network traffic on multiple network devices and creates HTML statistics with some network usage graphs. Sometimes it is good to know, how the network is used, how many bytes were received and how many bytes were sent.Therefore, here is Network Traffic Analyzer, which creates simple network usage statistics.


Such statistics can tell you, how good your network connection really is (who cares about what Internet provider say, when was the network down, which time is the best time for downloading large packages of data etc. etc. Or with this software you can just better imagine, how many traffic can your home computer generate.

Python Traffic Analyzer is a Python base class and sample driver script written to retrieve and manipulate images from the TrafficLand cameras and calculate a numeric value representing the current traffic flow.

PyTrAn, an example driver script, an image collector and an image mask creator are available for download from the link shown at the bottom. To use the PyTrAn package begin by choosing a camera that you wish to analyze, for this example well use the camera captioned above.

We want to construct a mask over the area of the image that we are interested in, namely the road. In this particular example the road takes up the majority of the image but that is not always the case.

We will apply the mask over captured images to fine tune the area over which we are looking for movement. To create the mask we will first need to collect a sequential series of snapshots from the target camera. The image_collector.py script was written for this task:

$ mkdir mask_200003
$ cd mask_200003
$ ../image_collector.py 200003 30
Collecting 30 images...
30

Done.

The script is hard coded to capture images on a 2-second delay. The delay is necessary to ensure the image has changed. I believe 2-seconds to be the absolute minimum. Once complete, 30 images numbered 1 through 30 will be created in the current directory.

We construct a mask from these captured images by creating a diff-image for each sequential image pair and then adding each diff-image together. Naturally, a script was written to automate this task as well:

$ ../mask_maker.py 1 30
Creating a diff for each sequential image pair.
Diffing 29

Creating the initial mask from the first image pair.

Adding the rest of the diffs to the mask.
Masking 29

Done.

A number of .diff files are generated in this process. These files repesent the movement between individual sequence pairs.

The .diff files are simply intermediary files, the important bit is the mask file, which is generated as the sum of all differences.

The mask file may be dirty (as in this case) and require manual cleanup. The basic shape of the road however is clearly visible, evidence that we can with minimal effort automate the mask generation process. Also, this run was conducted at night, day-time images yield better results.

There are a few final steps we need to take before we can use the example PyTrAn driver script. First we need to convert the mask to ASCII (noraw) format:

$ pnmnoraw mask > mask_200003.ascii

Then we need to open an ImageMagick display window and get its X-window-ID using xwininfo. Finally, update camera_id and window_id in pytran_sampling.py and launch the driver:

$ ../pytran_sampling.py
DEBUG> grabbing frame from camera 200003
DEBUG> rotating image: pytran.this > pytran.last
DEBUG> refreshing image in 3 secs
taking a 5 minute sample at various thresholds.
DEBUG> grabbing frame from camera 200003
DEBUG> generating frame diff on pytran.last, pytran.this
DEBUG> displaying image: pytran.diff
DEBUG> converting pytran.diff to ascii
DEBUG> calculating traffic ratio...
ratio[5]: 55%
DEBUG> calculating traffic ratio...
ratio[10]: 52%
...
...
5 minute sample[5]: 67.88
5 minute sample[10]: 42.66
5 minute sample[15]: 30.57
5 minute sample[20]: 23.03
5 minute sample[25]: 18.39
5 minute sample[30]: 14.79
5 minute sample[35]: 12.42
5 minute sample[40]: 10.53
5 minute sample[45]: 9.06
5 minute sample[50]: 7.85

The sampling script will take 5 minute samples at varying color thresholds. The optimal threshold must be manually chosen. Furthermore, you will need to sample the traffic ratios during both heavy and light traffic times to get a good feel for your acceptable range. Also, keep in mind that the traffic ratio value is simply the percent change detected, or in other words the movement detected within the masked region. This means that a completely empty road will register similar values to a road so congested it looks like a parking lot. The time of day can be combined with the traffic ration to determine the logical truth.

With this task implemented and abstracted more complex systems can be built. When I find the time Id like to create a system that will take multiple potential travel routes and times, and during the travel time e-mail the traveler with the best route to take. Another idea I had would be to record the traffic flow values for each camera, for each day and for each half hour interval. Travelers and other interested parties can then analyze traffic patterns to determine the fastest route dependant on date/time.

Statistics::Gap Perl module is an adaptation of the Gap Statistic.

SYNOPSIS

use Statistics::Gap;
$predictedk = &gap("prefix", "vec", INPUTMATRIX, "rbr", "h2", 30, 10, rep, 90, 4);

OR

use Statistics::Gap;
$predictedk = &gap("prefix", "vec", INPUTMATRIX, "rbr", "h2", 30, 10, rep, 90, 4, 7);

INPUTS

1. Prefix: The string that should be used to as a prefix while naming the intermediate files and the .dat files (plot files).
2. Space: Specifies the space in which the clustering should be performed. Valid parameter values: vec - vector space sim - similarity space
3. InputMatrix: Path to input matrix file. (More details about the input file-format below.)
4. ClusteringMethod: Specifies the clustering method to be used. (Learn more about this at: http://glaros.dtc.umn.edu/gkhome/cluto/cluto/overview)
Valid parameter values:
rb - Repeated Bisections
rbr - Repeated Bisections for by k-way refinement
direct - Direct k-way clustering
agglo - Agglomerative clustering
bagglo - Partitional biased Agglomerative clustering
NOTE: bagglo can be used only if space=vec
5. Crfun: Specifies the criterion function to be used for finding clustering solutions. (Learn more about this at: http://glaros.dtc.umn.edu/gkhome/cluto/cluto/overview)
Valid parameter values:
i1 - I1 Criterion function
i2 - I2 Criterion function
e1 - E1 Criterion function
h1 - H1 Criterion function
h2 - H2 Criterion function
6. K: This is an approximate upper bound for the number of clusters that may be present in the dataset.
7. B: The number of replicates/references to be generated.
8. TypeRef: Specifies whether to generate B replicates from a reference or to generate B references.
Valid parameter values:
rep - replicates
ref - references
9. Percentage: Specifies the percentage confidence to be reported in the log file. Since Statistics::Gap uses parametric bootstrap method for reference distribution generation, it is critical to understand the interval around the sample mean that could contain the population ("true") mean and with what certainty.
10. Precision: Specifies the precision to be used while generating the reference distribution.
11. Seed: The seed to be used with the random number generator. (This is an optional parameter. By default no seed is set.)

Statistics::LineFit module least squares line fit, weighted or unweighted.

SYNOPSIS

use Statistics::LineFit;
$lineFit = Statistics::LineFit->new();
$lineFit->setData (@xValues, @yValues) or die "Invalid data";
($intercept, $slope) = $lineFit->coefficients();
defined $intercept or die "Cant fit line if x values are all equal";
$rSquared = $lineFit->rSquared();
$meanSquaredError = $lineFit->meanSqError();
$durbinWatson = $lineFit->durbinWatson();
$sigma = $lineFit->sigma();
($tStatIntercept, $tStatSlope) = $lineFit->tStatistics();
@predictedYs = $lineFit->predictedYs();
@residuals = $lineFit->residuals();
(varianceIntercept, $varianceSlope) = $lineFit->varianceOfEstimates();

The Statistics::LineFit module does weighted or unweighted least-squares line fitting to two-dimensional data (y = a + b * x). (This is also called linear regression.) In addition to the slope and y-intercept, the module can return the square of the correlation coefficient (R squared), the Durbin-Watson statistic, the mean squared error, sigma, the t statistics, the variance of the estimates of the slope and y-intercept, the predicted y values and the residuals of the y values. (See the METHODS section for a description of these statistics.)
The module accepts input data in separate x and y arrays or a single 2-D array (an array of arrayrefs). The optional weights are input in a separate array. The module can optionally verify that the input data and weights are valid numbers. If weights are input, the line fit minimizes the weighted sum of the squared errors and the following statistics are weighted: the correlation coefficient, the Durbin-Watson statistic, the mean squared error, sigma and the t statistics.
The module is state-oriented and caches its results. Once you call the setData() method, you can call the other methods in any order or call a method several times without invoking redundant calculations. After calling setData(), you can modify the input data or weights without affecting the modules results.
The decision to use or not use weighting could be made using your a priori knowledge of the data or using supplemental data. If the data is sparse or contains non-random noise, weighting can degrade the solution. Weighting is a good option if some points are suspect or less relevant (e.g., older terms in a time series, points that are known to have more noise).

Visitors is a very fast web log analyzer for Linux, Windows, and other Unix-like operating systems. It takes as input a web server log file, and outputs statistics in form of different reports. The design principles are very different compared to other software of the same type:
No installation required, can process up to 150,000 lines of log entries per second in fast computers (20MB/s with my log files average length).
Designed to be executed by the command line, output html and text reports. The text report can be used in pipe to less to check web stats from ssh.
Support for real time statistics with the Visitors Stream Mode introduced with version 0.3.
To specify the log format is not needed at all. Works out of box with apache and most other web servers with a standard log format (see the documentation for more information on the format).
Its a portable C program, can be compiled on many different systems. Binaries for Windows systems are in the Download section of this page.
The produced html report doesnt contain images or external CSS, is self-contained, you can send it by email to users.
Visitors is free software (and of course, freeware), under the terms of the GPL license. You dont need to pay to use it. Visitors is supported, if you want a custom version made directly by the original author for a modest price, contact me at antirez (at) invece.org. ISPs may take advantage of the high processing speed.
Main features:
- Requested pages.
- Requested images.
- Referers by hits and age.
- Unique visitors in each day.
- Page views per visit.
- Pages accessed by the Google crawler (and the date of googles last access on every page).
- Percentage of visits originated from Google searches for every day.
- Users navigation patterns (web trails).
- Keyphrases used in Google searches.
- User agents.
- Weekdays and Hours distributions of accesses.
- Weekdays/Hours combined bidimentional map.
- Month/Year combined bidimentional map.
- Visual path analysis with Graphviz.
- Operating systems, browsers and domains popularity.
- 404 errors.
Enhancements:
- This release adds an important bugfix in the unique visitors algorithm.
- The output is now nearer to reality (though unique visitors stats are always a guess without the use of a cookie).

KSqlAnalyzer is a tool for easily accessing the data of a MS SQL database. It is made for developing and testing new SQL queries direclty on the server.
The functionality and look/feel of KSqlAnalyzer are similar to SQL Query Analyzer by Microsoft.
KSqlAnalyzer uses parts of the TDS library, and the editor uses parts of the KWrite source code becaus of its brilliant sytnax highlighting.
Enhancements:
- Trigger viewing and editing, and a new Edit menu item (Clear).