Free statistic software for linux

RRDStats is a Coyote Linux and BrazilFW add-on package for network traffic monitoring, link quality control, and QOS classes monitoring.
RRD Statistics project is based on RRDtool for storing data to round robin databases, and a slightly modified RRDcgi for visualizing data through a Web interface.
Main features:
- Realtime graphical statistics for bandwidth usage and link quality
- Graphical statistics of QOS priority classes usage
- Historical data stored for one week
Configuration:
All default configuration is stored in /etc/rrd.config. This version supports web based configuration and there is no need to manual configuration for basic package functionality. Just install the packages and browse to your web administration interface (by default its http://192.168.0.1:8180). There should be new link at left menu labeled "RRDStats configuration"
There are some basic options you should set up to fit your configuration. First get sure, the RRDstats package is enabled (its the first option at configuration screen). After that should you set up your line speed (just some basic approximation is good enough). The last this you should set up is your internet gateway IP address. This IP address is used to measure your internet link latency and packet loss.
Ignore other configuration options for now, save your configuration and reboot router. After your system boots up, you can browse RRD statistics.
After system startup, package is initialiazed with /etc/rc.d/pkgs/rc.rrdstats. This file start another copy of tiny webserver which listens by default on port 8080. It reads its homepage files from /var/rrd/www/ directory. After webserver startup there are also started some data gathering threads.
They read transfered data from network interfaces, QOS classes and measure link latency. These values are then stored in RRD databases. RRD databases are by default stored in /var/rrd/data/ directory
For further information how RRD databases work, please visit their homepage. Simply said RRD database has constant size, it does not grow over time and stores average data over period of time.
Last component of RRDStats package are .cgi and template files which display data from RRD databases using web interface. As said before, these files and templates are stored in /var/rrd/www/ and its subdirectories.

Statistics::SPC is a Perl module with calculations for Stastical Process Control (SPC).

Creates thresholds based on the variability of all data, # of samples not meeting spec, and variablity within sample sets, all from training data.
Note: this is only accurate for data which is normally distributed when the process is under control

Recommended usage: at least 15 sample sets, w/ sample size >=2 (5 is good) This module is fudged to work for sample size 1, but its a better idea to use >= 2

Important: the closer the process your are monitoring to how you would like it to be running (steady state), the better the calculated control limits will be.
Example: we take 5 recordings of the CPU utilization at random intervals over the course of a minute. We do this for 15 minutes, keeping all fifteen samples. Using this will be able to tell whether or not CPU use is in steady state.

SYNOPSIS

my $spc = new Statistics::SPC;
$spc->n(5) # set the number of samples per set
$spc->Uspec(.50); # CPU should not be above 50% utilization
$spc->Lspec(.05); # CPU should not be below 5%
# (0 is boring in an example)

# Now feed training data into our object
$return = $spc->history($history); # "train the system";
# $history is ref to 2d array;
# $return > 1 means process not likely to
# meet the constraints of your specified
# upper and lower bounds

# now check to see if the the latest sample of CPU util indicates
# CPU utilization was under control during the time of the sample

$return = $spc->test($data); # check one sample of size n
# $return < 0 there is something wrong with your data
# $return == 0 the sample is "in control"
# $return > 0 there are $return problems with the sample set

Statistics::Gap Perl module is an adaptation of the Gap Statistic.

SYNOPSIS

use Statistics::Gap;
$predictedk = &gap("prefix", "vec", INPUTMATRIX, "rbr", "h2", 30, 10, rep, 90, 4);

OR

use Statistics::Gap;
$predictedk = &gap("prefix", "vec", INPUTMATRIX, "rbr", "h2", 30, 10, rep, 90, 4, 7);

INPUTS

1. Prefix: The string that should be used to as a prefix while naming the intermediate files and the .dat files (plot files).
2. Space: Specifies the space in which the clustering should be performed. Valid parameter values: vec - vector space sim - similarity space
3. InputMatrix: Path to input matrix file. (More details about the input file-format below.)
4. ClusteringMethod: Specifies the clustering method to be used. (Learn more about this at: http://glaros.dtc.umn.edu/gkhome/cluto/cluto/overview)
Valid parameter values:
rb - Repeated Bisections
rbr - Repeated Bisections for by k-way refinement
direct - Direct k-way clustering
agglo - Agglomerative clustering
bagglo - Partitional biased Agglomerative clustering
NOTE: bagglo can be used only if space=vec
5. Crfun: Specifies the criterion function to be used for finding clustering solutions. (Learn more about this at: http://glaros.dtc.umn.edu/gkhome/cluto/cluto/overview)
Valid parameter values:
i1 - I1 Criterion function
i2 - I2 Criterion function
e1 - E1 Criterion function
h1 - H1 Criterion function
h2 - H2 Criterion function
6. K: This is an approximate upper bound for the number of clusters that may be present in the dataset.
7. B: The number of replicates/references to be generated.
8. TypeRef: Specifies whether to generate B replicates from a reference or to generate B references.
Valid parameter values:
rep - replicates
ref - references
9. Percentage: Specifies the percentage confidence to be reported in the log file. Since Statistics::Gap uses parametric bootstrap method for reference distribution generation, it is critical to understand the interval around the sample mean that could contain the population ("true") mean and with what certainty.
10. Precision: Specifies the precision to be used while generating the reference distribution.
11. Seed: The seed to be used with the random number generator. (This is an optional parameter. By default no seed is set.)

Statistics::ROC is a Perl module with receiver-operator-characteristic (ROC) curves with nonparametric confidence bounds.

SYNOPSIS

use Statistics::ROC;

my ($y) = loggamma($x);
my ($y) = betain($x, $p, $q, $beta);
my ($y) = Betain($x, $p, $q);
my ($y) = xinbta($p, $q, $beta, $alpha);
my ($y) = Xinbta($p, $q, $alpha);
my (@rk) = rank($type, @r);
my (@ROC) = roc($model_type,$conf,@val_grp);

This program determines the ROC curve and its nonparametric confidence bounds for data categorized into two groups. A ROC curve shows the relationship of probability of false alarm (x-axis) to probability of detection (y-axis) for a certain test. Expressed in medical terms: the probability of a positive test, given no disease to the probability of a positive test, given disease. The ROC curve may be used to determine an optimal cutoff point for the test.
The main function is roc(). The other exported functions are used by roc(), but might be useful for other nonparametric statistical procedures.

loggamma

This procedure evaluates the natural logarithm of gamma(x) for all x>0, accurate to 10 decimal places. Stirlings formula is used for the central polynomial part of the procedure. For x=0 a value of 743.746924740801 will be returned: this is loggamma(9.9999999999E-324).

betain

Computes incomplete beta function ratio

Remarks:
Complete beta function: B(p,q)=gamma(p)*gamma(q)/gamma(p+q)
log(B(p,q))=ln(gamma(p))+ln(gamma(q))-ln(gamma(p+q))

Incomplete beta function ratio:
I_x(p,q)=1/B(p,q) * int_0^x t^{p-1}*(1-t)^{q-1} dt

--> log(B(p,q)) has to be supplied to calculate I_x(p,q)
log denotes the natural logarithm
$beta = log(B(p,q))
$x = x
$p = p
$q = q
The subroutine returns I_x(p,q). If an error occurs a negative value
{-1,-2} is returned.

Betain

Computes the incomplete beta function by calling loggamma() and betain().

xinbta

Computes inverse of incomplete beta function ratio

Remarks:

Complete beta function: B(p,q)=gamma(p)*gamma(q)/gamma(p+q)
log(B(p,q))=ln(gamma(p))+ln(gamma(q))-ln(gamma(p+q))

Incomplete beta function ratio:
alpha = I_x(p,q) = 1/B(p,q) * int_0^x t^{p-1}*(1-t)^{q-1} dt

--> log(B(p,q)) has to be supplied to calculate I_x(p,q)
log denotes the natural logarithm
$beta = log(B(p,q))
$alpha= I_x(p,q)
$p = p
$q = q
The subroutine returns x. If an error occurs a negative value {-1,-2,-3}
is returned.

Xinbta

Computes the inverse of the incomplete beta function by calling loggamma() and xinbta().

rank

Computes the ranks of the values specified as the second argument (an array). Returns a vector of ranks corresponding to the input vector. Different types of ranking are possible (high, low, mean), and are specified as first argument. These differ in the way ties of the input vector, i.e. identical values, are treated:

high:

replace ranks of identical values with their highest rank

low:

replace ranks of identical values with their lowest rank

mean:

replace ranks of identical values with the mean of their ranks

roc

Determines the ROC curve and its nonparametric confidence bounds. The ROC curve shows the relationship of "probability of false alarm" (x-axis) to "probability of detection" (y-axis) for a certain test. Or in medical terms: the "probability of a positive test, given no disease" to the "probability of a positive test, given disease". The ROC curve may be used to determine an "optimal" cutoff point for the test.

The routine takes three arguments:

(1) type of model: decrease or increase, this states the assumption that a higher (increase) value of the data tends to be an indicator of a positive test result or for the model decrease a lower value.
(2) two-sided confidence interval (usually 0.95 is chosen).
(3) the data stored as a list-of-lists: each entry in this list consits of an "value / true group" pair, i.e. value / disease present. Group values are from {0,1}. 0 stands for disease (or signal) not present (prior knowledge) and 1 for disease (or signal) present (prior knowledge). Example: @s=([2, 0], [12.5, 1], [3, 0], [10, 1], [9.5, 0], [9, 1]); Notice the small overlap of the groups. The optimal cutoff point to separate the two groups would be between 9 and 9.5 if the criterion of optimality is to maximize the probability of detection and simultaneously minimize the probability of false alarm.

Returns a list-of-lists with the three curves: @ROC=([@lower_b], [@roc], [@upper_b]) each of the curves is again a list-of-lists with each entry consisting of one (x,y) pair.

Examples:

$,=" ";
print loggamma(10), "n";
print Xinbta(3,4,Betain(.6,3,4)),"n";

@e=(0.7, 0.7, 0.9, 0.6, 1.0, 1.1, 1,.7,.6);
print rank(low,@e),"n";
print rank(high,@e),"n";
print rank(mean,@e),"n";

@var_grp=([1.5,0],[1.4,0],[1.4,0],[1.3,0],[1.2,0],[1,0],[0.8,0],
[1.1,1],[1,1],[1,1],[0.9,1],[0.7,1],[0.7,1],[0.6,1]);
@curves=roc(decrease,0.95,@var_grp);
print "$curves[0][2][0] $curves[0][2][1] n";

Statistics::TTest is a Perl module to perform T-test on 2 independent samples.

Statistics::TTest::Sufficient - Perl module to perfrom T-Test on 2 indepdent samples using sufficient statistics

SYNOPSIS

#example for Statistics::TTest
use Statistics::PointEstimation;
use Statistics::TTest;
my @r1=();
my @r2=();
my $rand;

for($i=1;$iset_significance(90);
$ttest->load_data(@r1,@r2);
$ttest->output_t_test();
$ttest->set_significance(99);
$ttest->print_t_test(); #list out t-test related data

#the following thes same as calling output_t_test() (you can check if $ttest->{valid}==1 to check if the data is valid.)
my $s1=$ttest->{s1}; #sample 1 a Statistics::PointEstimation object
my $s2=$ttest->{s2}; #sample 2 a Statistics::PointEstimation object
print "*****************************************************nn";
$s1->output_confidence_interval(1);
print "*****************************************************nn";
$s2->output_confidence_interval(2);
print "*****************************************************nn";

print "Comparison of these 2 independent samples.n";
print "t F-statistic=",$ttest->f_statistic()," , cutoff F-statistic=",$ttest->f_cutoff(),
" with alpha level=",$ttest->alpha*2," and df =(",$ttest->df1,",",$ttest->df2,")n";
if($ttest->{equal_variance})
{ print "tequal variance assumption is accepted(not rejected) since F-statistic < cutoff F-statisticn";}
else
{ print "tequal variance assumption is rejected since F-statistic > cutoff F-statisticn";}

print "tdegree of freedom=",$ttest->df," , t-statistic=T=",$ttest->t_statistic," Prob >|T|=",$ttest->{t_prob},"n";
print "tthe null hypothesis (the 2 samples have the same mean) is ",$ttest->null_hypothesis(),
" since the alpha level is ",$ttest->alpha()*2,"n";
print "tdifference of the mean=",$ttest->mean_difference(),", standard error=",$ttest->standard_error(),"n";
print "t the estimate of the difference of the mean is ", $ttest->mean_difference()," +/- ",$ttest->delta(),"nt",
" or (",$ttest->lower_clm()," to ",$ttest->upper_clm," ) with ",$ttest->significance," % of confidencen";

#example for Statistics::TTest::Sufficient
use Statistics::PointEstimation;
use Statistics::TTest;

my %sample1=(
count =>30,
mean =>3.98,
variance =>2.63
);

my %sample2=(
count=>30,
mean=>3.67,
variance=>1.12
);


my $ttest = new Statistics::TTest::Sufficient;
$ttest->set_significance(90);
$ttest->load_data(%sample1,%sample2);
$ttest->output_t_test();
#$ttest->s1->print_confidence_interval();
$ttest->set_significance(99);
$ttest->output_t_test();
#$ttest->s1->print_confidence_interval();

Statistics::TTest

This is the Statistical T-Test module to compare 2 independent samples. It takes 2 array of point measures, compute the confidence intervals using the PointEstimation module (which is also included in this package) and use the T-statistic to test the null hypothesis. If the null hypothesis is rejected, the difference will be given as the lower_clm and upper_clm of the TTest object.

Statistics::TTest::Sufficient

This module is a subclass of Statistics::TTest. Instead of taking the real data points as the input, it will compute the confidence intervals based on the sufficient statistics and the sample size inputted. To use this module, you need to pass the sample size, the sample mean , and the sample variance into the load_data() function. The output will be exactly the same as the Statistics::TTest Module.

Statistics::Forecast is a Perl module that calculates a future value.

This is a dummy Oriented Object module that calculates a future value by using existing values. The new value is calculated by using linear regression.

SYNOPSIS

use Statistics::Forecast;

Create forecast object

my $FCAST = Statistics::Forecast->new("My Forecast Name");

Add data

$FCAST->{DataX} = @Array_X;
$FCAST->{DataY} = @Array_Y;
$FCAST->{NextX} = $NextX;

Calculate the result

$FCAST->calc;

Get the result

my $Result_Forecast = $FCAST->{ForecastY);

INTERNALS

The equation for Forecast is:
a+bx, where x is the predicted value and
_ _
a = y + bx

b = sum((x+x)(y-y))/sum(x-x)**2

METHODS

new

Receives a forecast name, only to remember and returns the blessed data structure as a Statistics::Forecast object.
my $FCAST = Statistics::Forecast->new("My Forecast");

calc

Calculate and return the forecast value.
$FCAST->calc;

dump

Prints data for debuging propose.
$FCAST->dump;

SumX

Returns the sum of X values.
my $SumOfX = $FCAST->{SumX};

SumY

Returns the sum of Y values.
my $SumOfY = $FCAST->{SumY};

SumXX

Returns the sum of X**2 values.
my $SumOfXX = $FCAST->{SumXX};

SumXY

Returns the sum of X * Y values.
my $SumOfXY = $FCAST->{SumXY};

AvgX

Returns the average of X values.
my $AvgX = $FCAST->{AvgX};

AvgY

Returns the average of Y values.
my $AvgY = $FCAST->{AvgY};

N

Return the number of X values.
my $N = $FCAST->{N};

EXAMPLE

use Statistics::Forecast;

my @Y = (1,3,7,12);
my @X = (1,2,3,4);

my $FCAST = Statistics::Forecast->new("My Forecast");

$FCAST->{DataX} = @X;
$FCAST->{DataY} = @Y;
$FCAST->{NextX} = 8;
$FCAST->calc;

print "The Forecast ", $FCAST->{ForecastName};
print " has the forecast value: ", $FCAST->{ForecastY}, "n";

Statistics::ChiSquare - How well-distributed is your data?

SYNOPSIS

use Statistics::Chisquare;

print chisquare(@array_of_numbers);
Statistics::ChiSquare is available at a CPAN site near you.

Suppose you flip a coin 100 times, and it turns up heads 70 times. Is the coin fair?

Suppose you roll a die 100 times, and it shows 30 sixes. Is the die loaded?
In statistics, the chi-square test calculates how well a series of numbers fits a distribution. In this module, we only test for whether results fit an even distribution. It doesnt simply say "yes" or "no". Instead, it gives you a confidence interval, which sets upper and lower bounds on the likelihood that the variation in your data is due to chance. See the examples below.

If youve ever studied elementary genetics, youve probably heard about Georg Mendel. He was a wacky Austrian botanist who discovered (in 1865) that traits could be inherited in a predictable fashion. He did lots of experiments with cross breeding peas: green peas, yellow peas, smooth peas, wrinkled peas. A veritable Brave New World of legumes.

But Mendel faked his data. A statistician by the name of R. A. Fisher used the chi-square test to prove it.

Theres just one function in this module: chisquare(). Instead of returning the bounds on the confidence interval in a tidy little two-element array, it returns an English string. This was a deliberate design choice---many people misinterpret chi-square results, and the string helps clarify the meaning.

The string returned by chisquare() will always match one of these patterns:

"Theres a >d+% chance, and a br bEXAMPLES/b br br Imagine a coin flipped 1000 times. The expected outcome is 500 heads and 500 tails: br br @coin = (500, 500); br print chisquare(@coin); br br prints "Theres a >90% chance, and a<<less