Free collectors software for linux

stic bundles a few Linux tools which are intended to support the task of collecting an unreasonable amount of pictures (preferrably in JPEG format).
similar
a program for detecting duplicate or very similar images. It maintains a
database of characteristic color samples which it compares with submitted
pictures. similar depends on libjpeg and ImageMagick s convert (on a
modern Linux desktop system these components should already be present).
Storage medium may be a usual filesystem or a MySQL database. There
also is a MySQL UDF extension to compare image samples within SQL queries.
similar contains the communications module described at sagent.
simv
a core program to perform file management tasks on an image collection.
Its main purpose is to coordinate file movements with the content of
similars database. This applies to importing new files which get tested
against the existent collection, as well as to inform similar about moving
and deleting files within the collection.
simv depends on an external image viewer like ImageMagick s display
(should already be present on a modern Linux desktop system) or John
Bradleys xv (quite a fast one).
simv contains the communications module described at sagent.
sagent
a standalone version of the communications module used in simv and
similar. This software receives input from its start terminal and multiple
clients, distributes several types of output back to them, and is also
able to act itself as such a client.
Since communications mainly use TCP/IP there is an encryption layer
(Blowfish with 128 bit keys) which provides user authentication. Any
single activity of such a user may be particularly permitted or denied.
Secure connections should be possible that way as long as one can defend
the keyfiles and programs on client and server host against foreign
access.
Front-end connection software is available in C, Tcl/Tk and PHP3 to build
custom clients. In the most primitive case even telnet can act as a
client.
The standalone program sagent may be used as communications node in a tree
of clients. Another purpose is to be a shell frontend which sends commands
to a server and receives its replies.
snntpbatch
a command line based NNTP (newsgroups) client. It is mainly intended for
automatic download of images by use of a filter language. Nevertheless it
also downloads the message texts and converts them to HTML code which
includes the downloaded images. Also, it is capable of automatically
posting sets of images to the newsgroups.
The tools are designed to be very independent of the system flavor. On an
average Linux desktop there should be no need to update existing system
components. Actually one could use stic without having display equipment for
graphics.
Any program activity which is possible in dialog may also be performed in
batch runs. Therefore the tools are quite suitable for users who like to get
boring tasks automated and manual tasks simplified.
All tools code is open source and distributed under BSD license.
Example images Credit: U. S. Fish and Wildlife Service (see images/CREDITS)
Enhancements:
- The new encrypted protocol version 0.2 is standard now : SHA-1 seal, 256 bit keys, variable chaining initialization vector.
- The protocol of a connection is chosen by the client and may or may not be accepted by the server.
- See sagent command -security options clientprotocol , serverprotocol.

It is a package useful to collect statistics from remote nodes. That includes standard server metrics and any other numerical quantity we can use to characterize the node state (as the number of users or processes).

It is built under a distributed agent-manager model. RRDtool is used as database, so representation capabilities are bundled.

Although rrdUtils are not a requisite, they are used in some auxiliar tasks and using both tools is highly recommended.

Installation:

Manager side

gunzip -c rrdUtils-3.1.tar.gz | tar -xf -
cd rrdUtils
./configure --with-rrddir=/stats
make install

gunzip -c stats-station-3.0.tar.gz | tar -xf -
cd stats-station
./configure --with-statsdir=/stats --with-port=666
make install

cd rrdConf
./install_confs.sh
./haz_index.sh -g

The -g flag creates graphs with the html pages, and reduces the number of error messages that haz_index will issue.
The only task that needs to be completed at this point is the periodic graphs generation by a crontab entry that calls the make_graphs.sh script.
Once we start the server using the provided initscript, we get the manager side ready.

Agent side

gunzip -c stats-3.0.tar.gz | tar -xf -
cd stats
./configure --with-host=central.node --with-port=666
--with-statsdir=/usr/local/stats
make install

With a crontab entry to execute the statlaunch script every 15 minutes, the whole system is up and running.

Webcomics Collector is a GPLed python script for downloading webcomics. Webcomics Collector can start from the frontpage and work its way back through the archive of a webcomic, downloading the entire archive independently of dates or naming of strip-images.

There is order in which the strips should appear is recorded in a file which the webinterface and frontends can use. FYI: Alot of webcomics dont name their strip-images in a way that allows you to just fire up an imageviewer and start reading, so it is recommended that you use the webinterface.

Xooctory project is an open source continuous integration server.
Main features:
- Open Source
- Massively scalable
- Flexible
- Build dependency aware
- Instant and rich feedback
- Secured
What is Continuous Integration?
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.
Which technologies do you use?
Xooctory is not only a continuous integration server, its also a project meant to experiment with top notch technologies.
As mostly Java developers and fellows, xooctory is java based, and tries to leverage the latest open source components and libraries.
As such, it can also be used only to see how the technologies we chose work together, and maybe how they could improve your own projects.
Here is a short list of some of them we are proud to use in xooctory:
- Java 6
- Spring 2
- Hibernate 3
- Mule 1.3
- Wicket 1.3
- Logback 0.9
- Jetty 6
- Ivy 2
Enhancements:
- Several new features, including job dependencies handling, new job result collectors (workspace, artifacts, and compilation), differential reporting, an improved push implementation, and last and last successful jobs permalinks.

System Configuration Collector Server generates summaries of scc-data sent by clients. System Configuration Collector Server offers a Web interface that supports searching the snapshots and the logbooks of the systems. It also supports comparing (parts of) the snapshots of systems.
Enhancements:
- This release uses CSS directives to indicate different colors when comparing data from systems.
- It handles the case of an empty log.html when sort is implemented by busybox.
- stdout and stderr are recorded from client runs in the log file when started by scc-pull.
- A puppylinux package has been added.
- style.css has been updated to show variable data in scc snapshots.

JProfiler is an award-winning all-in-one Java profiler. JProfilers intuitive GUI helps you find performance bottlenecks, pin down memory leaks and resolve threading issues.
Even though JProfiler has a powerful feature set, its a Java profiler that is extremely easy to use! JProfilers user interface provides a unified view on the profiled application and puts all the information you need right at your fingertips with its intuitive hierarchy of views.
Configure your application for profiling
When starting up JProfiler, the start center is brought up. Here you can manage and start your profiling sessions. To configure your application for profiling, change to the "New session" tab and click on the New session button. Enter the required information into the session config dialog as described below and select Ok to start profiling. Thats all it takes. Note the Help button which appears on all of JProfilers dialogs displays a context sensitive explanation of all features.
While the configuration dialog displays numerous options that you will not want to miss after becoming familiar with JProfiler, entering the basic pieces of information is really very simple:
1. Enter a name for your session.
2. Enter the name of your main class.
3. Enter your class path.
4. Press Ok.
Before profiling is actually started, the profiling settings dialog is displayed where you can select the focus for your profiling run. Although profiling produces an overhead when running your application, you can minimize it by choosing a setting for which JProfiler only records information that is interesting for you.
If you want to fine-tune your profiling settings, the [Edit] button brings up a dialog with all available configuration options for profiling.
Observe classes and allocations
To find out what is going on the heap in terms of objects and classes, turn to the classes monitor. It gives you continuous updates and lets you set marks to observe changes over time. To see references, allocations and object data for your selection, you can take a snapshot by clicking on the camera in JProfilers toolbar.
If you want to know where your objects are allocated, you can go to the allocation monitor. Here, you can inspect the call tree and find out what method calls have caused the allocation of a selected class or package. Just like in the allocation monitor, you can display this data for live and garbage collected objects.
An cumulated overview on which methods are responsible for allocations is given by the allocation hot spots view. Each hot spot can be expanded and the backtraces that lead to the invocation of the hot spot are displayed. You can also mark the current values and view the differences in allocations over time.
Find memory leaks
JProfilers heap walker works like a browser: it displays a current set of objects that can be changed by adding selection steps with the [Use selected] button. You can inspect the current object set in the four views of the heap walker.
In the classes view, you can select one or several classes and add a selection step. The new object set will contain only the selected instances. This is often the first thing you want to do after taking a snapshot. You can perform this step automatically by invoking the heap walker from the classes monitor.
The allocations view of the heap walker shows the allocation tree and the allocation hot spot list of the current object set. You can add a selection step for one or multiple allocation spots or hot spots. The new object set will then be restricted to the selected instances only.
When youre looking for the cause of a memory leak, the reference view is the most important view in JProfiler. Here, you can find out why an object has not been garbage collected. Use the "Show path to GC root" function to show where the JVM hangs on to the selected instance.
Not only is the heap walker your first stop for finding memory leaks, it also makes for an excellent debugging facility. In the data view, you can inspect instances, arrays and classes in the current object set and navigate along references. There are many situations where the usual debugging approach will fail and JProfiler will help you find out.
Zoom in on performance bottlenecks
Controlling your applications performance may be driven by quality of service constraints or by general quality assurance, JProfilers CPU views lend themselves naturally to both approaches.
To record CPU data, you switch to the CPU section and click on the recording button in JProfilers tool bar. The first view in the CPU section shows the invocation tree, cumulated for all threads. To find performance related problem spots, just follow the big percentages when opening the tree nodes. Also, the absolute times and the number of invocations that are displayed for each node can help you in getting a feeling for the situation. Once you become familiar with JProfiler, you may want to customize this view to best fit your personal preferences.
The invocation tree is a top-down view on the method calls of your application. This viewpoint is most useful when you experience an actual performance bottleneck in your application. If you want to improve overall performance, a bottom-up view may be more applicable.
This kind of view - called hot spots view - shows the methods where most of the time is spent. By opening these nodes you get backtraces which show the various invocation paths together with the percentages of their contributions. Note that by default, method calls within Java core classes are not shown separately - each of your library calls is treated as opaque. You can change this behavior by deactivating the corresponding filter sets.
The method graph combines both viewpoints. Incoming and outgoing method calls are presented in the same way. For a number of situations, the method graph can give you more insight than the invocation tree and the hot spots view. The best strategy is to use the invocation and hot spots views first and switch to the method graph for detailed analysis.
Solve thread-related problems
Threads can be exceptionally difficult to debug, especially if you dont have sufficient information on the actual sequence and status of the threads in your application. JProfilers thread views provide you with exactly this knowledge.
The thread history view shows a continuous update of the lifelines of all threads on the horizontal axis. The names of the threads are displayed on the vertical axis in the order of their creation. Each color signifies a different thread status. Orange means that the thread was sleeping, green stands for a runnable thread while red is displayed if the thread was waiting for a monitor. You can zoom in and out to your desired detail level or have the time axis fit your windows size automatically.
If youre more interested in whats happening right now, the thread monitor view is the right place to look at. Here, you can sort threads, filter with respect to thread status and view additional information on each thread.
Should you ever have a deadlock in your application, the deadlock detection graph will help you analyze the involved threads and the locking situation. Simple deadlocks like the one shown could be worked out by hand from the other views, but for complicated deadlock involving a greater number of threads this view is indispensable.
Reducing general monitor contention and debugging locking sequences requires a detailed view of the current monitor usage and a history of all monitor-related events. JProfiler has both. Below you see the monitor usage history which shows a blocking event together with the stack trace of the waiting thread.
Keep an eye on your JVM
Monitoring cumulative parameters of the virtual machine can be a highly fruitful activity, even if everything seems to be all right. Measuring and observing parameters like heap size, object count, loaded classes and thread numbers can point to dangerous trends and problematic behavior to look out for. In its VM telemetry view section, JProfiler features various telemetry controls which provide you with the information you need to stay one step ahead.
The number of objects on the heap, split in arrays and non-arrays. This is your first stop if you are suspecting a memory leak. All objects with live references are included, as well as those which are unreferenced but the garbage collector hasnt had a chance to collect yet.
The garbage collector activity which displays freed and moved objects. If your application is thrashing the heap excessively, this will show up here. Moving large numbers of objects places a high burden on the virtual machine and can lead to temporary freezes. Mostly this occurs when the virtual machine is enlarging the heap.
The number of threads in the virtual machine, split in active and inactive threads. This is useful if you create a large number of threads and need information on changes in their total number and how many of them actually run.
Enhancements:
- New futures:
- rewritten eclipse 3.x integration
- rewritten IDEA 4.x integration
- IDE integration for JDeveloper
- IDE integration for Netbeans 4.0
- considerably reduced memory consumption
- improved long-term stability for profiling
- heap walker: in the cumulated incoming reference view, reference holders as well as referenced objects can be displayed and selected
- option to keep the profiled JVM alive
- support for Java Web Start 1.5
- CSV export for graphs
- enhancements in XML export for trees
- integration wizard for profiling servers in IBM WSAD
- integration wizards for Pramati 3.5 and Pramati 4.x application server
- integration wizard for Websphere 4.0 Advanced Edition
- integration wizard for Sun Java System Web Server
- integration wizard for Sun Java System Application Server
- integration wizard for Oracle 10g Application server
- integration wizard for Apple WebObjects Developer 5.x
- snapshot files (*.jps) can be opened from the command line and from the Windows explorer
- JBuilder IDE integration now supports JBuilder 2005
- much better appearance of the JProfiler GUI with Windows native look and feel
- JProfiler GUI now also runs under Java 1.5
- Bug fixes:
- monitor statistics were broken
- invalid class files caused a shutdown of the profiled application
- many bug fixes in the GUI

flow-tools is a set of programs for processing and managing NetFlow exports from Cisco and Juniper routers. The tools included are: flow-capture, flow-cat, flow-dscan, flow-expire, flow-export, flow-fanout, flow-filter, flow-gen, flow-header, flow-import, flow-mask, flow-merge, flow-nfilter, flow-print, flow-receive, flow-report, flow-send, flow-split, flow-stat, flow-tag, and flow-xlate.

Flow data is collected and stored by default in host byte ordera nd the files are portable across every endian architectures.

Commands that utilize the network use a localip/remoteip/port designation for communication. "localip" is the IP address the host will use as a source for sending or bind to when receiving NetFlow PDUs (ie the destination address of the exporter. Configuring the "localip" to 0 will force the kernel to decide what IP address to use for sending and listen on all IP addresses for receiving. "remoteip" is the destination IP address used for sending or the expected address of the source when receiving. If the "remoteip" is 0 then the application will accept flows from any source address. The "port" is the UDP port number used for sending or receiving. When using multicast addresses the localip/remoteip/port is used to represent the source, group, and port respectively.

Flows are exported from a router in a number of different configurable versions. A flow is a collection of key fields and additional data. The flow key is {srcaddr, dstaddr, input, output, srcport, dstport, prot, ToS}. Flow-tools supports one export version per file.

Export versions 1, 5, 6, and 7 all maintain {nexthop, dPkts, dOctets, First, Last, flags}, ie the next-hop IP address, number of packets, number of octets (bytes), start time, end time, and flags such as the TCP header bits. Version 5 adds the additional fields {src_as, dst_as, src_mask, dst_mask}, ie source AS, destination AS, source network mask, and destination network mask. Version 7 which is specific to the Catalyst switches adds in addition to the version 5 fields {router_sc}, which is the Router IP address which populates the flow cache shortcut in the Supervisor. Version 6 which is not officially supported by Cisco adds in addition to the version 5 fields {in_encaps, out_encaps, peer_nexthop}, ie the input and output interface encapsulation size, and the IP address of the next hop within the peer. Version 1 exports do not contain a sequence number and therefore should be avoided, although it is safe to store the data as version 1 if the additional fields are not used.

Version 8 IOS NetFlow is a second level flow cache that reduces the data exported from the router. There are currently 11 formats, all of which provide {dFlows, dOctets, dPkts, First, Last} for the key fields.

8.1 - Source and Destination AS, Input and Output interface
8.2 - Protocol and Port
8.3 - Source Prefix and Input interface
8.4 - Destination Prefix and Output interface
8.5 - Source/Destination Prefix and Input/Output interface
8.9 - 8.1 + ToS
8.10 - 8.2 + ToS
8.11 - 8.3 + ToS
8.12 - 8.5 + ToS
8.13 - 8.2 + ToS
8.14 - 8.3 + ports + ToS

Version 8 CatIOS NetFlow appears to be a less fine grained first level flow cache.

8.6 - Destination IP, ToS, Marked ToS,
8.7 - Source/Destination IP, Input/Output interface, ToS, Marked ToS,
8.8 - Source/Destination IP, Source/Destination Port,
Input/Output interface, ToS, Marked ToS,

The following programs are included in the flow-tools distribution.

flow-capture - Collect, compress, store, and manage disk space for exported flows from a router.
flow-cat - Concatenate flow files. Typically flow files will contain a small window of 5 or 15 minutes of exports. Flow-cat can be used to append files for generating reports that span longer time periods.
flow-fanout - Replicate NetFlow datagrams to unicast or multicast destinations. Flow-fanout is used to facilitate multiple collectors attached to a single router.
flow-report - Generate reports for NetFlow data sets. Reports include source/destination IP pairs, source/destination AS, and top talkers. Over 50 reports are currently supported.
flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to group flows by customer network. The tags can later be used with flow-fanout or flow-report to generate customer based traffic reports.
flow-filter - Filter flows based on any of the export fields. Flow-filter is used in-line with other programs to generate reports based on flows matching filter expressions.
flow-import - Import data from ASCII or cflowd format.
flow-export - Export data to ASCII or cflowd format.
flow-send - Send data over the network using the NetFlow protocol.
flow-receive - Receive exports using the NetFlow protocol without storing to disk like flow-capture.
flow-gen - Generate test data.
flow-dscan - Simple tool for detecting some types of network scanning and Denial of Service attacks.
flow-merge - Merge flow files in chronoligical order.
flow-xlate - Perform translations on some flow fields.
flow-expire - Expire flows using the same policy of flow-capture.
flow-header - Display meta information in flow file.
flow-split - Split flow files into smaller files based on size, time, or tags.

THC-UnixHackingTools is a compilation of needfull unix hacking tools including backdoors, cleaners, tunnels, etc written by THC members, that compile on various unix platforms.

They are not elite of course, otherwise we wouldnt release them, but they are very useful - so take a look and tryem out...

Collector v1.0 library to transfer data to other hosts (sniffer!)
Hunter v1.2 easy sniffer for Linux
ICMP-Tunnel v1.0 an icmp tunnel program for transfering files
Searcher v8.0 check admins $HOME for .rhosts/.forward etc.
Smeagol v4.4.4 nice backdoor with acct/logclean and hiding functions
clear v1.3 elite log cleaner (delete!) for utmp(x)/wtmp(x)/lastlog
cnt-svr-filetransfer small sources to transfer files on any unix system
daemonshell tcp and udp daemonshell in perl
fingerd-fileserver patch to fingerd (linux) to transfer files (cool!)
paz v1.0 process accounting zap, deletes accouting info
probe v2.3 script for remote host probing, really good.
t-shirt v4.0 our THC T-Shirt for 98, 1st released on the Cebit
zap3 enhanced zap to delete entrys. also for sys v systems

on most files, the small README inside does not tell much about its options. browse the sourcecode instead.