Bioinformatics Benchmark System 3

The Bioinformatics Benchmark System is an attempt to build a reasonable testing framework, tests, and data, to enable end users and vendors to probe the performance of their systems.

What we are trying to do is to create a framework for testing, and a core set of tests that all may download and use to probe specific elements of systems performance.

Moreover, the source to these tests are available under GPL, and are hosted on Bioinformatics.org and Scalable Informatics LLC The idea is to enable end users, consumers, systems developers, and others to easily build and use meaningful tests for measurement and tuning reasons.

Joe Landman from Scalable Informatics LLC conceived the idea and wrote the original codes. We are looking for additional benchmark code suggestions, tests, data sets, etc.

Current baseline tests are several NCBI BLAST runs, several HMMer runs, and a variety of others. We plan to include ClustalW, X!Tandem, various chemistry, dynamics, and related tests, as well as several others.

Tests such as LINPACK or HPL simply do not provide meaningful performance indicators or predictive models for high performance informatics. Unfortunately, nor do a number of more recent and focused tests.

This is a problem as LINPACK and HPL specifically test the performance on various matrix operations, where you have effectively regular memory access patterns, and specific mathematical operations.

These codes are most useful for comparison to codes with heavy floating point operations, and interleaved memory traffic. These codes were not designed for comprehensive systems benchmarking, where disk I/O, memory latency, and other factors all contribute to the performance issues.

The best tests are the ones that are most similar to the codes you will run on the machine. The tests themselves should be reasonable approximations to a real execution of your code, using real data. You may need to pare it back in order to get realistic run times.

You should have a reasonable subset of data sizes. A single test does not tell you how your system scales, and one of the reasons for the existance of this test is specifically to allow you to test the performance while you increase various aspects of the workload.

You rarely get a quiescent system in a cluster, so we would recommend that you try to run in as realistic an operating environment as possible. A baseline in a quiescent system is fine, but it may set your expectations unreasonably.
top