• Home
• People
• Jobs
• Visit
• Thesis Topics
• Publications
• Research
• Performance
• PEMOGEN
• LibLSB
• ProbGraph
• Netgauge
• LogGOPSim
• Parallel Programming
• Secure Communication
• Scalable Networking
• Bcast
• Teaching
• Contact

SPONSORS

COLLABORATIONS

LibSciBench

Motivation

LibSciBench is a framework developed to facilitate the adoption of statistically sound measurements of parallel computations. It automates many of the guidelines and processes described in [1].

The main component of the framework is LibLSB, a performance measurement library that can be linked to any application to either measure execution times or it can be used as a building block for a new benchmark suite. The library seamlessly integrates with MPI and OpenMP applications and can easily be extended to new models of parallelism.

The datasets produced by LibLSB can be read directly with established statistical tools such as GNU R. LibSciBench is coupled with an R script for the postprocessing of the gathered data. It allows to check for normality, compute CIs, perform ANOVA tests, and quantile regression. Furthermore, LibSciBench’s R script supports the generation of Q-Q plots, box plots and violin plots.

Timers

LibSciBench offers high-resolution timers for many architectures (currently x86, x86-64, PowerPC, and Sparc). The library automatically reports the timer resolution and overhead on the target architecture. LibSciBench has support for arbitrary PAPI counters and, as showed in the example below, allows to create group of measurements identified by an arbitrary number of parameters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#include <mpi.h>
#include <liblsb.h>
#include <stdlib.h>
#include <time.h>

#define N 1024
#define RUNS 10

int main(int argc, char *argv[]){
    int i, j, rank;
    int buffer[N];
    
    srand(time(NULL));
    for (int i=0; i<N; i++) buffer[i]=rand();

    MPI_Init(&argc, &argv);
    LSB_Init("test_bcast", 0);
    
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    /* Output the info (i.e., rank, runs) in the results file */
    LSB_Set_Rparam_int("rank", rank);
    LSB_Set_Rparam_int("runs", RUNS);

    for (i=1; i<=N; i*=2){

        for (j=0; j<RUNS; j++){
            /* Reset the counters */
            LSB_Res();

            /* Perform the operation */
            MPI_Bcast(buffer, i, MPI_INT, 0, MPI_COMM_WORLD);

            /* Register the j-th measurement with the size i */        
            LSB_Rec(i);
        }
    }

    LSB_Finalize();
    MPI_Finalize();
    return 0;
}

Synchronization

Most of today’s parallel systems are asynchronous and do not have a common clock source. Furthermore, clock drift between processes could impact measurements and network latency variations make time synchronization tricky [2].

Many evaluations use an (MPI or OpenMP) barrier to synchronize processes for time measurement. This may be unreliable because neither MPI nor OpenMP provides timing guarantees for their barrier calls. While a barrier commonly synchronizes processes enough, we recommend checking the implementation. For accurate time synchronization, LibLSB offers a simple delay window scheme [2].

Post analysis

Download

References

SC15	[1] T. Hoefler, R. Belli:
		Scientific Benchmarking of Parallel Computing Systems presented in Austin, TX, USA, pages 73:1--73:12, ACM, ISBN: 978-1-4503-3723-6, Nov. 2015, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC15) (acceptance rate: 22%, 79/358)

IJPEDS	[2] T. Hoefler, T. Schneider, A. Lumsdaine:
		Accurately Measuring Overhead, Communication Time and Progression of Blocking and Nonblocking Collective Operations at Massive Scale International Journal of Parallel, Emergent and Distributed Systems. Vol 25, Nr. 4, pages 241-258, Taylor & Francis Group, ISSN: 1744-5779, Jul. 2010,