benchbuild.project module

Project

A project in benchbuild is an abstract representation of a software system that can live in various stages throughout an experiment. It defines two extension points for an experiment to attach on, the compile-time phase and the (optional) run-time phase.

An experiment can intercept the compilation phase of a project and perform any experiment that requires the source artifacts as input.

Furthermore, it is possible to intercept a project’s run-time pahse with a measurement.

The project definition ensures that all experiments run through the same series of commands in both phases and that all experiments run inside a separate build directory in isolation of one another.

class benchbuild.project.Project(experiment, name=NOTHING, domain=NOTHING, group=NOTHING, src_file=NOTHING, container=NOTHING, version=NOTHING, builddir=NOTHING, testdir=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_f=NOTHING, run_uuid=NOTHING, compiler_extension=NOTHING, runtime_extension=None)

Bases: object

Abstract class for benchbuild projects.

A project is an arbitrary software system usable by benchbuild in experiments. Subclasses of Project are registered automatically by benchbuild, if imported in the same interpreter session. For this to happen, you must list the in the settings under plugins -> projects.

A project implementation must provide the following method:

compile: Downloads & Compiles the source.

A project implementation may provide the following functions:

run_tests: Wrap any binary that has to be run under the

runtime_extension wrapper and execute an implementation defined set of run-time tests. Defaults to a call of a binary with the name run_f in the build directory without arguments.

clean: Clean the project’s build directory. Defaults to

recursive ‘rm’ on the build directory and can be disabled by setting the environment variable BB_CLEAN=false.

Raises:

• AttributeError – Class definition raises an attribute error, if the implementation does not provide a value for the attributes NAME, DOMAIN, and GROUP
• TypeError – Validation of properties may throw a TypeError.

experiment

benchbuild.experiment.Experiment – The experiment this project is assigned to.

name

str, optional – The name of this project. Defaults to NAME.

domain

str, optional – The application domain of this project. Defaults to DOMAIN.

group

str, optional – The group this project belongs to. Defaults to GROUP.

src_file

str, optional – A main src_file this project is assigned to. Defaults to SRC_FILE

container

benchbuild.utils.container.Container, optional – A uchroot compatible container that we can use for this project. Defaults to benchbuild.utils.container.Gentoo.

version

str, optional – A version information for this project. Defaults to VERSION.

builddir

str, optional – The build directory for this project. Auto generated, if not set.

testdir

str, optional – The location of any additional test-files for this project, usually stored out of tree. Auto generated, if not set. Usually a project implementation will define this itself.

cflags

list of str, optional – A list of cflags used, for compilation of this project.

ldflags

list of str, optional – A list of ldflags used, for compilation of this project.

run_f

str, optional – A filename that points to the binary we want to track. Usually a project implementation will define this itself.

run_uuid

uuid.UUID, optional – An UUID that identifies all binaries executed by a single run of this project. In the database schema this is named the ‘run_group’.

compiler_extension

Callable[str, iterable[str], RunInfo], optional – A composable extension that will be used in place of the real compiler. Defaults to running the compiler with a timeout command wrapped around it.

runtime_extension

Callable[str, iterable[str], RunInfo], optional – A composable extension that will be used in place of any binary this project wants to execute. Which binaries to replace is defined by the implementation using benchbuild.utils.wrapping.wrap. Defaults to None.

CONTAINER = None

DOMAIN = None

GROUP = None

NAME = None

SRC_FILE = None

VERSION = None

clean()

Clean the project build directory.

clone()

Create a deepcopy of ourself.

compile()

Compile the project.

download(version=None)

Auto-generated by with_* decorators.

id

prepare()

Prepare the build diretory.

redirect()

Redirect execution to a containerized benchbuild instance.

run()

Run the tests of this project.

This method initializes the default environment and takes care of cleaning up the mess we made, after a successfull run.

Parameters:experiment – The experiment we run this project under

run_tests(runner)

Run the tests of this project.

Clients override this method to provide customized run-time tests.

Parameters:

• experiment – The experiment we run this project under
• run – A function that takes the run command.

class benchbuild.project.ProjectDecorator(name, bases, attrs)

Bases: benchbuild.project.ProjectRegistry

Decorate the interface of a project with the in_builddir decorator.

This is just a small safety net for benchbuild users, because we make sure to run every project method in the project’s build directory.

decorated_methods = ['redirect', 'compile', 'run_tests']

class benchbuild.project.ProjectRegistry(name, bases, attrs)

Bases: type

Registry for benchbuild projects.

projects = Trie((('info/gentoo', <class 'benchbuild.projects.gentoo.info.Info'>), ('nw/rodinia', <class 'benchbuild.projects.apollo.rodinia.NW'>), ('nussinov/polybench', <class 'benchbuild.projects.polybench.polybench.Nussinov'>), ('nussinov/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Nussinov'>), ('xz/benchbuild', <class 'benchbuild.projects.benchbuild.xz.XZ'>), ('xz/gentoo', <class 'benchbuild.projects.gentoo.xz.XZ'>), ('ccrypt/benchbuild', <class 'benchbuild.projects.benchbuild.ccrypt.Ccrypt'>), ('fib/bots', <class 'benchbuild.projects.benchbuild.bots.Fib'>), ('povray/benchbuild', <class 'benchbuild.projects.benchbuild.povray.Povray'>), ('bzip2/benchbuild', <class 'benchbuild.projects.benchbuild.bzip2.Bzip2'>), ('bzip2/gentoo', <class 'benchbuild.projects.gentoo.bzip2.BZip2'>), ('ludcmp/polybench', <class 'benchbuild.projects.polybench.polybench.LuDCMP'>), ('ludcmp/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.LuDCMP'>), ('lulesh/benchbuild', <class 'benchbuild.projects.benchbuild.lulesh.Lulesh'>), ('leukocyte/rodinia', <class 'benchbuild.projects.apollo.rodinia.Leukocyte'>), ('MultiSourceApplications/lnt', <class 'benchbuild.projects.lnt.lnt.MultiSourceApplications'>), ('p7zip/gentoo', <class 'benchbuild.projects.gentoo.sevenz.SevenZip'>), ('seidel-2d/polybench', <class 'benchbuild.projects.polybench.polybench.Seidel2D'>), ('seidel-2d/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Seidel2D'>), ('strassen/bots', <class 'benchbuild.projects.benchbuild.bots.Strassen'>), ('atax/polybench', <class 'benchbuild.projects.polybench.polybench.Atax'>), ('atax/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Atax'>), ('floyd-warshall/polybench', <class 'benchbuild.projects.polybench.polybench.FloydWarshall'>), ('floyd-warshall/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.FloydWarshall'>), ('sparselu/bots', <class 'benchbuild.projects.benchbuild.bots.SparseLU'>), ('adi/polybench', <class 'benchbuild.projects.polybench.polybench.Adi'>), ('adi/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Adi'>), ('b+tree/rodinia', <class 'benchbuild.projects.apollo.rodinia.BPlusTree'>), ('mcrypt/benchbuild', <class 'benchbuild.projects.benchbuild.mcrypt.MCrypt'>), ('syr2k/polybench', <class 'benchbuild.projects.polybench.polybench.Syr2k'>), ('syr2k/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Syr2k'>), ('fdtd-2d/polybench', <class 'benchbuild.projects.polybench.polybench.FDTD2D'>), ('fdtd-2d/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.FDTD2D'>), ('SingleSourceBenchmarks/lnt', <class 'benchbuild.projects.lnt.lnt.SingleSourceBenchmarks'>), ('symm/polybench', <class 'benchbuild.projects.polybench.polybench.Symm'>), ('symm/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Symm'>), ('lulesh-omp/benchbuild', <class 'benchbuild.projects.benchbuild.lulesh.LuleshOMP'>), ('ruby/benchbuild', <class 'benchbuild.projects.benchbuild.ruby.Ruby'>), ('3mm/polybench', <class 'benchbuild.projects.polybench.polybench.ThreeMM'>), ('3mm/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.ThreeMM'>), ('SPEC2006/lnt', <class 'benchbuild.projects.lnt.lnt.SPEC2006'>), ('Povray/lnt', <class 'benchbuild.projects.lnt.lnt.Povray'>), ('knapsack/bots', <class 'benchbuild.projects.benchbuild.bots.Knapsack'>), ('bicg/polybench', <class 'benchbuild.projects.polybench.polybench.BicG'>), ('bicg/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.BicG'>), ('trisolv/polybench', <class 'benchbuild.projects.polybench.polybench.Trisolv'>), ('trisolv/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Trisolv'>), ('durbin/polybench', <class 'benchbuild.projects.polybench.polybench.Durbin'>), ('durbin/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Durbin'>), ('python/benchbuild', <class 'benchbuild.projects.benchbuild.python.Python'>), ('gzip/benchbuild', <class 'benchbuild.projects.benchbuild.gzip.Gzip'>), ('gzip/gentoo', <class 'benchbuild.projects.gentoo.gzip.GZip'>), ('2mm/polybench', <class 'benchbuild.projects.polybench.polybench.TwoMM'>), ('2mm/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.TwoMM'>), ('fft/bots', <class 'benchbuild.projects.benchbuild.bots.FFT'>), ('particlefilter/rodinia', <class 'benchbuild.projects.apollo.rodinia.ParticleFilter'>), ('crafty/benchbuild', <class 'benchbuild.projects.benchbuild.crafty.Crafty'>), ('crafty/gentoo', <class 'benchbuild.projects.gentoo.crafty.Crafty'>), ('pathfinder/rodinia', <class 'benchbuild.projects.apollo.rodinia.PathFinder'>), ('backprop/rodinia', <class 'benchbuild.projects.apollo.rodinia.Backprop'>), ('hotspot/rodinia', <class 'benchbuild.projects.apollo.rodinia.Hotspot'>), ('openblas/benchbuild', <class 'benchbuild.projects.benchbuild.lapack.OpenBlas'>), ('test-fail/test', <class 'benchbuild.projects.test.test.TestProjectRuntimeFail'>), ('nn/rodinia', <class 'benchbuild.projects.apollo.rodinia.NN'>), ('eix/gentoo', <class 'benchbuild.projects.gentoo.eix.Eix'>), ('syrk/polybench', <class 'benchbuild.projects.polybench.polybench.Syrk'>), ('syrk/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Syrk'>), ('bfs/rodinia', <class 'benchbuild.projects.apollo.rodinia.BFS'>), ('linpack/benchbuild', <class 'benchbuild.projects.benchbuild.linpack.Linpack'>), ('ffmpeg/benchbuild', <class 'benchbuild.projects.benchbuild.ffmpeg.LibAV'>), ('hotspot3D/rodinia', <class 'benchbuild.projects.apollo.rodinia.Hotspot3D'>), ('streamcluster/rodinia', <class 'benchbuild.projects.apollo.rodinia.StreamCluster'>), ('covariance/polybench', <class 'benchbuild.projects.polybench.polybench.Covariance'>), ('covariance/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Covariance'>), ('alignment/bots', <class 'benchbuild.projects.benchbuild.bots.Alignment'>), ('gemm/polybench', <class 'benchbuild.projects.polybench.polybench.Gemm'>), ('gemm/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Gemm'>), ('Rasdaman/benchbuild', <class 'benchbuild.projects.benchbuild.rasdaman.Rasdaman'>), ('floorplan/bots', <class 'benchbuild.projects.benchbuild.bots.FloorPlan'>), ('lud/rodinia', <class 'benchbuild.projects.apollo.rodinia.LUD'>), ('7z/benchbuild', <class 'benchbuild.projects.benchbuild.sevenz.SevenZip'>), ('jacobi-1d/polybench', <class 'benchbuild.projects.polybench.polybench.Jacobi1D'>), ('jacobi-1d/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Jacobi1D'>), ('lammps/benchbuild', <class 'benchbuild.projects.benchbuild.lammps.Lammps'>), ('lammps/gentoo', <class 'benchbuild.projects.gentoo.lammps.Lammps'>), ('trmm/polybench', <class 'benchbuild.projects.polybench.polybench.Trmm'>), ('trmm/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Trmm'>), ('lapack/benchbuild', <class 'benchbuild.projects.benchbuild.lapack.Lapack'>), ('cholesky/polybench', <class 'benchbuild.projects.polybench.polybench.Cholesky'>), ('cholesky/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Cholesky'>), ('heat-3d/polybench', <class 'benchbuild.projects.polybench.polybench.Heat3D'>), ('heat-3d/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Heat3D'>), ('libressl/benchbuild', <class 'benchbuild.projects.benchbuild.openssl.LibreSSL'>), ('doitgen/polybench', <class 'benchbuild.projects.polybench.polybench.Doitgen'>), ('doitgen/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Doitgen'>), ('sort/bots', <class 'benchbuild.projects.benchbuild.bots.Sort'>), ('postgresql/gentoo', <class 'benchbuild.projects.gentoo.postgresql.Postgresql'>), ('mvt/polybench', <class 'benchbuild.projects.polybench.polybench.Mvt'>), ('mvt/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Mvt'>), ('myocyte/rodinia', <class 'benchbuild.projects.apollo.rodinia.Myocyte'>), ('tcc/benchbuild', <class 'benchbuild.projects.benchbuild.tcc.TCC'>), ('correlation/polybench', <class 'benchbuild.projects.polybench.polybench.Correlation'>), ('correlation/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Correlation'>), ('gemver/polybench', <class 'benchbuild.projects.polybench.polybench.Gemver'>), ('gemver/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Gemver'>), ('kmeans/rodinia', <class 'benchbuild.projects.apollo.rodinia.KMeans'>), ('lu/polybench', <class 'benchbuild.projects.polybench.polybench.Lu'>), ('lu/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Lu'>), ('nqueens/bots', <class 'benchbuild.projects.benchbuild.bots.NQueens'>), ('uts/bots', <class 'benchbuild.projects.benchbuild.bots.UTS'>), ('js/benchbuild', <class 'benchbuild.projects.benchbuild.js.SpiderMonkey'>), ('deriche/polybench', <class 'benchbuild.projects.polybench.polybench.Deriche'>), ('deriche/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Deriche'>), ('x264/benchbuild', <class 'benchbuild.projects.benchbuild.x264.X264'>), ('x264/gentoo', <class 'benchbuild.projects.gentoo.x264.X264'>), ('sqlite3/benchbuild', <class 'benchbuild.projects.benchbuild.sqlite3.SQLite3'>), ('cfd/rodinia', <class 'benchbuild.projects.apollo.rodinia.CFD'>), ('minisat/benchbuild', <class 'benchbuild.projects.benchbuild.minisat.Minisat'>), ('scimark/apollo', <class 'benchbuild.projects.apollo.scimark.SciMark'>), ('heartwall/rodinia', <class 'benchbuild.projects.apollo.rodinia.HeartWall'>), ('crocopat/benchbuild', <class 'benchbuild.projects.benchbuild.crocopat.Crocopat'>), ('jacobi-2d/polybench', <class 'benchbuild.projects.polybench.polybench.Jacobi2Dimper'>), ('jacobi-2d/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Jacobi2Dimper'>), ('srad-1/rodinia', <class 'benchbuild.projects.apollo.rodinia.SRAD1'>), ('gramschmidt/polybench', <class 'benchbuild.projects.polybench.polybench.Gramschmidt'>), ('gramschmidt/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Gramschmidt'>), ('MultiSourceBenchmarks/lnt', <class 'benchbuild.projects.lnt.lnt.MultiSourceBenchmarks'>), ('leveldb/benchbuild', <class 'benchbuild.projects.benchbuild.leveldb.LevelDB'>), ('lavaMD/rodinia', <class 'benchbuild.projects.apollo.rodinia.LavaMD'>), ('srad-2/rodinia', <class 'benchbuild.projects.apollo.rodinia.SRAD2'>), ('test/test', <class 'benchbuild.projects.test.test.TestProject'>), ('gesummv/polybench', <class 'benchbuild.projects.polybench.polybench.Gesummv'>), ('gesummv/polybench-mod', <class 'benchbuild.projects.polybench.polybench-mod.Gesummv'>), ('health/bots', <class 'benchbuild.projects.benchbuild.bots.Health'>), ('sdcc/benchbuild', <class 'benchbuild.projects.benchbuild.sdcc.SDCC'>),))

benchbuild.project.populate(projects_to_filter=None, group=None)

Populate the list of projects that belong to this experiment.

Parameters:

• projects_to_filter (list(Project)) – List of projects we want to assign to this experiment. We intersect the list of projects with the list of supported projects to get the list of projects that belong to this experiment.
• group (list(str)) – In addition to the project filter, we provide a way to filter whole groups.