benchbuild package

Subpackages

Submodules

benchbuild.bootstrap module

class benchbuild.bootstrap.BenchBuildBootstrap(executable)[source]

Bases: plumbum.cli.application.Application

Bootstrap benchbuild external dependencies, if possible.

main(*args)[source]
store_config

Sets an attribute

benchbuild.container module

class benchbuild.container.BashStrategy[source]

Bases: benchbuild.container.ContainerStrategy

The user interface for setting up a bash inside the container.

run(context)[source]
class benchbuild.container.Container(exe)[source]

Bases: plumbum.cli.application.Application

Manage uchroot containers.

VERSION = '2.0.1-$Id: b9b13c9242c155f6fda66e981d19e23e39941f36 $'
builddir(tmpdir)[source]

Set the current builddir of the container.

input_file(container)[source]

Find the input path of a uchroot container.

main(*args)[source]
mounts(user_mount)[source]

Save the current mount of the container into the settings.

output_file(container)[source]

Find and writes the output path of a chroot container.

shell(custom_shell)[source]

The command to run inside the container.

verbosity

Sets an attribute

class benchbuild.container.ContainerBootstrap(executable)[source]

Bases: plumbum.cli.application.Application

Check for the needed files.

install_cmake_and_exit()[source]

Tell the user to install cmake and aborts the current process.

main(*args)[source]
class benchbuild.container.ContainerCreate(executable)[source]

Bases: plumbum.cli.application.Application

Create a new container with a predefined strategy.

We offer a variety of creation policies for a new container. By default a basic ‘spawn a bash’ policy is used. This just leaves you inside a bash that is started in the extracted container. After customization you can exit the bash and pack up the result.

main(*args)[source]
strategy(strategy)[source]
class benchbuild.container.ContainerList(executable)[source]

Bases: plumbum.cli.application.Application

Prints a list of the known containers.

main(*args)[source]
class benchbuild.container.ContainerRun(executable)[source]

Bases: plumbum.cli.application.Application

Execute commannds inside a prebuilt container.

main(*args)[source]
class benchbuild.container.ContainerStrategy[source]

Bases: object

Interfaces for the different containers chosen by the experiment.

run(context)[source]
class benchbuild.container.MockObj(**kwargs)[source]

Bases: object

class benchbuild.container.SetupPolyJITGentooStrategy[source]

Bases: benchbuild.container.ContainerStrategy

Interface of using gentoo as a container for an experiment.

configure()[source]

Configure the gentoo container for a PolyJIT experiment.

run(context)[source]

Setup a gentoo container suitable for PolyJIT.

write_bashrc(path)[source]

Write inside a bash and update the shell if necessary.

write_layout(path)[source]

Create a layout from the given path.

write_makeconfig(path)[source]

Create the stringed to be written in the settings.

write_wgetrc(path)[source]

Wget the project from a specified link.

benchbuild.container.clean_directories(builddir, in_dir=True, out_dir=True)[source]

Remove the in and out of the container if confirmed by the user.

benchbuild.container.find_hash(container_db, key)[source]

Find the first container in the database with the given key.

benchbuild.container.main(*args)[source]
benchbuild.container.pack_container(in_container, out_file)[source]
benchbuild.container.run_in_container(command, container_dir, mounts)[source]

Run a given command inside a container.

Mounts a directory as a container at the given mountpoint and tries to run the given command inside the new container.

benchbuild.container.set_input_container(container, cfg)[source]

Save the input for the container in the configurations.

benchbuild.container.setup_bash_in_container(builddir, container, outfile, mounts, shell)[source]

Setup a bash environment inside a container.

Creates a new chroot, which the user can use as a bash to run the wanted projects inside the mounted container, that also gets returned afterwards.

benchbuild.container.setup_container(builddir, container)[source]

Prepare the container and returns the path where it can be found.

benchbuild.container.setup_directories(builddir)[source]

Create the in and out directories of the container.

benchbuild.driver module

class benchbuild.driver.PollyProfiling(executable)[source]

Bases: plumbum.cli.application.Application

Frontend for running/building the benchbuild study framework.

VERSION = '2.0.1-$Id: b9b13c9242c155f6fda66e981d19e23e39941f36 $'
debug

Sets an attribute

main(*args)[source]
verbosity

Sets an attribute

benchbuild.driver.main(*args)[source]

Main function.

benchbuild.experiment module

BenchBuild’s skeleton for experiments.

An benchbuild.experiment defines a series of phases that constitute a benchbuild compatible experiment. This is the default implementation of an experiment.

Clients can derive from class class::benchbuild.experiment.Experiment and override the methods relvant to their experiment.

An experiment can have a variable number of phases / steps / substeps.

Phases / Steps / Substeps

All phases/steps/substeps support being used as a context manager. All 3 of them catch ProcessExecutionErrors that may be thrown from plumbum, without aborting the whole experiment. However, an error is tracked.

Actions

An experiment performs the following actions in order:
  1. clean - Clean any previous runs that collide with our directory
  2. prepare - Prepare the experiment, this is a good place to copy relevant
    files over for testing.
  3. run (run_tests) - Run the experiment. The ‘meat’ lies here. Override
    This to perform all your experiment needs.
class benchbuild.experiment.Configuration(project=None, config=None)[source]

Bases: object

Build a set of experiment actions out of a list of configurations.

class benchbuild.experiment.Experiment(projects=None, group=None)[source]

Bases: object

A series of commands executed on a project that form an experiment.

The default implementation should provide a sane environment for all derivates.

One important task executed by the basic implementation is setting up the default set of projects that belong to this project. As every project gets registered in the ProjectFactory, the experiment gets a list of experiment names that work as a filter.

NAME = None
actions()[source]
actions_for_project(project)[source]

Get the actions a project wants to run.

Args:
project (benchbuild.Project): the project we want to run.
static default_compiletime_actions(project)[source]

Return a series of actions for a compile time experiment.

static default_runtime_actions(project)[source]

Return a series of actions for a run time experiment.

class benchbuild.experiment.ExperimentRegistry(name, bases, dict)[source]

Bases: type

Registry for benchbuild experiments.

experiments = {}
class benchbuild.experiment.RuntimeExperiment(projects=None, group=None)[source]

Bases: benchbuild.experiment.Experiment

Additional runtime only features for experiments.

get_papi_calibration(project, calibrate_call)[source]

Get calibration values for PAPI based measurements.

Args:
project (Project):
Unused (deprecated).
calibrate_call (benchbuild.utils.cmd):
The calibration command we will use.
persist_calibration(project, cmd, calibration)[source]

Persist the result of a calibration call.

Args:
project (benchbuild.Project):
The calibration values will be assigned to this project.
cmd (benchbuild.utils.cmd):
The command we used to generate the calibration values.
calibration (int):
The calibration time in nanoseconds.
benchbuild.experiment.get_group_projects(group: str, experiment) → typing.List[benchbuild.project.Project][source]

Get a list of project names for the given group.

Filter the projects assigned to this experiment by group.

Args:

group (str): The group. experiment (benchbuild.Experiment): The experiment we draw our projects

to filter from.
Returns (list):
A list of project names for the group that are supported by this experiment.

benchbuild.extensions module

Extension base-classes for compile-time and run-time experiments.

class benchbuild.extensions.Extension(*extensions, config=None, **kwargs)[source]

Bases: object

call_next(*args, **kwargs)[source]

Call all child extensions with the same arguments.

print(indent=0)[source]

Print a structural view of the registered extensions.

class benchbuild.extensions.ExtractCompileStats(project, experiment, *extensions, config=None)[source]

Bases: benchbuild.extensions.Extension

get_compilestats(prog_out)[source]

Get the LLVM compilation stats from :prog_out:.

class benchbuild.extensions.LogAdditionals(*extensions, config=None, **kwargs)[source]

Bases: benchbuild.extensions.Extension

Log any additional log files that were registered.

class benchbuild.extensions.LogTrackingMixin[source]

Bases: object

Add log-registering capabilities to extensions.

add_log(path)[source]
logs
class benchbuild.extensions.RunCompiler(project, experiment, *extensions, config=None)[source]

Bases: benchbuild.extensions.Extension

class benchbuild.extensions.RunWithTime(*extensions, config=None, **kwargs)[source]

Bases: benchbuild.extensions.Extension

Wrap a command with time and store the timings in the database.

class benchbuild.extensions.RunWithTimeout(*extensions, limit='10m', **kwargs)[source]

Bases: benchbuild.extensions.Extension

class benchbuild.extensions.RuntimeExtension(project, experiment, *extensions, config=None)[source]

Bases: benchbuild.extensions.Extension

class benchbuild.extensions.SetThreadLimit(*extensions, config=None, **kwargs)[source]

Bases: benchbuild.extensions.Extension

benchbuild.likwid module

Likwid helper functions.

Extract information from likwid’s CSV output.

benchbuild.likwid.fetch_cols(fstream, split_char=', ')[source]

Fetch columns from likwid’s output stream.

Args:
fstream: The filestream with likwid’s output. split_car (str): The character we split on, default ‘,’
Returns (list(str)):
A list containing the elements of fstream, after splitting at split_char.
benchbuild.likwid.get_likwid_perfctr(infile)[source]

Get a complete list of all measurements.

Args:
infile: The filestream containing all likwid output.
Returns:
A list of all measurements extracted from likwid’s file stream.
benchbuild.likwid.get_measurements(region, core_info, data, extra_offset=0)[source]

Get the complete measurement info from likwid’s region info.

Args:
region: The region we took a measurement in. core_info: The core information. data: The raw data. extra_offset (int): default = 0
Returns (list((region, metric, core, value))):
A list of measurement tuples, a tuple contains the information about the region, the metric, the core and the actual value.
benchbuild.likwid.read_struct(fstream)[source]

Read a likwid struct from the text stream.

Args:
fstream: Likwid’s filestream.
Returns (dict(str: str)):
A dict containing all likwid’s struct info as key/value pairs.
benchbuild.likwid.read_structs(fstream)[source]

Read all structs from likwid’s file stream.

Args:
fstream: Likwid’s output file stream.
Returns:
A generator that can be used to iterate over all structs in the fstream.
benchbuild.likwid.read_table(fstream)[source]

Read a likwid table info from the text stream.

Args:
fstream: Likwid’s filestream.
Returns (dict(str: str)):
A dict containing likwid’s table info as key/value pairs.
benchbuild.likwid.read_tables(fstream)[source]

Read all tables from likwid’s file stream.

Args:
fstream: Likwid’s output file stream.
Returns:
A generator that can be used to iterate over all tables in the fstream.

benchbuild.log module

Analyze the BB database.

class benchbuild.log.BenchBuildLog(executable)[source]

Bases: plumbum.cli.application.Application

Frontend command to the benchbuild database.

experiment(experiments)[source]

Set the experiments to fetch the log for.

experiment_ids(experiment_ids)[source]

Set the experiment ids to fetch the log for.

log_type(types)[source]

Set the output types to print.

main()[source]

Run the log command.

project(projects)[source]

Set the projects to fetch the log for.

project_ids(project_ids)[source]

Set the project ids to fetch the log for.

benchbuild.log.print_logs(query, types=None)[source]

Print status logs.

benchbuild.log.print_runs(query)[source]

Print all rows in this result query.

benchbuild.project module

Project handling for the benchbuild study.

class benchbuild.project.Project(exp, group: str = None)[source]

Bases: object

benchbuild’s Project class.

A project defines how run-time testing and cleaning is done for this
IR project
CONTAINER = <benchbuild.utils.container.Gentoo object>
DOMAIN = None
GROUP = None
NAME = None
SRC_FILE = None
VERSION = None
build()[source]

Build the project.

clean()[source]

Clean the project build directory.

clone()[source]

Create a deepcopy of ourself.

compiler_extension

Return the compiler extension registered to this project.

configure()[source]

Configure the project.

download()[source]

Download the input source for this project.

prepare()[source]

Prepare the build diretory.

run(experiment)[source]

Run the tests of this project.

This method initializes the default environment and takes care of cleaning up the mess we made, after a successfull run.

Args:
experiment: The experiment we run this project under
run_tests(experiment, run)[source]

Run the tests of this project.

Clients override this method to provide customized run-time tests.

Args:
experiment: The experiment we run this project under run: A function that takes the run command.
run_uuid

Get the UUID that groups all tests for one project run.

Args:
create_new: Create a fresh UUID (Default: False)
runtime_extension

Return the runtime extension registered for this project.

setup_derived_filenames()[source]

Construct all derived file names.

class benchbuild.project.ProjectDecorator(name, bases, attrs)[source]

Bases: benchbuild.project.ProjectRegistry

Decorate the interface of a project with the in_builddir decorator.

This is just a small safety net for benchbuild users, because we make sure to run every project method in the project’s build directory.

decorated_methods = ['build', 'configure', 'download', 'prepare', 'run_tests']
class benchbuild.project.ProjectRegistry(name, bases, attrs)[source]

Bases: type

Registry for benchbuild projects.

projects = Trie()
benchbuild.project.populate(projects_to_filter=None, group=None)[source]

Populate the list of projects that belong to this experiment.

Args:
projects_to_filter (list):
List of projects we want to assign to this experiment. We intersect the list of projects with the list of supported projects to get the list of projects that belong to this experiment.
group (str):
In addition to the project filter, we provide a way to filter whole groups.

benchbuild.report module

class benchbuild.report.BenchBuildReport(executable)[source]

Bases: plumbum.cli.application.Application

Generate Reports from the benchbuild db.

experiment_ids(ids)[source]
experiments(experiments)[source]
main(*args)[source]
outfile(outfile)[source]

benchbuild.run module

benchbuild’s run command.

This subcommand executes experiments on a set of user-controlled projects. See the output of benchbuild run –help for more information.

class benchbuild.run.BenchBuildRun(executable)[source]

Bases: plumbum.cli.application.Application

Frontend for running experiments in the benchbuild study framework.

experiment_tag(description)[source]
experiments(experiments)[source]
full()[source]
group(group)[source]
list_experiments()[source]
list_projects()[source]
main()[source]

Main entry point of benchbuild run.

pretend

Sets an attribute

projects(projects)[source]
show_config

Sets an attribute

store_config

Sets an attribute

benchbuild.run.print_projects(exp)[source]

Print a list of projects registered for that experiment.

Args:
exp: The experiment to print all projects for.

benchbuild.settings module

Settings module for benchbuild.

All settings are stored in a simple dictionary. Each setting should be modifiable via environment variable.

class benchbuild.settings.Configuration(parent_key, node=None, parent=None, init=True)[source]

Bases: object

Dictionary-like data structure to contain all configuration variables.

This serves as a configuration dictionary throughout benchbuild. You can use it to access all configuration options that are available. Whenever the structure is updated with a new subtree, all variables defined in the new subtree are updated from the environment.

Environment variables are generated from the tree paths automatically.
CFG[“build_dir”] becomes BB_BUILD_DIR CFG[“llvm”][“dir”] becomes BB_LLVM_DIR

The configuration can be stored/loaded as JSON.

Examples:
>>> from benchbuild import settings as s
>>> c = s.Configuration('bb')
>>> c['test'] = 42
>>> c['test']
BB_TEST=42
>>> str(c['test'])
'42'
>>> type(c['test'])
<class 'benchbuild.settings.Configuration'>
filter_exports()[source]
has_default()[source]

Check, if the node contains a ‘default’ value.

has_value()[source]

Check, if the node contains a ‘value’.

init_from_env()[source]

Initialize this node from environment.

If we’re a leaf node, i.e., a node containing a dictionary that consist of a ‘default’ key, compute our env variable and initialize our value from the environment. Otherwise, init our children.

is_leaf()[source]

Check, if the node is a ‘leaf’ node.

load(_from)[source]

Load the configuration dictionary from file.

store(config_file)[source]

Store the configuration dictionary to a file.

update(cfg_dict)[source]

Update the configuration dictionary with new content.

This just delegates the update down to the internal data structure. No validation is done on the format, be sure you know what you do.

Args:
cfg_dict: A configuration dictionary.
value()[source]

Return the node value, if we’re a leaf node.

Examples:
>>> from benchbuild import settings as s
>>> c = s.Configuration("test")
>>> c['x'] = { "y" : { "value" : None }, "z" : { "value" : 2 }}
>>> c['x']['y'].value() == None
True
>>> c['x']['z'].value()
2
>>> c['x'].value()
TEST_X_Y=null
TEST_X_Z=2
exception benchbuild.settings.InvalidConfigKey[source]

Bases: RuntimeWarning

Warn, if you access a non-existing key benchbuild’s configuration.

class benchbuild.settings.UUIDEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]

Bases: json.encoder.JSONEncoder

Encoder module for UUID objects.

default(o)[source]

Encode UUID objects as string.

benchbuild.settings.available_cpu_count()[source]

Get the number of available CPUs.

Number of available virtual or physical CPUs on this system, i.e. user/real as output by time(1) when called with an optimally scaling userspace-only program.

Returns:
Number of avaialable CPUs.
benchbuild.settings.escape_json(raw_str)[source]

Shell-Escape a json input string.

Args:
raw_str: The unescaped string.
benchbuild.settings.find_config(test_file=None, defaults=['.benchbuild.yml', '.benchbuild.yaml', '.benchbuild.json'], root='.')[source]

Find the path to the default config file.

We look at :root: for the :default: config file. If we can’t find it there we start looking at the parent directory recursively until we find a file named :default: and return the absolute path to it. If we can’t find anything, we return None.

Args:
default: The name of the config file we look for. root: The directory to start looking for.
Returns:
Path to the default config file, None if we can’t find anything.
benchbuild.settings.is_yaml(cfg_file)[source]
benchbuild.settings.to_env_dict(config)[source]

Convert configuration object to a flat dictionary.

benchbuild.settings.update_env()[source]

benchbuild.slurm module

Dump SLURM script that executes the selected experiment with all projects.

This basically provides the same as benchbuild run, except that it just dumps a slurm batch script that executes everything as an array job on a configurable SLURM cluster.

class benchbuild.slurm.Slurm(executable)[source]

Bases: plumbum.cli.application.Application

Generate a SLURM script.

experiment(cfg_experiment)[source]

Specify experiments to run

experiment_tag(description)[source]

A description for this experiment run

group(groups)[source]

Run a group of projects under the given experiments

main()[source]

Main entry point of benchbuild run.

projects(projects)[source]

Specify projects to run

benchbuild.test module

class benchbuild.test.BenchBuildTest(executable)[source]

Bases: plumbum.cli.application.Application

Create regression tests for polyjit from the measurements database.

get_check_line(name, module)[source]
main()[source]
opt_flags()[source]
prefix(prefix)[source]