benchbuild.experiments package

Experiments module.

By default, only experiments that are listed in the configuration are loaded automatically. See configuration variables:

*_PLUGINS_AUTOLOAD *_PLUGINS_EXPERIMENTS

benchbuild.experiments.discover()

Import all experiments listed in PLUGINS_EXPERIMENTS.

Tests:

>>> from benchbuild.settings import CFG
>>> from benchbuild.experiments import discover
>>> import logging as lg
>>> import sys
>>> l = lg.getLogger('benchbuild')
>>> lg.getLogger('benchbuild').setLevel(lg.DEBUG)
>>> lg.getLogger('benchbuild').handlers = [lg.StreamHandler(stream=sys.stdout)]
>>> CFG["plugins"]["experiments"] = ["benchbuild.non.existing", "benchbuild.experiments.raw"]
>>> discover()
Could not find 'benchbuild.non.existing'
ImportError: No module named 'benchbuild.non'

Subpackages

• benchbuild.experiments.polly package
  • Submodules
  • benchbuild.experiments.polly.openmp module
    • Measurements
  • benchbuild.experiments.polly.openmpvect module
    • Measurements
  • benchbuild.experiments.polly.polly module
    • The ‘polly’ Experiment
      • Measurements
  • benchbuild.experiments.polly.pollyperformance module
    • The ‘polly’ Experiment
      • Measurements
  • benchbuild.experiments.polly.vectorize module
    • The ‘polly-vectorize’ Experiment
      • Measurements

Submodules

benchbuild.experiments.compilestats module

The ‘compilestats’ experiment.

This experiment is a basic experiment in the benchbuild study. It simply runs all projects after compiling it with -O3 and catches all statistics emitted by llvm.

class benchbuild.experiments.compilestats.CompilestatsExperiment(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The compilestats experiment.

NAME = 'cs'

actions_for_project(project)

class benchbuild.experiments.compilestats.PollyCompilestatsExperiment(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The compilestats experiment with polly enabled.

NAME = 'p-cs'

actions_for_project(project)

benchbuild.experiments.empty module

The ‘empty’ Experiment.

This experiment is for debugging purposes. It only prepares the basic directories for benchbuild. No compilation & no run can be done with it.

class benchbuild.experiments.empty.Empty(projects=None, group=None)

Bases: benchbuild.experiment.Experiment

The empty experiment.

NAME = 'empty'

actions_for_project(project)

Do nothing.

class benchbuild.experiments.empty.NoMeasurement(projects=None, group=None)

Bases: benchbuild.experiment.Experiment

Run everything but do not measure anything.

NAME = 'no-measurement'

actions_for_project(project)

Execute all actions but don’t do anything as extension.

benchbuild.experiments.papi module

PAPI based experiments.

These types of experiments (papi & papi-std) need to instrument the project with libbenchbuild support to work.

class benchbuild.experiments.papi.Analyze(project_or_experiment, action_fn=None)

Bases: benchbuild.utils.actions.Step

DESCRIPTION = 'Analyze the experiment after completion.'

NAME = 'ANALYZE'

class benchbuild.experiments.papi.Calibrate(project_or_experiment, action_fn=None)

Bases: benchbuild.utils.actions.Step

DESCRIPTION = 'Calibrate libpapi measurement functions.'

NAME = 'CALIBRATE'

class benchbuild.experiments.papi.PapiScopCoverage(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

PAPI-based dynamic SCoP coverage measurement.

NAME = 'papi'

actions()

Do the postprocessing, after all projects are done.

actions_for_project(project)

Create & Run a papi-instrumented version of the project.

This experiment uses the -jitable flag of libPolyJIT to generate dynamic SCoP coverage.

class benchbuild.experiments.papi.PapiStandardScopCoverage(projects=None, group=None)

Bases: benchbuild.experiments.papi.PapiScopCoverage

PAPI Scop Coverage, without JIT.

NAME = 'papi-std'

actions_for_project(project)

Create & Run a papi-instrumented version of the project.

This experiment uses the -jitable flag of libPolyJIT to generate dynamic SCoP coverage.

benchbuild.experiments.pj_sequence module

The ‘sequence analysis’ experiment suite.

Each experiment generates sequences of flags for a compiler command using an algorithm that calculates a best sequence in its own way. For calculating the value of a sequence (called fitness) regions and scops are being compared to each other and together generate the fitness value of a sequence. The used metric depends on the experiment, the fitness is being calculated for.

The fittest generated sequences and the compilestats of the whole progress are then written into a persisted data base for further analysis.

class benchbuild.experiments.pj_sequence.FindFittestSequenceGenetic1(project, experiment, *extensions, config=None)

Bases: benchbuild.extensions.RuntimeExtension

class benchbuild.experiments.pj_sequence.FindFittestSequenceGenetic2(project, experiment, *extensions, config=None)

Bases: benchbuild.extensions.RuntimeExtension

class benchbuild.experiments.pj_sequence.FindFittestSequenceGreedy(project, experiment, *extensions, config=None)

Bases: benchbuild.extensions.RuntimeExtension

class benchbuild.experiments.pj_sequence.FindFittestSequenceHillclimber(project, experiment, *extensions, config=None)

Bases: benchbuild.extensions.RuntimeExtension

class benchbuild.experiments.pj_sequence.Genetic1Sequence(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

This experiment is part of the sequence generating suite.

The sequences for Poly are getting generated using the first of two genetic algorithms. Only the compilestats are getting written into a database for further analysis.

NAME = 'pj-seq-genetic1-opt'

actions_for_project(project)

Execute the actions for the test.

class benchbuild.experiments.pj_sequence.Genetic2Sequence(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that excecutes all projects with PolyJIT support.

It is part of the sequence generating experiment suite.

The sequences are getting generated for Poly using another than the first genetic algorithm. The compilestats are getting written into a database for further analysis.

NAME = 'pj-seq-genetic2-opt'

actions_for_project(project)

Execute the actions for the test.

class benchbuild.experiments.pj_sequence.GreedySequences(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

This experiment is part of the sequence generating experiment suite.

Instead of the actual actions the compile stats for executing them are being written into the database. The sequences are getting generated with the greedy algorithm. This shall become the default experiment for sequence analysis.

NAME = 'pj-seq-greedy'

actions_for_project(project)

Execute the actions for the test.

class benchbuild.experiments.pj_sequence.HillclimberSequences(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

This experiment is part of the sequence generating suite.

The sequences for poly are getting generated using a hillclimber algorithm. The ouptut gets thrown away and the statistics of the compiling are written into a database to be analyzed later on.

NAME = 'pj-seq-hillclimber'

actions_for_project(project)

Execute the actions for the test.

class benchbuild.experiments.pj_sequence.RunSequence(project, experiment, *extensions, config=None)

Bases: benchbuild.extensions.ExtractCompileStats

Execute and compile a given sequence, to calculate its fitness value with a given function and metric.

class benchbuild.experiments.pj_sequence.SequenceReport(exp_name, exp_ids, out_path)

Bases: benchbuild.reports.Report

Handles the view of the sequences in the database.

QUERY_TOTAL = <sqlalchemy.sql.selectable.Select at 0x7fe40cf694a8; Select object>

SUPPORTED_EXPERIMENTS = ['pj-seq-hillclimber', 'pj-seq-genetic1-opt', 'pj-seq-genetic2-opt', 'pj-seq-greedy']

generate()

Generates the output of what is written in the database.

report()

benchbuild.experiments.pj_sequence.create_ir()

Read out the ir to compare it before and after adding a flag from the pass to the sequence or work with its returnings.

benchbuild.experiments.pj_sequence.filter_compiler_commandline(cmd, predicate=<function <lambda>>)

Filter unnecessary arguments for the compiler.

benchbuild.experiments.pj_sequence.filter_invalid_flags(item)

Filter our all flags not needed for getting the compilestats.

benchbuild.experiments.pj_sequence.get_args(cmd)

Returns the arguments of a command. Asserts if the given command is not part of the experiment. Args:

cmd: The clang command of which the arguments are returned.

benchbuild.experiments.pj_sequence.get_defaults()

Return the defaults for the experiment.

benchbuild.experiments.pj_sequence.get_genetic_defaults()

Returns the needed defaults for the genetic algorithms.

benchbuild.experiments.pj_sequence.link_ir(run_f)

Connect the intermediate representation of llvm with the files that are to be compiled.

benchbuild.experiments.pj_sequence.persist_sequence(run, sequence, fitness_val)

Persist the sequence and its fitness value in the database.

Args:

run: The current run we are attached to, with all its information. sequence: The fittest sequence generated by an algorithm. fitness_val: The fittest algorithm generated by an algorithm.

benchbuild.experiments.pj_sequence.set_args(cmd, new_args)

Sets the arguments of a command. Also asserts if the command is empty. Args:

cmd: The clang command that is getting its arguments set. new_args: The new additional arguments of the command.

benchbuild.experiments.pj_sequence.unique_compiler_cmds(run_f)

Verifys that compiler comands are not excecuted twice.

benchbuild.experiments.pjtest module

A test experiment for PolyJIT.

This experiment should only be used to test various features of PolyJIT. It provides only 1 configuration (maximum number of cores) and tests 2 run-time execution profiles of PolyJIT:

1. PolyJIT enabled, with specialization
2. PolyJIT enabled, without specialization

class benchbuild.experiments.pjtest.EnableDBExport(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

Call the child extensions with an activated PolyJIT.

class benchbuild.experiments.pjtest.JitExportGeneratedCode(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that executes all projects with PolyJIT support.

This is our default experiment for speedup measurements.

NAME = 'pj-db-export'

actions_for_project(project)

class benchbuild.experiments.pjtest.Test(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that executes all projects with PolyJIT support.

This is our default experiment for speedup measurements.

NAME = 'pj-test'

actions_for_project(project)

class benchbuild.experiments.pjtest.TestReport(exp_name, exp_ids, out_path)

Bases: benchbuild.reports.Report

Writes report to the database.

QUERY_REGION = <sqlalchemy.sql.selectable.Select at 0x7fe40cada438; Select object>

QUERY_TOTAL = <sqlalchemy.sql.selectable.Select at 0x7fe40cada1d0; Select object>

SUPPORTED_EXPERIMENTS = ['pj-test']

generate()

report()

sa = <module 'sqlalchemy' from '/home/docs/checkouts/readthedocs.org/user_builds/pprof-study/envs/v2.0.3/lib/python3.5/site-packages/sqlalchemy/__init__.py'>

benchbuild.experiments.pollytest module

The ‘pollytest’ experiment.

This experiment uses four different configs to analyse the compilestats’ and the time’s behavior regarding the position of polly and unprofitable processes.

class benchbuild.experiments.pollytest.PollyTest(projects=None, group=None)

Bases: benchbuild.experiment.Experiment

An experiment that executes projects with different configurations.

The time and the compilestats are collected.

NAME = 'pollytest'

actions_for_project(project)

class benchbuild.experiments.pollytest.PollyTestReport(exp_name, exp_ids, out_path)

Bases: benchbuild.reports.Report

QUERY_EVAL = <sqlalchemy.sql.selectable.Select at 0x7fe40ca5b208; Select object>

SUPPORTED_EXPERIMENTS = ['pollytest']

generate()

report()

benchbuild.experiments.polyjit module

The ‘polyjit’ experiment.

This experiment uses likwid to measure the performance of all binaries when running with polyjit support enabled.

class benchbuild.experiments.polyjit.ClearPolyJITConfig(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

class benchbuild.experiments.polyjit.DisableDelinearization(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

Deactivate the JIT for the following extensions.

class benchbuild.experiments.polyjit.DisablePolyJIT(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

Deactivate the JIT for the following extensions.

class benchbuild.experiments.polyjit.EnableJITDatabase(*args, project=None, experiment=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

The run and given extensions store polli’s statistics to the database.

class benchbuild.experiments.polyjit.EnablePolyJIT(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

Call the child extensions with an activated PolyJIT.

class benchbuild.experiments.polyjit.EnablePolyJIT_Opt(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.Extension

Call the child extensions with an activated PolyJIT.

class benchbuild.experiments.polyjit.PolyJIT(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The polyjit experiment.

actions_for_project(project)

classmethod init_project(project)

Execute the benchbuild experiment.

We perform this experiment in 2 steps:

1. with likwid disabled.
2. with likwid enabled.

Args:

project: The project we initialize.

Returns:

The initialized project.

class benchbuild.experiments.polyjit.PolyJITConfig

Bases: object

Object that stores the configuraion of the JIT.

argv

Getter for the configuration held by the config object.

clear()

value_to_str(key)

Prints the value of a given key.

class benchbuild.experiments.polyjit.PolyJITFull(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

An experiment that executes all projects with PolyJIT support.

This is our default experiment for speedup measurements.

NAME = 'pj'

actions_for_project(project)

class benchbuild.experiments.polyjit.PolyJITSimple(projects=None, group=None)

Bases: benchbuild.experiments.polyjit.PolyJIT

Simple runtime-testing with PolyJIT.

NAME = 'pj-simple'

actions_for_project(project)

class benchbuild.experiments.polyjit.RegisterPolyJITLogs(*extensions, config=None, **kwargs)

Bases: benchbuild.experiments.polyjit.PolyJITConfig, benchbuild.extensions.LogTrackingMixin, benchbuild.extensions.Extension

Extends the following RunWithTime extensions with extra PolyJIT logs.

benchbuild.experiments.polyjit.verbosity_to_polyjit_log_level(verbosity: int)

Transfers the verbosity level to a useable polyjit format.

benchbuild.experiments.raw module

The ‘raw’ Experiment.

This experiment is the basic experiment in the benchbuild study. It simply runs all projects after compiling it with -O3. The binaries are wrapped with the time command and results are written to the database.

This forms the baseline numbers for the other experiments.

Measurements

3 Metrics are generated during this experiment:

time.user_s - The time spent in user space in seconds (aka virtual time) time.system_s - The time spent in kernel space in seconds (aka system time) time.real_s - The time spent overall in seconds (aka Wall clock)

class benchbuild.experiments.raw.RawRuntime(projects=None, group=None)

Bases: benchbuild.experiment.RuntimeExperiment

The polyjit experiment.

NAME = 'raw'

actions_for_project(project)

Compile & Run the experiment with -O3 enabled.