NAME
   Benchmark::Lab - Tools for structured benchmarking and profiling

VERSION
   version 0.001

EXPERIMENTAL
   This module is still in the early experiment stage. Breaking API changes
   could occur in any release before 1.000.

   Use and feedback is welcome if you are willing to accept that risk.

SYNOPSIS
       # Load as early as possible in case you want profiling
       use Benchmark::Lab -profile => $ENV{DO_PROFILING};

       # Define a task to benchmark as functions in a namespace
       package My::Task;

       # do once before any iterations (not timed)
       sub setup {
           my $context = shift;
           ...
       }

       # do before every iteration (not timed)
       sub before_task {
           my $context = shift;
           ...
       }

       # task being iterated and timed
       sub do_task {
           my $context = shift;
           ...
       }

       # do after every iteration (not timed)
       sub after_task {
           my $context = shift;
           ...
       }

       # do once after all iterations (not timed)
       sub teardown {
           my $context = shift;
           ...
       }

       # Run benchmarks on a namespace
       package main;

       my $context = {}; # any data needed

       my $result = Benchmark::Lab->new()->start( 'My::Task', $context );

       # XXX ... do stuff with results ...

DESCRIPTION
   This module provides a harness to benchmark and profile structured
   tasks.

   Structured tasks include a task to be benchmarked, as well as work to be
   done to prepare or cleanup from benchmarking that should not be timed.

   This module also allows the same structured task to be profiled with
   Devel::NYTProf, again with only the task under investigation being
   profiled. During prep/cleanup work, the profiler is paused.

   On systems that support "Time::HiRes::clock_gettime" and
   "CLOCK_MONOTONIC", those will be used for timing. On other systems, the
   less precise and non-monotonic "Time::HiRes::time" function is used
   instead.

   Future versions will add features for analyzing and comparing benchmarks
   timing data.

USAGE
 Loading and initializing
   If you want to use the profiling feature, you MUST load this module as
   early as possible so that Devel::NYTProf can instrument all subsequent
   compiled code.

   To correctly initialize "Benchmark::Lab" (and possibly Devel::NYTProf),
   you MUST ensure its "import" method is called. (Loading it with "use" is
   sufficient.)

   Here is an example that toggles profiling based on an environment
   variable:

       use Benchmark::Lab -profile => $ENV{DO_PROFILING};

       # loading other modules is now OK
       use File::Spec;
       use HTTP::Tiny;
       ...

 Creating a structured task
   A structured task is a Perl namespace that implements some of the
   following *task phases* by providing a subroutine with the corresponding
   name:

   *   "setup" – run once before any iteration begins (not timed)

   *   "before_task" – run before *each* "do_task" function (not timed)

   *   "do_task" – specific task being benchmarked (timed)

   *   "after_task" – run after *each* "do_task" function (not timed)

   *   "teardown" – run after all iterations are finished (not timed)

   Each task phase will be called with a *context object*, which can be
   used to pass data across phases.

       package Foo;

       sub setup {
           my $context = shift;
           $context->{filename} = "foobar.txt";
           path($context->{filename})->spew_utf8( _test_data() );
       }

       sub do_task {
           my $context = shift;
           my $file = $context->{filename};
           # ... do stuff with $file
       }

   Because structured tasks are Perl namespaces, you can put them into .pm
   files and load them like modules. Or, you can define them on the fly.

   Also, since "Benchmark::Lab" finds task phase functions with the "can"
   method, you can use regular Perl inheritance with @ISA to reuse
   setup/teardown/etc. task phases for related "do_task" functions.

       package Foo::Base;

       sub setup { ... }
       sub teardown { ... }

       package Foo::Case1

       use parent 'Foo::Base';
       sub do_task { ... }

       package Foo::Case2

       use parent 'Foo::Base';
       sub do_task { ... }

 Running benchmarks
   A "Benchmark::Lab" object defines the conditions of the test – currently
   just the constraints on the number of iterations or duration of the
   benchmarking run.

   Running a benchmark is just a matter of specifying the namespace for the
   task phase functions, and a context object, if desired.

       use Benchmark::Lab -profile => $ENV{DO_PROFILE};

       sub fact { my $n = int(shift); return $n == 1 ? 1 : $n * fact( $n - 1 ) }

       *Fact::do_task = sub {
           my $context = shift;
           fact( $context->{n} );
       };

       my $bl      = Benchmark::Lab->new;
       my $context = { n => 25 };
       my $res     = $bl->start( "Fact", $context );

       printf( "Median rate: %d/sec\n", $res->{median_rate} );

 Analyzing results
   TBD. Analysis will be added in a future release.

METHODS
 new
   Returns a new Benchmark::Lab object.

   Valid attributes include:

   *   "min_secs" – minimum elapsed time in seconds; default 0

   *   "max_secs" – maximum elapsed time in seconds; default 300

   *   "min_reps" - minimum number of task repetitions; default 1; minimum
       1

   *   "max_reps" - maximum number of task repetitions; default 100

   *   "verbose" – when true, progress will be logged to STDERR; default
       false

   The logic for benchmark duration is as follows:

   *   benchmarking always runs until both "min_secs" and "min_reps" are
       satisfied

   *   when profiling, benchmarking stops after minimums are satisfied

   *   when not profiling, benchmarking stops once one of "max_secs" or
       "max_reps" is exceeded.

   Note that "elapsed time" for the "min_secs" and "max_secs" is wall-clock
   time, not the cumulative recorded time of the task itself.

 start
       my $result = $bm->start( $package, $context, $label );

   This method executes the structured benchmark from the given $package.
   The $context parameter is passed to all task phases. The $label is used
   for diagnostic output to describe the benchmark being run.

   If parameters are omitted, $package defaults to "main", an empty hash
   reference is used for the $context, and the $label defaults to the
   $package.

   It returns a hash reference with the following keys:

   *   "elapsed" – total wall clock time to execute the benchmark
       (including non-timed portions).

   *   "total_time" – sum of recorded task iterations times.

   *   "iterations" – total number of "do_task" functions called.

   *   "percentiles" – hash reference with 1, 5, 10, 25, 50, 75, 90, 95 and
       99th percentile iteration times. There may be duplicates if there
       were fewer than 100 iterations.

   *   "median_rate" – the inverse of the 50th percentile time.

   *   "timing" – array reference with individual iteration times as
       (floating point) seconds.

CAVEATS
   If the "do_task" executes in less time than the timer granularity, an
   error will be thrown. For benchmarks that do not have before/after
   functions, just repeating the function under test in "do_task" will be
   sufficient.

RATIONALE
   I believe most approaches to benchmarking are flawed, primarily because
   they focus on finding a *single* measurement. Single metrics are easy to
   grok and easy to compare ("foo was 13% faster than bar!"), but they
   obscure the full distribution of timing data and (as a result) are often
   unstable.

   Most of the time, people hand-wave this issue and claim that the Central
   Limit Theorem (CLT) solves the problem for a large enough sample size.
   Unfortunately, the CLT holds only if means and variances are finite and
   some real world distributions are not (e.g. hard drive error frequencies
   best fit a Pareto distribution).

   Further, we often care more about the shape of the distribution than
   just a single point. For example, I would rather have a process with
   mean µ that stays within 0.9µ - 1.1µ than one that varies from 0.5µ -
   1.5µ.

   And a process that is 0.1µ 90% of the time and 9.1µ 10% of the time
   (still with mean µ!) might be great or terrible, depending on the
   application.

   This module grew out of a desire for detailed benchmark timing data,
   plus some additional features, which I couldn't find in existing
   benchmarking modules:

   *   Raw timing data – I wanted to be able to get raw timing data, to
       allow more flexible statistical analysis of timing distributions.

   *   Monotonic clock – I wanted times from a high-resolution monotonic
       clock (if available).

   *   Setup/before/after/teardown – I wanted to be able to
       initialize/reset state not just once at the start, but before each
       iteration and without it being timed.

   *   Devel::NYTProf integration – I wanted to be able to run the exact
       same code I benchmarked through Devel::NYTProf, also limiting the
       profiler to the benchmark task alone, not the setup/teardown/etc.
       code.

   Eventually, I hope to add some more robust graphic visualization and
   statistical analyses of timing distributions. This might include both
   single-point estimates (like other benchmarking modules) but also more
   sophisticated metrics, like non-parametric measures for comparing
   samples with unequal variances.

SEE ALSO
   There are many benchmarking modules on CPAN with a mix of features that
   may be sufficient for your needs. To my knowledge, none give timing
   distributions or integrate with Devel::NYTProf.

   Here is a brief rundown of some that I am familiar with:

   *   Benchmark – ships with Perl, but makes it hard to get timing
       distributions in a usable form.

   *   Benchmark::Timer – times only parts of code, targets a degree of
       statistical confidence (assuming data is normally distributed).

   *   Dumbbench – attempts to improve on Benchmark with a more robust
       statistical estimation of runtimes; no before/after capabilities.

   *   Surveyor::App – also runs benchmarks from a package, but doesn't
       have before/after task capabilities and relies on Benchmark for
       timing.

SUPPORT
 Bugs / Feature Requests
   Please report any bugs or feature requests through the issue tracker at
   <https://github.com/dagolden/Benchmark-Lab/issues>. You will be notified
   automatically of any progress on your issue.

 Source Code
   This is open source software. The code repository is available for
   public review and contribution under the terms of the license.

   <https://github.com/dagolden/Benchmark-Lab>

     git clone https://github.com/dagolden/Benchmark-Lab.git

AUTHOR
   David Golden <[email protected]>

COPYRIGHT AND LICENSE
   This software is Copyright (c) 2016 by David Golden.

   This is free software, licensed under:

     The Apache License, Version 2.0, January 2004

You are viewing proxied material from ftp.icm.edu.pl. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.