NAME

   Parallel::ForkManager::Scaled - Run processes in parallel based on CPU
   usage

VERSION

   Version 0.17

SYNOPSIS

       use Parallel::ForkManager::Scaled;

       # my $pm = Parallel::ForkManager::Scaled->new( attrib => value, ... );
       my $pm = Parallel::ForkManager::Scaled->new;

       # Used just like Parallel::ForkManager, so I'll paraphrase its documentation

       for my $data (@all_data) {
           # $pid is set to the child process' PID
           my $pid = $pm->start and next;

           # In the child process now
           # do some work ..

           # Exit the child
           $pm->finish;
       }

DESCRIPTION

   This module inherits from Parallel::ForkManager and adds the ability to
   automatically manage the number of processes running based on how busy
   the system is by watching the CPU idle time. Each time a child is about
   to be start()ed a new value for max_procs may be calculated (if enough
   time has passed since the last calculation). If a new value is
   calculated, the number of processes to run will be adjusted by calling
   set_max_procs with the new value.

   Without specifying any attributes to the constructor, some defaults
   will be set for you (see Attributes below).

Attributes

   Attributes are just methods that may be passed to the constructor
   (new()) and most may be changed during the life of the returned object.
   They take as a parameter a new value to set for the attribute and
   return the current value (or new value if one was passed).

   hard_min_procs

     The number of running processes will never be adjusted lower than
     this value.

     default: 1

   hard_max_procs

     The number of running processes will never be adjusted higher than
     this value.

     default: The detected number of CPUs * 2

   soft_min_procs

   soft_max_procs

     This is initially set to hard_min_procs and hard_max_procs
     respectively and is adjusted over time. These are used when
     calculating adjustments as the minimum and maximum number of
     processes respectively.

     Over time soft_min_procs and soft_max_procs should approach the same
     value for a consistent workload and a machine not otherwise busy.

     Depending on the needs of the system, these values may also diverge
     if necessary to try to reach idle_target.

     You may adjust these values if you wish by passing a value to the
     method but you probably shouldn't. :)

   initial_procs (read-only)

     The number of processes to start running before attempting any
     adjustments, max_procs will be set to this value upon initialization.

     default: half way between soft_min_procs and soft_max_procs

   update_frequency

     The minimum amount of time, in seconds, that must elapse between
     checks of the system CPU's idle % and updates to the number of
     running processes.

     Set this to 0 to cause a check before each call to start().

     Before each call to start() the time is compared with the last time a
     check/update was performed. If this much time has passed, a new check
     will be made of how busy the CPU is and the number of processes may
     be adjusted.

     default: 1

   idle_target

     Percentage of CPU idle time to try to maintain by adjusting the
     number of running processes between hard_min_procs and hard_max_procs

     default: 0 # try to keep the CPU 100% busy (0% idle)

   idle_threshold

     Only make adjustments if the current CPU idle % is this distance away
     from idle_target. In other words, only adjust if abs(cur_idle -
     idle_target) > idle_threshold. This may be a fractional value
     (floating point).

     You may notce that the default idle_target of 0 and idle_threshold of
     1 would seem to indicate that the processes would never be adjusted
     as idle can never be less than 0%. At the limits, the threshold is
     adjusted so that we will still attempt adjustments, something like
     this:

         min_ok = max(0,   idle_target - idle_threshold)
         max_ok = min(100, idle_target - idle_threshold)

         adjust if idle >= max_ok
         adjust if idle <= min_ok

     default: 1

   run_on_update

     This is a callback function that is run immediately after (possibly)
     adjusting max_procs. This allows you to override the default behavior
     of this module for your own nefarious purposes.

     run_on_update expects a coderef which will be called with two
     parameters:

       * The object being adjusted. ($obj)

       * The old value for $obj->max_procs. If you decide you have a
       better idea of what max_procs should be, in your callback just set
       it via $obj->set_max_procs($new_value).

     The return value from the callback is ignored.

     Example:

       $pm->run_on_update( sub{
           my ($obj, $old_max_procs) = @_;
           $obj->set_max_procs($old_max_procs+1);
       });

   tempdir

     This is passed to the Parallel::ForkManager constructor to set
     tempdir. Where Parallel::ForkManager is constructed thusly:

       my $pm = Parallel::ForkManager->new($procs, $tempdir);

     The equivalent for this module would be:

       my $pm = Parallel::ForkManager::Scaled->new(initial_procs => $procs, tempdir => $tempdir);

Methods

   All methods inherited from Parallel::ForkManager plus the following:

   dump_stats

     Print the string returned by stats to STDERR. This may be used as a
     callback with run_on_update to see diagnostics as processes are run:

       $pm->run_on_update(\&Parallel::ForkManager::Scaled::dump_stats)

   last_update

     Returns the last time() stats were updated via update_stats_pct.

   idle

     Returns the system's idle percentage as of last_update.

     Note that it's possible for idle to be NaN if not enough time has
     elapsed between the when the object was built and the most recent
     call to update_stats_pct. Once enough time has elapsed for an idle %
     to be calculated, idle will never contain an NaN value.

   ncpus

     The number of CPUs detected on the system, this is just a wrapper to
     the cpus function from Unix::Statgrab.

   set_max_procs

     This method overrides set_max_procs from Parallel::ForkManager and
     automatically constrains the new value to be within soft_min_procs
     and soft_max_procs inclusive.

   stats

     Returns a formatted string with information about the current status.
     Takes a single parameter, the old value for max_procs. If no
     parameter is passed, the vlaue max_procs will be used.

 Method(s) you probably don't need to use

   These are not meant for general consumption but are available anyway.
   Probably best to avoid them :)

   update_stats_pct

     This method will attempt to update CPU stats (idle, etc). It is
     automatically called before each child process is start()ed if at
     least update_frequency seconds has elapsed since the last call.

     If not enough time has elapsed since the last call to
     update_stats_pct it's possible to get NaN for the new idle stat. In
     this case no updates will be made.

     If idle is updated, last_update will also be updated with the time.

EXAMPLES

   These examples are also provided in the examples/ directory of this
   distribution.

Maximize CPU usage

   see: examples/prun.pl

   Run shell commands that are passed into the program and try to keep the
   CPU busy, i.e. 0% idle

       use Parallel::ForkManager::Scaled;

       my $pm = Parallel::ForkManager::Scaled->new(
           run_on_update => \&Parallel::ForkManager::Scaled::dump_stats
       );

       # just to be sure we can saturate the CPU
       $pm->hard_max_procs($pm->ncpus * 4);

       $pm->set_waitpid_blocking_sleep(0);

       while (<>) {
           chomp;
           $pm->start and next;

           # In the child now, run the shell process
           system $_;
           $pm->finish;
       }

Dummy Load

   see: examples/dummy_load.pl

   This example provides a way to test the capabilities of this module.
   Try changing the idle_target and other settings to see the effect.

       use Parallel::ForkManager::Scaled;

       my $pm = Parallel::ForkManager::Scaled->new(
           run_on_update => \&Parallel::ForkManager::Scaled::dump_stats,
           idle_target => 50,
       );

       $pm->set_waitpid_blocking_sleep(0);

       for my $i (0..1000) {
           $pm->start and next;

           my $start = time;
           srand($$);
           my $lifespan = 5+int(rand(10));

           # Keep the CPU busy until it's time to exit
           while (time - $start < $lifespan) {
               my $a = time;
               my $b = $a^time/3;
           }

           $pm->finish;
       }

NOTES

   Currently this module only works on systems where Unix::Statgrab is
   available, which is probably any system where the libstatgrab library
   can compile.

AUTHOR

   Jason McCarver <[email protected]>

SEE ALSO

   Parallel::ForkManager

   Unix::Statgrab

REPOSITORY

   The mercurial repository for this module may be found here:

     https://bitbucket.org/jmccarv/parallel-forkmanager-scaled

   You can clone it with

     hg clone https://bitbucket.org/jmccarv/parallel-forkmanager-scaled

COPYRIGHT AND LICENSE

   This software is copyright (c) 2016 by Jason McCarver

   This is free software; you can redistribute it and/or modify it under
   the same terms as the Perl 5 programming language system itself.

You are viewing proxied material from ftp.icm.edu.pl. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.