NAME
Win32-ProcFarm - system for parallelization of code under Win32
OVERVIEW
What is Win32::ProcFarm?
"Win32::ProcFarm" is the code I wrote to speed up tasks that are limited
by network latency, but not by network bandwidth or local computer
power. For instance, say you want to ping every address on a subnet. The
simple approach (excluding pinging the broadcast address) is to
sequentially ping every address on the subnet. If only 30% of the
addresses are in use and you wait 1 second before deciding an address is
not in use, it will take roughly 3 minutes to ping a class C subnet. The
limitation here is obviously not the local CPU or even network
bandwidth, but rather latency. One solution would be to break up the
task. Unfortunately, the thread support in Perl doesn't work with
ActivePerl, and in any event the support is currently experimental.
Another approach would be to spin off 10 processes, have each take 25
addresses, and funnel the information back into a single process for
reporting.
This is the approach "Win32::ProcFarm" takes, but it is somewhat more
sophisticated. A "pool" of processes is created that communicate with
the parent process using TCP sockets. The parent process communicates
with the child processes using a "RPC" style library to assign tasks to
the child processes and to retrieve the return data from those tasks.
Each child process is comprised of a library file that includes the
communications routines, as well as whatever subroutines pertain to the
problem at hand. The parent process spins off the child process, which
then connects back to the parent process through a TCP port. The parent
process uses "Data::Dumper" to package up the desired subroutine name
along with any associated parameters and ships it off to the child
process. The child process then executes that subroutine and uses
"Data::Dumper" to package up the return values and send them back to the
parent. What makes the library useful is that the child process can
operate asynchronously from the parent; the parent simply calls
"execute" to instruct the child process to execute a subroutine. The
parent process can then periodically call "get_state", which will return
"wait" while the child process is still executing the subroutine. When
the child process finishes and ships the return values back up the
socket, the "get_state" method call on the parent object will return the
"fin" state. The parent then calls "get_retval" to obtain the returned
values, and the child process can then be used to execute another task.
The pool system is based upon this simplistic "RPC" system. To use the
"Win32::ProcFarm::Pool" object, one simply creates a new pool, passing
it the number of child processes to start as well as the name of the
child process and a few other parameters. Once the pool has been
created, one adds jobs to the waiting pool. This might be a list of IP
addresses to ping, for instance. Then one tells the
"Win32::ProcFarm::Pool" object to execute all the jobs. The pool assigns
a job to each of the child processes until all the child processes are
busy. It then checks the child processes periodically to see if they
have finished with the task. If they have, it places the return values
into a hash, identified by an ID passed when the job was created, and
sends the child process another job. When all the jobs have finished,
one simply requests the hash of return values and proceeds on.
Process Farm Advantages
Speed
By farming the work out over a large number of processes (I
typically use from 5 to 30), large speedup factors can be achieved
fairly easily.
Reuse
The process farm system is designed to be fairly easy to use. Simply
write the function of use, include it in a child process, and add
roughly 10 lines of boilerplate code to the parent.
Efficiency in face of variable length jobs
Because jobs are assigned one-by-one to the child processes as they
come free, jobs are allocated as efficiently as possible given the
constraint that the job execution time cannot be predicted.
Low probability of child process orphaning
Because the code to kill the child processes when everything is over
is implemented in the "DESTROY" for the parent, orphaning is a rare
event.
Process Farm Limitations
The Process Farm code is very useful in certain situations, but it has a
number of limitations that should be kept in mind.
Child Process Startup Time
On a dual Pent-Pro/200 with 128MB of RAM, child process startup time
is roughly 1/3rd of a second. This means spinning off 30 child
processes takes 10 seconds. The code already uses asynchronous
startup, and I believe the major limitation remaining is the time
necessary to start up a Perl process and create the TCP socket.
Child Process Memory Utilization
By keeping an eye on total memory utilization, it appears that each
bare child process uses roughly 2.3MB of memory. A child process
that also uses "Net::Ping" to implement a ping function uses roughly
2.6MB of memory. If you spin off 30 of these processes, that's 75MB
of RAM. If you start swapping, the thrash of 30 processes running
simultaneously is going to kill any speed benefit, so keep memory
utilization in mind when selecting the number of child processes to
use.
Real World Results
Despite the limitations, I have found the Process Farm system to be very
useful. In the previous example of pinging a range of IP addresses, with
roughly 10% coverage on a Class C, and 31 child processes, total ping
time runs roughly 21 seconds, a speed up of a factor of 10 on a problem
that otherwise takes an obnoxious amount of time.
Further Information
Please see the "tutorial" in "Docs/tutorial.pod" for more information,
as well as the POD contained within the actual Perl modules.
