NAME
IPC::Fork::Simple - Simplified interprocess communication for forking
processes.
SYNOPSIS
use IPC::Fork::Simple;
my $ipc = IPC::Fork::Simple->new();
my $pid = fork();
if ( $pid ) {
$ipc->spawn_data_handler();
# Do important stuff here.
# ...
#
waitpid( $pid, 0 );
$ipc->collect_data_from_handler();
warn "Child sent: " . ${$ipc->from_child( $pid, 'test' )};
} else {
$ipc->init_child();
$ipc->to_master( 'test', 'a' x 300 ) || die $!;
}
DESCRIPTION
IPC::Fork::Simple is a module designed to simplify interprocess
communication used between a parent and its child forks. This version of
the module only supports one-way communication, from the child to the
parent.
THEORY OF OPERATION
The basic idea behind this module is to one or more forks to return data
to their parent easily. This module divides a forking program into
"master", "child", and "other" forks. The "master" fork creates the
first IPC::Fork::Simple module and then calls fork() any number of
times. Any children created by the "master" will then call init_child to
specify their participation in the system. Child forks that do not call
init_child, prior forks that may have created the "master", or other
unrealted processes in the same process group, will be considered
"other" forks and will not have a role in the system.
When a "child" is ready to send data to the "master", it must assign
that data a name by which it will be retrieved later by the "master".
When the "master" is ready to collect the data from a "child", it will
request that data by name and CID. Data passed from the "child" to the
"master" will be automatically serialized/unserialized by Storable, so
almost any data type can be transmitted, of up to 4 gigabytes in size.
Once a fork calls init_child, the "master" will then be able to track
the "child" fork, returning any data that is sent, and returning whether
or not the "child" has closed its connection with he "master".
USAGE
There are three methods of use for IPC::Fork::Simple, each relating to
the actions taken by the "master" while the "children" are running.
Blocking Wait
A single call to process_child_data with the appropriate BLOCK flag will
cause process_child_data to block until a "child" has disconnected. By
calling process_child_data once for each "child", all data from all
"children" can be collected easily. Using this method makes it hard for
the "master" process to do anything other than spawn and monitor
"children".
Polling
A call to process_child_data with a "false" parameter will cause
process_child_data to only process pending data. If placed inside of a
loop, the "master" process can still gather data while it performs other
work. To determine when the "children" have ended the master can poll
finished_children for the number and CIDs of children who have
disconnected. This method will allow the "master" to perform other tasks
while the "children" are running, but it will have to make periodic
callbacks to process_child_data.
Data Handler
Calling spawn_data_handler will cause the "master" to fork, and create a
process which will automatially listen for and gather data from any
"children" spawned by the "master", either before or after the call to
spawn_data_handler. When the "master" is ready to collect the data from
the children, the "data handler" will copy all data to the "master" and
exit. To determine when a "child" has exited finished_children can be
polled or the appropriate BLOCK flag can be passed to
collect_data_from_handler. This method completely frees up the "master"
to perform other tasks. This method uses less memory and performs faster
than the others for large numbers of forks or for "master" processes
that consume large amounts of memory.
Notes
It was previously documented that calling wait(2) (or a similar
function) to determine if a "child" had ended was valid. This will
correctly detect when a "child" has exited, but an immediate call to one
of the data or finished "child" retrieval functions may not return that
"child's" data. The only way to be sure a "child's" data has been
received is to check finished_children or attempt to fetch the data.
CHILD IDENTIFICATION
Internally, "children" are identified by a "child id" number, or CID.
This number is guaranteed to be unique for each "child" (and is
currently implemented as an integer starting with 0).
"Child" processes also have a symbolic name used to identify themselves.
This name defaults to the "child's" PID, but can be changed. Symbolic
names can be re-used, and attempting to access data by symbolic name
after a symbolic name has been re-used will return the data from one of
the "children" at random. It is recommended that the symbolic name be
unique, but it is not required. PIDs are not guaranteed to be unique.
See from_cid and NOTES for details.
finished_children will return a list of "children" who have ended, and
running_children will do the same for "children" who have called
init_child but not yet ended.
EXPORTS
By default, nothing is exported by IPC::Fork::Simple. Two tags are
available to export specific flags.
:packet_flags
FLAG_PACKET flags are used to describe the reason process_child_data has
returned, and generally describing the the last action by a "child".
Note: Other flags, and thus other return values, do exist, however they
should never be returned to the caller unless due to a bug in
IPC::Fork::Simple.
FLAG_PACKET_NOERROR
No error has occurred. This flag is only returned when
process_child_data is called without blocking, but no data or events
were pending.
FLAG_PACKET_CHILD_DISCONNECTED
A "child" has ended (successfully or otherwise).
FLAG_PACKET_DATA
A "child" has sent data and it has been successfully received.
FLAG_PACKET_CHILD_HELLO
A "child" has called init_child.
:block_flags
Block flags define different blocking methods for calls to
process_child_data. See process_child_data for details.
BLOCK_NEVER
Never blocks. Processes all available data on the socket and then
returns.
Note: Technically, it is possible for this flag to block. For example,
if a "child" sends partial data, the call will block until the rest of
the data is received. These cases should be extremely rare.
BLOCK_UNTIL_CHILD
Blocks until a "child" disconnects.
Note: This flag will cause a return in other cases which are only used
internally, however it's possible a bug may cause a process_child_data
to return to the caller under other conditions.
BLOCK_UNTIL_DATA
Blocks until a "child" returns data or disconnects. The notes for
BLOCK_UNTIL_CHILD apply here too (as this is simply a superset of
BLOCK_UNTIL_CHILD).
METHODS
new
Constructor for an IPC::Fork::Simple object. Takes no arguments. Returns
an IPC::Fork::Simple object on success, or die()'s on failure.
new_child
Constructor for an IPC::Fork::Simple child-only object, used for bi-
directional with a "master".
The first parameter is an opaque value containing "master" connection
info as returned by get_connection_info on an existing IPC::Fork::Simple
object.
The second, optional, parameter is a symbolic name for this process. See
init_child for information on symbolic process names. If not set,
defaults to the process ID.
spawn_data_handler
Only usable by the "master".
Runs the parent in "data hander" mode (see above). Causes the caller to
fork(), which may be undesirable in some circumstances. Calls die() on
failure.
collect_data_from_handler
Only usable by the "master" when using the "data handler" method.
When using the "data hander" method of operation (see above), this
function will cause the "data hander" fork to return all data it has
received from "children" to the "master" and will cause the "data
hander" to clear its cache of "child" data.
The first, optional, parameter defines whether or not the "data handler"
should stay running after returning all data. For backwards
compatibility, the default (false) is to exit after collecting all data.
If this parameter is set to true, the "data handler" will not exit after
the data is sent, allowing the caller to collect data again at a later
time.
If this parameter is set to false, no more "child" processes will be
able to send data back to the "master", as the "data handler" will have
exited. This should only be called after all "children" have ended.
The second, optional, parameter is one of the BLOCK flags, as used by
process_child_data. See EXAMPLES for details on the meaning of these
flags.
init_child
Only usable by a "child".
Only to be called by a "child" after a fork, this method configured this
"child" for communication with the "master" (or "data handler"). Will
die on failure.
The first, optional, parameter is a symbolic name for this "child" with
which the "master" can retrieve data. Each child will automatically be
assigned a unique id (cid), but the optional symbolic name can be used
to simplify development. If not set, the symbolic name will be set to
the process ID. The symbolic name can not be a zero-length string.
Note: If a symbolic name is re-used, fetching data by symbolic name will
fetch data for one randomly chosen "child" that shares that name. If
symbolic names will be re-used, it's suggested that data is fetched
instead by cid.
Be aware that PIDs, the default symbolic name, may be re-used on a
system, leading to a collision of symbolic names. In order to avoid this
issue, do not call wait (or otherwise reap the child process) until you
have fetched (and then cleared) all of its data. Alternately, address
child processes by cid instead.
to_master
Only usable by a "child".
Sends data to the "master" (or "data handler"). Takes two parameters,
the first a string, used as a symbolic name for the data by which it
will be retrieved. The second parameter is the data (a scalar) that
should be sent. Data can be in any format understandable by Storable,
however since this data is sent between forks, data containing
filehandles should not be passed.
push_to_master
Only usable by a "child".
Pushes data into a queue sent to the "master". Unlike to_master, data
sent with push_to_master is not overwritten, but appended to, much like
when working with an array. Function semantics are otherwise identical
to to_master.
The first parameter is the symbolic name for the data, and the second is
a reference to the data that will be sent.
from_cid
Only usable by the "master".
Retrieves data from a "child" after the "child" has sent it. Takes two
parameters, the first is the cid from which the data was sent, and the
second is a symbolic name (a string) for the data, which the "child"
specified when the data was sent.
Returns nothing if no data is available, or a reference to whatever data
the "child" sent. Note: You may need to use ref() in order to determine
the type of the data sent.
from_child
Only usable by the "master".
Semantics are the same as from_cid, but searches by symbolic name
instead of cid.
pop_from_cid
Only usable by the "master".
Retrieves pushed data from a "child" after the "child" has sent it.
Takes two parameters, the first is the cid from which the data was sent,
and the second is a symbolic name (a string) for the data, which the
"child" specified when the data was sent.
Called in scalar context, returns nothing if no data is available, or a
reference to the oldest data the "child" pushed. Called in array
context, returns an empty array if no data is available, or an array of
references to the data pushed by the "child", ordered oldest to most
recent.
After the data is returned, it is removed from the internal list, so a
subsequent call to pop_from_cid will return the next oldest set of data.
Note: You may need to use ref() in order to determine the type of the
data sent.
pop_from_child
Only usable by the "master".
Semantics are the same as from_cid, but searches by symbolic name
instead of cid.
finished_children
Only usable by the "master".
In scalar context, returns the number of "children" who have finished.
In array contaxt and the first, optional, parameter is true, returns a
hash of cid-to-symbolic name mappings for these "children". If the first
parameter is not set, or is false, returns a list of CIDs that have
finished.
running_children
Only usable by the "master".
In scalar context, returns the number of "children" who have called
init_child but have not yet ended.
In array contaxt and the first, optional, parameter is true, returns a
hash of cid-to-symbolic name mappings for these "children". If the first
parameter is not set, or is false, returns a list of CIDs that have not
yet finished.
process_child_data
Only usable by the "master" when using the "blocking wait" and "polling"
methods.
Processes data from all "children". Takes a single parameter, a BLOCK
flag that determines if, and how, process_child_data should block. See
the EXPORTS section for details on these flags.
child_data and finished_children can be called between calls to
process_child_data, but there is no guarantee there will be any data
available.
If process_child_data is not called often or fast enough, "children"
will be forced to block on calls to to_master, and data loss is
possible.
Returns a FLAG_PACKET flag describing the last "child" action. See the
EXPORTS section for details on these flags.
clear_finished_children
Only usable by the "master".
Deletes the "master's" copy of the list of "children" who have ended. If
a "data handler" is being used, its copy of the list is not affected.
The only optional parameter is the list of "child" PIDs to remove data
for. If specified, only the entries for those specified "children" will
be removed. If no list is passed, then all data will be cleared.
clear_child_data
Only usable by the "master".
Deletes the "master's" copy of the data (standard and enqueued)
"children" who have ended. If a "data handler" is being used, its copy
of the lists are not affected.
The only optional parameter is the list of "child" PIDs to remove data
for. If specified, only the entries for those specified "children" will
be removed. If no list is passed, then all data will be cleared.
get_connection_info
Only usable by the "master".
Retrieves an opaque value representing connection data for this object
(or its "data handler"). Only useful to pass into new_child.
EXAMPLES
Data Handler
use warnings;
use strict;
use IPC::Fork::Simple;
my $ipc = IPC::Fork::Simple->new();
my $pid = fork();
if ( $pid ) {
$ipc->spawn_data_handler();
waitpid( $pid, 0 );
$ipc->collect_data_from_handler();
warn length(${$ipc->from_child( $pid, 'test' )});
} else {
$ipc->init_child();
$ipc->to_master( 'test', 'a' x 300 ) || die $!;
}
Blocking
use warnings;
use strict;
use IPC::Fork::Simple;
use POSIX ":sys_wait_h";
my $ipc = IPC::Fork::Simple->new();
my $pid = fork();
die 'stupid fork' unless defined $pid;
if ( $pid ) {
$ipc->process_child_data(1);
my @finished = $ipc->finished_children();
die unless 1 == scalar( $ipc->finished_children() );
die unless 300 == length(${$ipc->from_child( $pid, 'test' )});
die unless 300 == length(${$ipc->from_cid( $finished[0], 'test' )});
} else {
$ipc->init_child();
$ipc->to_master( 'test', 'a' x 300 ) || die $!;
}
Polling
use warnings;
use strict;
use IPC::Fork::Simple;
use POSIX ":sys_wait_h";
my $ipc = IPC::Fork::Simple->new();
my $pid = fork();
if ( $pid ) {
while ( ! $ipc->finished_children() ) {
$ipc->process_child_data(0);
waitpid( -1, WNOHANG );
sleep(0);
}
warn length(${$ipc->from_child( $pid, 'test' )});
} else {
$ipc->init_child();
$ipc->to_master( 'test', 'a' x 300 ) || die $!;
}
Data queues
use warnings;
use strict;
use IPC::Fork::Simple;
my $ipc = IPC::Fork::Simple->new();
my $pid = fork();
die 'stupid fork' unless defined $pid;
if ( $pid ) {
$ipc->process_child_data(1);
die unless 300 == length(${$ipc->pop_from_child( $pid, 'test' )});
die unless 301 == length(${$ipc->pop_from_child( $pid, 'test' )});
die unless 302 == length(${$ipc->pop_from_child( $pid, 'test' )});
} else {
$ipc->init_child();
$ipc->push_to_master( 'test', 'a' x 300 ) || die $!;
$ipc->push_to_master( 'test', 'b' x 301 ) || die $!;
$ipc->push_to_master( 'test', 'c' x 302 ) || die $!;
}
Bi-directional communication
use warnings;
use strict;
use IPC::Fork::Simple qw/:block_flags/;
my $ipc = IPC::Fork::Simple->new();
my $master_pid = $$;
my $pid = fork();
die 'stupid fork' unless defined $pid;
if ( $pid ) {
$ipc->process_child_data(BLOCK_UNTIL_DATA);
my $child_connection_data = $ipc->from_child( $pid, 'connection_info' );
my $ipc2 = IPC::Fork::Simple->new_child( ${$child_connection_data} ) || die;
$ipc2->to_master( 'master_test', 'a' x 300 );
} else {
$ipc->init_child();
my $ipc2 = IPC::Fork::Simple->new();
$ipc->to_master( 'connection_info', $ipc2->get_connection_info() ) || die $!;
$ipc2->process_child_data(BLOCK_UNTIL_DATA);
die unless length( ${$ipc2->from_child( $master_pid, 'master_test' )} ) == 300;
}
Bi-directional communication with data handlers
use warnings;
use strict;
use IPC::Fork::Simple qw/:block_flags/;
my $ipc = IPC::Fork::Simple->new();
my $master_pid = $$;
my $pid = fork();
die 'stupid fork' unless defined $pid;
if ( $pid ) {
$ipc->spawn_data_handler();
my $child_connection_data;
my $ipc2 = IPC::Fork::Simple->new_child( ${$child_connection_data} ) || die;
$ipc2->to_master( 'master_test', 'a' x 300 );
} else {
$ipc->init_child();
my $ipc2 = IPC::Fork::Simple->new();
$ipc2->spawn_data_handler();
$ipc->to_master( 'connection_info', $ipc2->get_connection_info() ) || die $!;
my $test;
do {
sleep(0);
$ipc2->collect_data_from_handler(1);
$test = $ipc2->from_child( $master_pid, 'master_test' )
} until ( $test );
die unless length( ${$test} ) == 300;
}
Further examples
Further examples can be found in the t/functional directory supplied
with the distribution.
NOTES
Zombies
Child processes are not reaped automatically by this module, so the
caller will need to call wait (or similar function) as usual to reap
"child" processes.
Security
This module creates a TCP listen socket on a random high-numbered port
on 127.0.0.1. If a malicious program connects to that socket, it could
cause the "master" process to hang waiting for that socket to
disconnect. Resolving this issue is considered a "TODO".
Invalid connections
If someone connects, but does not send the proper data, it is possible
that we could return from process_child_data with
FLAG_PACKET_CHILD_DISCONNECTED but without updating any data or the
finished child list.
Callers checking for a return value of FLAG_PACKET_CHILD_DISCONNECTED
should therefor also check finished_children to make sure a real "child"
actually finished.
Unit tests
The module currently lacks unit tests but does have a collection of
functional tests. During "make test" these functional tests are not run,
as they can be system intensive. Ideally, unit tests will be developed
for this purpose, but until then they can be run by hand. They can be
found in the t/functional directory as part of the distribution.
TO DO
Implement protocol security to reject/expire lingering connections and
connections that do not send a HELLO.
Return better information for clients that connect and disconnect but
send no data (currently, we return with FLAG_PACKET_CHILD_DISCONNECTED
but don't update internal state, so a caller checking for that updated
internal state could get confused.)
Merge the internal finished_children hash with the internal child_info
hash. The child_info hash already holds most of the data, a flag to
determine whether or not that child is still connected would be simple
to add, but removing the quick lookups against finished_children would
make the code more verbose in places. Merging the two hashes would also
reduce data duplication of the symbolic name.
Add unit tests, or make functional tests run as part of "make test".
CHANGES
1.46 - 20100830, jeagle
Version bump and repackage for CPAN.
1.45 - 20100623, jeagle
Clean and prepare for export to CPAN.
Version bump to synchronize source repository version with module
version.
0.8 - 20100506, jeagle
Replace MSG_NOSIGNAL with an ignored SIGPIPE, because we can't rely on
MSG_NOSIGNAL to be defined everywhere.
0.7 - 20100427, jeagle
Disable SIGPIPE for failed send()s, returns error instead (to match
documentation/intention).
Correctly process large reads (>64k).
0.6 - 20100309, phirince
Extra check in pop_from_cid to get rid of undefined value errors.
0.5 - 20100219, jeagle
Correct layout issues with example documentation.
Clarify the use of wait(2) in determining if a "child" has ended.
0.4 - 20100219, jeagle
Fix more bugs related to PID size assumptions.
Fix various networking bugs that could cause data loss.
Implement new bi-directional communication abilities.
Implement new data queue types.
Allow processes to identify themselves by a symbolic name, instead of
pid (if not set, defaults to pid).
0.3 - 20090512, phirince
Fixed bug 2741310 - IPC::Fork::Simple assumed pids are 16 bits instead
of 32 bits.
0.2 - 20090217, jeagle
Fixed a bug with process_child_data returning early when a signal is
received.
