NAME
Queue::Q - Mix-and-match Queue Implementations and Backends
SYNOPSIS
# Pick you queue type and a back-end:
# Very primitive FIFO queue (abstract interface)
use Queue::Q::NaiveFIFO;
# ... and its pure-Perl, in-memory implementation
use Queue::Q::NaiveFIFO::Perl;
# ... its Redis-based implementation
use Queue::Q::NaiveFIFO::Redis;
# "Reliable" enqueue, claim, mark-as-done FIFO queue:
use Queue::Q::ClaimFIFO; # abstract interface
# ... and again, its pure-Perl, in-memory implementation
use Queue::Q::ClaimFIFO::Perl;
# ... and its Redis-based implementation
use Queue::Q::ClaimFIFO::Redis;
# A composite, distributed queue, built from shards of
# the above types with weak global and strict local
# ordering, scaling beyond single nodes.
use Queue::Q::DistFIFO;
# Usage is somewhat similar for most queue types:
my $queue = Queue::Q::ClaimFIFO::Redis->new(
server => 'redis-01',
port => '6379',
queue_name => "image_queue"
);
$q->enqueue_item( $img_url );
# In another process on another machine:
while (defined( my $item = $q->claim_item )) {
# Do some work!
$q->mark_item_as_done($item);
}
# n-ary versions exist, too:
$q->enqueue_items( @img_urls );
my @items = $q->claim_items(10); # 10 at a time, padded with undef
$q->mark_items_as_done(@items);
DESCRIPTION
This is an experimental module. The interface may change without notice.
Before using it in production, please get in touch with the authors!
"Queue::Q" is a collection of queue implementations each with multiple
backends. Right now, it comes with three basic queues:
Queue::Q::NaiveFIFO
A naive FIFO queue "NaiveFIFO". Supports enqueuing data and later
claiming it in the order it was enqueued. Strict ordering,
low-latency, high-throughput, no resilience against crashing workers.
Queue::Q::ClaimFIFO
A claim/mark-as-done FIFO queue "ClaimFIFO". Supports enqueuing data
(as Queue::Q::ClaimFIFO::Item objects), claiming items, and keeping
track of items-being-worked-on until they are reported as done. Strict
ordering, slightly higher latency and lower throughput than the naive
FIFO queue. Resilience against crashing workers when combined with a
clean-up script that checks for old, claimed items.
Queue::Q::ReliableFIFO
Similar interface to the "ClaimFIFO" queue type, but only comes with a
Redis-backed implementation now. Much faster for the basic operations
then "ClaimFIFO" since it doesn't always have to execute Lua scripts.
See Queue::Q::ReliableFIFO for details.
The first two basic queues come with two back-end implementations each
right now: One is a very simple, in-memory, single-process
implementation Queue::Q::NaiveFIFO::Perl and Queue::Q::ClaimFIFO::Perl
respectively. The other is an implementation based on Redis:
Queue::Q::NaiveFIFO::Redis and Queue::Q::ClaimFIFO::Redis.
As noted above, the "ReliableFIFO" queue has a Redis-based backend only
for now.
In addition to the basic queue types, the distribution contains
Queue::Q::DistFIFO, an implementation of a distributed queue that can
use the basic queues as shards (but only one type of shard per
distributed queue). It supports all operations of the basic queues, but
does not enforce strict global ordering, but weak global and strict
local ordering. In an early test, two mid-range servers running multiple
instances of Redis each sustained 800k-1M transactions per second with a
naive-type distributed queue. "DistFIFO" has not been tested with the
"ReliableFIFO" implementation as building blocks yet!
ACKNOWLEDGMENT
This module was originally developed for Booking.com. With approval from
Booking.com, this module was generalized and put on CPAN, for which the
authors would like to express their gratitude.
AUTHOR
Herald van der Breggen, <
[email protected]>
Steffen Mueller, <
[email protected]>
COPYRIGHT AND LICENSE
Copyright (C) 2012, 2013, 2014 by Steffen Mueller
This library is free software; you can redistribute it and/or modify it
under the same terms as Perl itself, either Perl version 5.8.1 or, at
your option, any later version of Perl 5 you may have available.
