All, I have an application using Thread::Queue that I thought was pretty simply in implementation
To summarize the code roughly it works like this (there's more to it, but it's too voluminous for a post):
# create thread input and output queues
my $in_q = new Thread::Queue;
my $out_q = new Thread::Queue;
# create worker threads
threads->create("ProcessDataLine") for (1 .. $worker_threads);
while ( <INPUTDATA> ) {
$in_q->enqueue($_);
}
sub ProcessDataLine {
my $input_line;
while ( $input_line = $in_q->dequeue ) {
# do some stuff, put in $result
$out_q->enqueue($result);
}
}
Basically, each thread performs some parsing and formatting on a line of data from the in_q and sticks the result in the out_q.
However, what I'm finding is that no matter how man $worker_threads I add, I get no aggregate performance increase. I can verify that each thread is running, and is picking up about an equal share if the in_q data, but the in_q is consumed no faster with 5, 10 or 16 threads than it is with one thread -- the overall consumption rate from the queue remains the same.
I am not at the system limit for how fast data can be read from the queue -- I experimented by simplifying the thread code and saw huge increases in speed of queue consumption, but only in the context of a single thread: one thread still performed the same as multiple threads.
I'm using perl v5.8.6 built for i686-linux-thread-multi and Thread::Queue 2.12 -- I'm kind of stuck with that version of perl for project compatibility reasons.
Does anyone have any ideas about why I am not seeing any performance improvements for additional worker threads?
