Any reason not to split perl filtering?

[bill davidsen]

In article <8765h08m3d.fsf at windlord.stanford.edu>,
Russ Allbery  <rra at stanford.edu> wrote:
| bill davidsen <davidsen at tmr.com> writes:
| 
| > Single threaded innd is simply not keeping up with the incoming volume,
| > and the obvious trouble spot is the perl filter. I have one CPU 98% used
| > and the rest idle, is there any reason not to move filtering into a
| > separate process to use some extra power. Looks cheaper than buying a
| > new system :-(
| 
| > Has anyone done this, and if so are there records of how it worked or
| > why it didn't?
| 
| > The actual reading of the articles is the next target, but the perl
| > filter takes way too much CPU!
| 
| You take a fairly sizable I/O performance hit by doing this since you have
| to start talking back and forth with a separate process (hence going
| through the kernel).  Whether that's better or not depends a lot on the
| system configuration.

Other news programs do this, but I was actually going to go pthreads. As
long as I use a lock to keep from running more than one at a time and
exploring how thread-safe the perl stuff might be, I think I can do what
I need.

My fallback is to use two processes sharing a memory segment large
enough to hold the max size article. That would cut the overhead down to
one memcopy, although not desirable.
| 
| The other obvious problem is that no one's submitted the code for it yet.

If it works, I will. Pthreads are not bleeding edge anymore, but I am
not at all familiar with the way the perl interface works. I have done a
lot of IPC, I can try several possibilities as long as there's no
obvious reason not to do it at all.
| 
| I have plans at some point to redo the filtering setup to provide a
| cleaner API, which would make this much easier, but I have no idea when
| I'll find the time.

Just don't find the time in the next month or so :-( 
-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.