pure transit server

Russ Allbery rra at stanford.edu
Thu Feb 1 21:56:49 UTC 2001


Fabien Tassin <fta at sofaraway.org> writes:

> I've converted quite easily my server to be multi-threaded. Lets call
> this beast "inntd" for InterNetNews Transit(er?) daemon.

I should really write up the various thoughts I had about how to do this
at one point or another.  (Does your version allow for filtering?  That
was tricky to do in my mental model.)

What I basically had was a swarm of NNTP listeners, one per incoming
connection, that submitted history requests using reader/writer locks
around the history code and then accepted articles, handing articles off
to a queue.  The queue would be harvested by a single thread doing
filtering (which is pretty much forced by the way that most of the
embedded languages deal with threading, but threading if properly designed
shouldn't be too much of a bottleneck) and then handed off to be stored
and sent to remote sites.

The sending part, the threaded innfeed, I pictured as a central article
queue manager that dispatched the articles to the pending queues of
individual sending threads, one for each remote site, with reference
counting thrown in there somewhere so that one would know when to free the
article.

Storage required locks (since it's writing to disk files), but one should
be able to get to the point where all CNFS buffers can be written to
simultaneously.

> I wanted to directly use INN history and CNFS code but I don't think
> they are thread-safe.

Both of them involve disk writes to shared data structures, so both will
require locking to be added to make them thread-safe.

> I've first looked at the CNFS code but, ootch, how dense and fuzzy ! 
> APIs over APIs, conf files over conf files.

Yes.  The storage API is really itching for a rewrite.

> I'm currently looking at the dbz code but like CNFS and anything that
> can be useful for me, it is too tied to libinn, autoconf and the like.

Much of libinn should already be threadsafe and the rest shouldn't be too
hard to make that way.  The basic utility functions, like concat, xmalloc,
error, and the like, should be reasonably thread-safe.

> About threading, I'm using Linux pthreads. I try to write portable code
> (I'm not using autoconf yet) but threads are always a problem.
> Thoughts ?

Pretty much nothing to be done there beyond trying it on several different
platforms once it's written and porting.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>


More information about the inn-workers mailing list