pure transit server

[Russ Allbery]

Fabien Tassin <fta at sofaraway.org> writes:

> Sure. But I start to think that have a filtering mecanism available in
> the scheme of inntd will allow it to have the same set of
> functionnalities than innd. So there no point in doing inntd. hmm..

Well, there's a point; it's threaded.  :)  Or do you mean that you're
worried that adding filtering would limit what you can do to a degree that
there's no real performance gain?  I would hope that wouldn't be the case;
the filtering is CPU-bound, whereas the stuff we want to multiplex using
threading is all I/O-bound.

Cyclone manages to gain a lot of performance even with a considerably less
efficient filtering model, from all accounts.

I have a lot of personal interest in getting filtering to work, since I
can't use the server unless it does filtering.  *grin*  But I'm willing to
do that work provided that it's possible within the architecture; I've
been meaning to rework the filtering stuff anyway.

>> Perl can't cope with being run from more than one thread, and from what
>> I've heard Python copes badly, or can cope badly.

> Right. I'm no longer following Perl developpments but I thought it was a
> point that need to be worked on in 5.7 and the future 6.x.

Yeah, and it won't happen for 5.7.

> No, I wanted to move it at the end of the line, just before the storage
> API. This will let us the opportunity to have both filtred and unfiltred
> peers.

Hm, yeah, that would work.  That isn't much different than where I was
putting it, actually.  The model would be something like:

  NNTP Listener \    / Some outgoing feeds  / Other outgoing feeds
                 \  /                      /
  NNTP Listener ---------- Filter ----------- Storage thread
                 /                         \
  NNTP Listener /                           \ Storage thread

perhaps?

>> History should be a thread-safe cache front-ending a thread-safe
>> history mechanism vaguely like what we have already, I think.

> What about WIP then ? looks like it becomes unless in this design..

I peered at that for a while some time back, and while I couldn't convince
myself that I fully understood what the WIP stuff was there for and what
it was doing, I was pretty sure that with a few tweaks one could subsume
the WIP cache entirely into the caching layer of the history API.  You
just have to add a list for WIP entries that get kicked out of the history
cache before they expire and a new type of entry to insert into history
(WIP).  Once we have a history API, it should be easy enough to add.

[threaded innfeed]
>> Yeah, I actually semi-started on that at one point, but never really
>> got anywhere.

> hmm. Can it be resurected ?

I really hadn't gotten anywhere at all, just a few struct definitions that
I've already pretty much described here or will in this message.

> if it needs to be rewritten to be threadable, I see no point in making
> it a separate process, it can be part of inntd.

It was mostly in the interests of making it easier to test.  One could
make it listen to innd pretty easily and with fairly minor changes and it
could be tested in isolation from a new innd to be sure of where problems
lie.

>> Right, the model for innfeed... keep a central list of pending articles
>> that's reference-counted, and insert incoming articles into that list.
>> Then add the article to the outgoing queue of each site sender by
>> locking its queue and walking it to find the least full block.  Queues
>> are divided into blocks, each one with a size equal to the number of
>> CHECKs you're willing to send before a TAKETHIS.  Each time the sender
>> finishes with a block, it grabs a new block and replaces it with the
>> old, empty block, so it basically pulls N articles off its pending
>> queue at a time rather than just one.

> Looks like the Diablo dnewsfeed mechanism..

Heh.  Wouldn't surprise me, although I wouldn't know since I've never
looked at Diablo code.  I keep meaning to download it and read through it
and keep not getting around to it; everything I know about the
architecture I've picked up from discussions in various places.

>> Throw in some additional blocks for backlog and for deferrals; I
>> shouldn't take the time to try to write up all the details right now,
>> but it's a logical extension of the idea.

> please, do. We are in inn-*workers* here, it's the place :)

Okay, so the idea is that you have a queue of articles, equal in size to
the maximum number of outstanding CHECKs you're allowed to have.  The
basic feed function takes such a queue, offers all those articles, sends
the articles that the peer wanted, and returns a queue emptied of
everything but the deferred articles.

An outgoing peer consists of a number of sending threads, each of which
has its own pending queue.  When a new article is ready for sending, scan
the queues for a peer and add the article to the shortest queue.

At the same time, if you have a backlog for that site, you may have a
thread (or part of a thread, sharing duties with other stuff) reading from
that backlog file and filling an article queue with articles from the
backlog file.

Finally, if all the pending queues for a peer are full, there should be a
set of overflow queues, which eventually get spilled to disk.

The basic processing model for each sending thread is to look to see
what's in its pending queue.  If fewer than X articles are pending and
there's a backlog queue, grab the backlog queue and process it (freeing
the backlog reader to create another backlog queue).  Otherwise, grab the
pending queue and process it.  When finished, take the articles that were
deferred and dump them into the deferral queue with timestamps (so that we
don't retry deferrals for X seconds), and then swap the now-empty queue
with whichever queue we process next.

>>> My (purely mental) design for that is not very clear at the moment.  I
>>> was thinking to another set of NNTP threads (outgoing this time) each
>>> with a list of ref counted mmapped articles (directly into CNFS
>>> buffers). These lists will be in-memory borned fifos, ie articles
>>> enter the queue and they will have three ways to leave it: a) beeing
>>> sent, b) being pushed out and then delegated to innfeed if the queue
>>> is overloaded or c) being expired (and then delegated to innfeed) if
>>> they are in the queue longer than a expiration time.

>> Yeah, that's where I started too, but then I was worried about how
>> stuff kept wanting to lock that queue

> as it is a per thread queue, it will never be locked.

How so?  You've got the accepting threads that are reading in articles and
the outgoing threads that are sending them, and they're communicating
through that queue, so wouldn't the incoming thread have to lock the queue
while it's putting something in there for the outgoing thread?

>> When integrated into INN, I think what you really want to do is hand
>> innfeed the live article in memory to start working with and try to
>> start writing it to storage at the same time, and then let the storage
>> system replace the allocated memory with mmap'd memory when it finishes
>> writing to disk.

> Having 2 process makes things difficult but my point is to feed articles
> ASAP.

I was being imprecise; by "innfeed" in the above I meant the outgoing
threads.  Right, the idea is to start sending as fast as possible, and
swap out the in-memory copy with mmap'd data from storage if the article
sits around in memory for too long.

> There are some limits I'm not ready to pass yet but if there was a clean
> way to pass them, I'll do it. For example, start proposing articles to
> peers as soon as we've acked it (238 and 335) from a peer. Are some
> servers already doing this ? (cyclone, nntprelay ?)

That cuts down on latency, but I'm not sure it really gains us anything
except better *apparent* speed at the cost of some degree of reliability
(in the sense of offering articles we may never obtain).

>> If you can get it, you really want the storage API, I think.  This is
>> one of the things that's kept INN going against, say, Diablo; if
>> someone down the road comes up with a new, cool way of storing articles
>> that's faster than CNFS, you want to be able to drop it in.  Plus, the
>> storage API is a good layer at which to handle at least some of the
>> locking issues, I think.

> you're probably right but this forces me to maintain all methods when
> I want to extend/change the API.

Kind of.  It's not that hard to add a new method with just stubs in all
the storage methods you don't care about, and I can help out with that
(and I think I can make it easier to do that).  Other folks can jump in
and help with the other storage methods that you don't care about.

>> Part of this is actually on the TODO list for things that I want to do
>> if we ever start over from a fresh repository and completely reorganize
>> the code base.

> just create one :) We already have STABLE and CURRENT, we can have DEVEL
> or PROJECT or FUTURE or whatever you want.

I've considered it, but I can do a lot without going this far, and I think
it's better to do that until it really feels like the right time to make a
big jump to a different structure.  Besides, I keep getting new ideas
about the organization as I go.

The other problem with maintaining things in parallel is having to do more
commits in more separate places whenever anything changes.

>> I can see three libraries at least (portable replaceements for OS
>> functionality, including wrappers like xwrite and so forth; libnntp for
>> various basic NNTP functionality; and something to pick up all the
>> various utility stuff like error and xmalloc and daemonize and
>> setfdlimit).  Maybe a fourth library for configuration file parsing.

> I agree except for the NNTP lib for which I see no point.. and no easy
> way to do this either.

There arguably isn't for the servers, but there is for all the various
client stuff in the tree (getlist, inews, rnews, etc.).  There's already a
bunch of NNTP stuff in libinn, and some things like header parsing are
also rather general.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>