pure transit server

Thu Feb 8 00:00:30 UTC 2001

According to Russ Allbery:
> 
> Fabien Tassin <fta at sofaraway.org> writes:
> 
> > Sure. But I start to think that have a filtering mecanism available in
> > the scheme of inntd will allow it to have the same set of
> > functionnalities than innd. So there no point in doing inntd. hmm..
> 
> Well, there's a point; it's threaded.  :)  Or do you mean that you're
> worried that adding filtering would limit what you can do to a degree that
> there's no real performance gain?  I would hope that wouldn't be the case;
> the filtering is CPU-bound, whereas the stuff we want to multiplex using
> threading is all I/O-bound.

no, my point was no performances but pure transiter vs full feature nntp
server. I have no real doubts that threads will help and I'm convinced
that a new design is needed to obtain real gains. 

> Cyclone manages to gain a lot of performance even with a considerably less
> efficient filtering model, from all accounts.

I don't really know Cyclone tricks but it seems that some of them are good..

> I have a lot of personal interest in getting filtering to work, since I
> can't use the server unless it does filtering.  *grin*  But I'm willing to
> do that work provided that it's possible within the architecture; I've
> been meaning to rework the filtering stuff anyway.

I try to keep filtering possible. I will probably not write it myself
but I'll add a hook for it. Probably using an API. If you have some
precise requierements, please, speak now.

> > No, I wanted to move it at the end of the line, just before the storage
> > API. This will let us the opportunity to have both filtred and unfiltred
> > peers.
> 
> Hm, yeah, that would work.  That isn't much different than where I was
> putting it, actually.  The model would be something like:
> 
>   NNTP Listener \    / Some outgoing feeds  / Other outgoing feeds
>                  \  /                      /
>   NNTP Listener ---------- Filter ----------- Storage thread
>                  /                         \
>   NNTP Listener /                           \ Storage thread
> 
> perhaps?

right.

> >> History should be a thread-safe cache front-ending a thread-safe
> >> history mechanism vaguely like what we have already, I think.
> 
> > What about WIP then ? looks like it becomes unless in this design..
> 
> I peered at that for a while some time back, and while I couldn't convince
> myself that I fully understood what the WIP stuff was there for and what
> it was doing, I was pretty sure that with a few tweaks one could subsume
> the WIP cache entirely into the caching layer of the history API.  You
> just have to add a list for WIP entries that get kicked out of the history
> cache before they expire and a new type of entry to insert into history
> (WIP).  Once we have a history API, it should be easy enough to add.

I've started a history API hiding everything to INNT. WIP and cache are
not visible but I want more. I'm thinking of a way to avoid expire.
A kind of thread dedicated to self-expiring history based upon its
size and upon hierarchy local prefs. Avoiding make(history|dbz) would be
nice too but it doesn't fit into my current model.. still searching.

> > if it needs to be rewritten to be threadable, I see no point in making
> > it a separate process, it can be part of inntd.
> 
> It was mostly in the interests of making it easier to test.  One could
> make it listen to innd pretty easily and with fairly minor changes and it
> could be tested in isolation from a new innd to be sure of where problems
> lie.

this will require a lot of memory copies or disk reads that can be avoided
in a single process. OTOH, testing is important too.

> > There are some limits I'm not ready to pass yet but if there was a clean
> > way to pass them, I'll do it. For example, start proposing articles to
> > peers as soon as we've acked it (238 and 335) from a peer. Are some
> > servers already doing this ? (cyclone, nntprelay ?)
> 
> That cuts down on latency, but I'm not sure it really gains us anything
> except better *apparent* speed at the cost of some degree of reliability
> (in the sense of offering articles we may never obtain).

As I've said, I'm not ready to do it :)

> >> If you can get it, you really want the storage API, I think.  This is
> >> one of the things that's kept INN going against, say, Diablo; if
> >> someone down the road comes up with a new, cool way of storing articles
> >> that's faster than CNFS, you want to be able to drop it in.  Plus, the
> >> storage API is a good layer at which to handle at least some of the
> >> locking issues, I think.
> 
> > you're probably right but this forces me to maintain all methods when
> > I want to extend/change the API.
> 
> Kind of.  It's not that hard to add a new method with just stubs in all
> the storage methods you don't care about, and I can help out with that
> (and I think I can make it easier to do that).  Other folks can jump in
> and help with the other storage methods that you don't care about.

My article structure is very different from the existing one. So I'm not
able to use the existing code as is. This is probably a good thing because
I'm not satisfied by the current model. Too many indirections and conf files,
in particular.

-- 
Fabien Tassin -+- fta at sofaraway.org