INNT overview..

[Bill Davidsen]

On 24 Feb 2001, Russ Allbery wrote:

> Fabien Tassin <fta at sofaraway.org> writes:
> > Use of macros is also questionnable. Russ did a lot of work in the macro
> > and compatibility systems in INN. I will probably use the compatibility
> > code (strerror, strcasecmp, snprintf, etc.) but I'm not fond of macros
> > such as NEW/DISPOSE/SIZEOF if it adds nothing. I see interest of macros
> > for systematic error checking and I will probably use some for that in a
> > near future.
> 
> Dump NEW/DISPOSE/SIZEOF/etc.  I'd recommend dumping all of macros.h if you
> can.  I don't like anything in that file except maybe EQ and friends.

  Do I remember that NEW was added to ease problems with systems which had
broken or slow malloc family implementations? In the sense that they make
it easy to drop our own memory management under the library functions if
needed, they are a plus, like an API. I don't know that this problem
exists right now, but I'm sure someone will re-invent the bad malloc
implementation.

> Don't forget dot-stuffing for lines beginning with a period.  Oh, and
> while you're designing the article propagation path, try to make it
> completely binary clean (including nuls) if you can.  There's no real
> reason not to, and we can always reject such articles anyway.  It's hard
> to retrofit that to an existing design.

  Dot-stuffing? Shouldn't that be done by the inserting server? For
performance and binary cleanliness, I don't think you want to look at the
text at all. Let the insertion server generate it, let the reader server
filter it, a transit server whould work like a router and leave the
contents alone.

  But I would like to see input level newsgroups filtering, because it's
more efficient to drop it once than decide not to feed it to all my
outgoing feeds. This also allows deciding in the header that the article
will be dropped, and dumping it in a bitbucket, possibly gaining a little
performance in so doing.

> Right.  I'll get to that.  (I do really want the full capabilities of the
> current newsfeeds file if at all possible.)

  That's high on my list as well. Flexibility is very important.

> > - no plan for a ctlinntd yet.
> 
> I recommend it; it's one of INN's really nice features.  It's a great way
> to be able to peak at the running state on demand without having to wait
> for periodic logs.

  While we're talking you into it, without ctlinnd how would you make
changes, other than reboot the app? I suppose you could stat the config
files from time to time, or something, but that's moderately ugly.

> > here is the essential. I'd like to hear your comments. If someone is
> > interested to work on the code before I release it, contact me.
> 
> I'm happy to merge even an early version into the INN CVS tree and we can
> probably get you CVS write access to make that easier if you want.

  I certainly have some places to test this...

  My only thought is that the 'check' processing might be passed to
another process to get down to the 'takethis' faster. If this is going to
be fast (it should be), then there may be more gain in doing check in
parallel than loss in IPC. And rather than use select() polling, perhaps a
faster IPC, like message queues, could be used for check. Worst case for
checking could be a few disk i/o, so moving it out of the critical path
could be a sizable win.

  Okay, one more... I usually break my feeds into two or three streams by
size, to avoid backlogging small articles behind large. It strikes me that
if I had (a) a "large" socket open to a host, unused, and (b) a backlog of
small articles, that I could push small articles into the large queue
without backlogging the large articles. The reverse is not true.

-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.