INN multi-server thoughts.

Russ Allbery rra at stanford.edu
Mon Mar 5 02:19:15 UTC 2001


Dan Merillat <harik at chaos.ao.net> writes:

> It's very common, and since normal INN is already finished for a hobby
> installation, the most pressing need is large site scaleability.

Well, I don't think INN is really finished there either; for a hobby
installation it badly needs some serious work done in the ease of use and
configuration department.  But yeah, INN's running hard into the limits of
its ability to scale.

> There's a number of people who've gone ahead and done this for their
> sites... generally meaning code fork and making it very difficult for
> them to update to the latest code.  Some of these forks go back to
> 1.4.1.

Yeah.  :/  I really want to avoid that happening again and getting as much
of the divergent stuff as possible back into the main code base.
Otherwise, they suffer from bit rot.  Part of that is putting together
some sort of API in front of every major component of INN so that people
can easily plug in different options.

> There's a lot of wasted effort with everyone writing the same code. I'd
> like to get at least a framework for multi-server tools into standard
> INN, so that sites don't have to fork so badly.  It would be great to
> see some of the various patches updated and merged back into the
> standard tree.

Definitely agreed.

> I was going to use a variant on the timecaf code to cache articles on
> the storage server.  Modify to expire the files based on access time
> rather then modification time. (and expire if their SM token is expired
> on the main server.)

> Of course, this means keeping a seperate database of access times for
> nnrpd to use.  It may be simpler to just throw a CNFS-type store on.
> Since the same 9 gig of articles get read every day, a 20 gig IDE drive
> on the reader will probably do quite well, even though the cache is
> FIFO.

I recommend trying the simplest thing first and seeing how well it works
before you try to do anything too complicated.  Just as a general starting
point.  I think CNFS would probably work pretty well as a cache.

> Of course, the way things are handled now this whole idea would be one
> big kludge.  What's needed is a way to feed both positive and negative
> article information to the reader servers.  I.E. every time I overrun an
> article on CNFS I send out a kill message on the sync stream, followed
> by the overview information on the article that overwrote it.  If we
> threw the message-ID hash of the article right before the article in
> CNFS, it would be very easy to read, lookup and kill.  (Or do we
> already?  There's some binary data there)

I'd like to see something like this happen (although be careful to make
sure that earlier versions of the CNFS cycbuffs can still be done).  This
would also make expiring overview data immediately as articles are
overwritten much easier as well for the people who want to do that.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>


More information about the inn-workers mailing list