Offloading readers in INN + CNFS

Dan Merillat harik at chaos.ao.net
Wed Oct 20 20:52:15 UTC 1999


Paul Tomblin writes:
> Quoting Dan Merillat (harik at chaos.ao.net):
> > 
> > I've had a co-worker propose a solution to making multiple newservers.
> > 
> > Obviously, duplicating 10s of gigs of storage for each reader box is
> > infeasable, so he proposed the following solution:
> > 
> > send the overview information to the reader boxes, where they maintain
> > it.  They then call the storage box to grab specific article-IDs for
> > the client.
> 
> Since you're grabbing articles from the server anyway, I don't see how this
> would improve things over not having reader machines at all.

Because I forgot to mention something.  Same thing you just proposed,
cache the article locally.  99% of the traffic is to the same
alt.binaries.pictures.erotica groups anyway. :-)

> How about this small modification of your proposal instead:
> Instead of having "reader" boxes, have boxes that run a program that caches
> nntp requests.  They have an in-memory copy of the active and maybe overview
> files, which they keep in-sync with the feeder machine through some sort of
> RPC.  These cache boxes just have a circular buffer of articles, so when a

... CNFS.   Except the idea of reader boxes is the syncronization is a
lot cheaper when the article create/article destroy comes from the
central storage.

The idea is: nearly all the reader CPU load is from overview, and nearly
all the reader disk IO load is from overview.  Actually seeking to the
start of an article and writing from mmap() is nothing...  so why not
have one server do that and offload the expensive parts to a cluster?

--Dan



More information about the inn-workers mailing list