huge news system architecture

[bill davidsen]

In article <20020225015252.GA16549 at wonderland.linux.it>,
Marco d'Itri <md at Linux.IT> wrote:
| 
| On Feb 24, Marek Iwanejko <M.Iwanejko at elka.pw.edu.pl> wrote:
| 
|  >   Backend should deliver content to frontends via NNTP or NFS?
| NFS and modern inn releases just do not work (at least without huge
| effort on admin's part), so I think that until somebody will write a
| storage method which can use NNTP, INN is not useful as a readers server
| for big sites.

  Sad but true. It was not desigend to be that.

|  >I am die-hard fan of INN on reader server...
| Me too, and I'd love to be proved wrong, but I tried for months to make
| INN work well with NFS (linux 2.4 clients, procom filer) and failed.

  I run 15 servers in six states serving four million potention readers,
and I would run INN if I could, but I can't. There are two solutions I
can say work. One if Twister or the NNTP only brother they don't sell
anymore. It will chain off an INN server, it works well, you feed it
articles, and if it doesn't have them it chains them from the master.
Run a master/slave INN for the master and you can make it run and scale
nicely. I use earthquake rather than INN for transit machines, until INN
solves polling problems caused by select() it has a problem with a few
hundred feeds.

  The alternative is Diablo. It needs to have a header feed to be a good
slave, a small program to pull the overview from the master and do the
header feed locally will let it get up to speed quickly, you need to do
that frequently. I did proof of concept only on that, unlike Twister
Diablo saves what it caches, so an article goes at most once.

  The overview to header feed trick solved my problem of delay from boot
to useful, I can't think of a good way to bring a new machine online
other than that.

  innd would make a good caching slave, but it would take a large
investment of time to get it working, initialize the overview, and do
caching fetching. On the other hand the SM can be what you will, and
there's no reason why it can't chain if the article isn't online. If I
had time (or funding, which is the same thing), I'd love to write the
pieces, assuming that people like the idea and would agree that it
should be mainline when working.


-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.