INN multi-server thoughts.

[Dan Merillat]

I'm looking into making a multi-server addon to INN.  Basically, the
ideal setup would be transit server -> storage server -> cluster of
readers.

Now, the storage server is an interesting one.  What if that could be
split into multiple machines?  If I could take the hashed message ID,
and send it to server[hash % #servers] the load would be reduced.  Now,
the interesting part is syncronizing article numbers... Possibly done on
the transit server?

Innfeed can handle the splitting, since it can be quickly hacked to drop
articles not destined for the proper hash.  

So, that leaves me with modifying overview and the storage token... the
token would have to contain both the CNFS info and a server-ID.

I'm thinking a CNFSd running on each storage server that's queried by
the reader servers every 30 seconds as to the current buffer
position/cycle that'd allow nnrpd to know when articles were
overwritten.  

But who should generate the overview information?  Should the transit do
it?  Or should the storage servers do it and each feed all readers?

I'm thinking the transit's the best option.

Anyway, opinions?  I've actually been given the go-ahead at work to 
spend some time seeing if this is possible.  The idea I've heard here
before, it's been kicking around for years, probably.

Of course, this leaves the problem of INN keeping up with a full transit
feed as well.  May have to use diablo/cyclone there and have a custom
load balancer box.

As a second question, anyone think I could keep up with article writes
on 8 80gig IDE drives?  (All primary, no slaves)   The price disparity
between UDMA IDE and ultra-160 SCSI is getting to the 8-10x range.

Of course overview/history/boot would end up on the ultra-160 chain,
that's just a given.  Think an IDE drive can handle 25gig/day?

It's probably worth a try.  If it works it could make terrabyte
newsservers more popular.  (2 servers with 8x80 drives each is 1.28T)

--Dan