INN multi-server thoughts.

[Alex Kiernan]

Dan Merillat <harik at chaos.ao.net> writes:

> I'm looking into making a multi-server addon to INN.  Basically, the
> ideal setup would be transit server -> storage server -> cluster of
> readers.
> 

We're proposing exactly the same thing.

> Now, the storage server is an interesting one.  What if that could be
> split into multiple machines?  If I could take the hashed message ID,
> and send it to server[hash % #servers] the load would be reduced.  Now,
> the interesting part is syncronizing article numbers... Possibly done on
> the transit server?
> 

More or less what we intend doing, I've an extra piece which we're
proposing - a numbering server... the transit server will push a feed
to a numbering cluster (we intend changing the active file management
so we can cluster two innds on separate boxes for resilience), that
then distributes articles to each storage server (distributed somehow,
probably not just message-id hash, more likely by moving average of
article and size).

> Innfeed can handle the splitting, since it can be quickly hacked to drop
> articles not destined for the proper hash.  
> 
> So, that leaves me with modifying overview and the storage token... the
> token would have to contain both the CNFS info and a server-ID.
> 
> I'm thinking a CNFSd running on each storage server that's queried by
> the reader servers every 30 seconds as to the current buffer
> position/cycle that'd allow nnrpd to know when articles were
> overwritten.  
> 

I was proposing to implement an NNTP storage manager which would query
the appropriate backend storage server. The Xref header in each
message would indicate which storage server an article was located on.

> But who should generate the overview information?  Should the transit do
> it?  Or should the storage servers do it and each feed all readers?
> 

We're planning on implementing a header only feed on the storage
servers which feeds the readers, each reader then generates a unified
overview database. In addition I've been toying with ways of pushing
articles for small, text only, frequently read groups to the reader
servers.

-- 
Alex Kiernan, Principal Engineer, Development, Thus PLC