A feed modality that someone may want to prototype for INN :-;

[Joe St Sauver]

For those of you on the inn-workers list who aren't on the NANOG 
(North American Network Operators Group) mailing list, there's been an
ongoing discussion of ways to reduce Usenet bandwidth... Stephen Stuart 
mentioned that peer-to-peer technologies might be a way of handling some of
the issues associated with delivering Usenet, and just as an exercise, I 
sketched out one way that you might do it (although quite frankly I doubt
that it would be either more efficient or faster than current Usenet feed
methodologies...

Regards,

Joe

#Date: Sat, 09 Feb 2002 10:34:00 -0800 (PST)
#From: Joe St Sauver <JOE at OREGON.UOREGON.EDU>
#Subject: Re: Reducing Usenet Bandwidth
#To: nanog at merit.edu
#Message-id: <01KE2E9SQQCQ90MZGR at OREGON.UOREGON.EDU>
#
#>Date: Fri, 08 Feb 2002 22:09:12 -0800
#>From: Stephen Stuart <stuart at tech.org>
#>Subject: Re: Reducing Usenet Bandwidth
#>To: David Schwartz <davids at webmaster.com>
#>Cc: nanog at merit.edu
#
#[snip]
#
#>Some of those problems in that vein that appeared insurmountable ten years 
#>ago might have solutions in current "peer-to-peer" networking technologies
#>(thus the off-hand comment about Napster).
#
#What you're really asking for is a NNTP-over-FastTrack (Kazaa/Morpheus) feed
#object. This is sort of/really an odd idea, but if you think about it, all 
#the required parts are pretty easy to articulate in skeleton form:
#
#1) Each file dumped into FastTrack/Kazaa/Morpheus would get as its "filename"
#the article's message-ID. A site that wanted to retrieve an article would 
#issue (via FastTrack/Kazaa/Morpheus) a search request for that message-ID, 
#retrieve the article from a FastTrack/Kazaa/Morpheus server that offered that
#file, and then add it to the local news news server spool (and/or, optionally,
#a FastTrack/Kazaa/Morpheus shared directory on the local server). 
#
#2) How would servers know what article IDs to ask for? Well, Usenet overview 
#data for each group could also be published by news servers via 
#FastTrack/Kazaa/Morpheus with filenames of the format 
#pathtag!hierarchy.subhierarchy.group!yy:mm:dd:hh:mm:ss
#
#The pathtag would be required because obviously different news servers will
#know about different articles, and you will want to disambiguate which 
#server's overview data you want (not because there's any expectation that 
#any news server will in fact provide public access to any articles, but 
#simply because one might expect that N servers might all simultaneously offer 
#comparable but non-identical overview data for any given group, and you might
#want to choose the view from server foo, or bar, or baz over other 
#alternatives). 
#
#The hierarchy.subhierarchy.group component of the overview file name simply 
#allows folks to identify the overview data file they'd want.
#
#The date/time stamp (GMT) provides a means of selecting the most recently 
#available overview data from a list of matching responses.
#
#Overview data would obviously need to be cryptographically signed with a 
#key tied to the pathtag to avoid problems with people providing forged 
#overview data (wouldn't want people to be fed lists of bogus/inappropriate 
#lists of message ID's as a DOS/spam attack, right?).
#
#Overview files could be generated at some server-determined periodicity, and
#to avoid inspiring any sort of synchronization effect, I'll omit mentioning
#a value here except to say that I've got something in the double-digits-of-
#minutes in mind here. (Besides, if you're going to publish overview data for
#some tens of thousands of groups, it will take a while to walk that tree). 
#
#3) Client servers interested in retrieving articles via 
#FastTrack/Kazaa/Morpheus could routinely retrieve overview records for 
#groups of interest, retrieving all/some of the articles as a "prefetch" 
#for popular groups routinely read by their users, or the server could wait
#until articles have been requested, then retrieve an overview file, then 
#retrieve articles. 
#
#Alternatively, individual users with "News over FastTrack/Kazaa/Morpheus" 
#clients could be retrieving articles directly from Kazaa, removing any need 
#for an intervening news server whatsoever (although hopefully a local 
#"regular" news server would always be faster than a distributed 
#FastTrack/Morpheus/Kazaa-based news spool). 
#
#Regards,
#
#Joe
#
#P.S. Needless to say, this would really change the FastTrack/Kazaa/Morpheus
#content base, and given the volumes Usenet is currently transferring, would
#be sort of an interesting experiment if we're talking about a full feed...