Scaling

[Joe St Sauver]

>Date: Tue, 05 Dec 2000 17:10:05 -0500 (EST)
>From: "Forrest J. Cavalier III" <mibsoft at epix.net>
>Subject: Scaling
>To: inn-workers at isc.org
>Cc: forrest at mibsoftware.com
>Message-id: <200012052210.eB5MA5H07256 at lima.epix.net>

[snip]

>So why not NNTP?  Full feed growth is outstripping
>Moore's law, and that is a problem.

I believe Usenet traffic is doubling every six months, ASSUMING you 
carry "everything." And there's the key point...

>There is a huge amount of redundancy in NNTP transport and storage,

This is true

>and it is providing little benefit.

This, on the other hand, I disagree with.

>The benefits of redundancy are typically reliability,
>and reduced load for each unit.  Almost none of that
>is happening now. 

I believe the major benefits of redundancy are:

-- survivability (no single server failure can take down Usenet flows)
-- immunity to local policy decisions
-- ability to recover from brief duration outages

although there are other benefits, too. 

>There are still single points of failure, as far as
>clients are concerned, even when there is redundancy in
>transport.

Maybe, but you don't needto have this sort of problem. 

>Back when feeds were only 2GB/day, I did some traffic
>studies and servers needed to have 100 simultaneous
>readers (equivalent to a service population of 10,000)
>to break-even bandwidth.  (The bandwidth of nnrpd
>requests would equal 2GB/day.)  Most universities and
>small ISPs don't have that kind of service population.
>If they outsourced NNTP, they would save bandwidth (and
>of course, storage and CPU.)  And that was 2 or 3 years
>ago!

Some flaws with that analysis:

1) I personally do LOTS of feeds with higher education
peers via Internet2, as do a lot of other I2 folks. 

75% of all I2 schools connect at OC3; the rest are OC12 
attached. All new connections will be OC12 or OC48 only. 
E.G., there's plenty of bandwidth available to handle 
inter-University news flows within the context of I2.

2) Many higher education folks connect to peering points,
and exchange news at those locations at no direct charge 
for bandwidth, thereby helping the local community benefit
from I2, too., 

3) A growing number of sites now do news via satellite.

So... I'm not sure that bandwidth is really the issue here,
at least not for large universities. 

>Ways to split the loads.
>
>  0. Full outsourcing.  Distasteful to some for a number of reasons,
>     but if NNTP (and INN) had name-based virtual news server 
>     support, it would help.  We keep seeing lots of 
>     news.software.nntp requests for it.

Had a chance to try an outsourced news server in conjunction with a 
DSL account the other day. REALLY didn't find it a replacement for a 
local news server for a variety of reasons, including no Clarinet
service, waves of spam, bizarre group selection practices, plus
enough extra latency that it is hard to go fast. Oh, and did I mention
that any questions about thatnews server which were sent to the DSL 
ISP got ignored? (The biggest thing that ISPs get when they outsource news
is the ability to become selectively deaf to complaints/comments
about news from their customers). 

Regards,

Joe