innfeed questions/recommendations

[Jeffrey M. Vinocur]

On Tue, 30 Sep 2003, Kathryn Hemness wrote:

> Is there a recommended size for backlog-limit?  Just looking at my backlog
> directory, I've determined that a 1MB *.output file contains about 300 lines.
> One of my peer.output files contains over 3 million lines.

I've never used the innfeed backlog stuff, but I can try to reason through 
a bit.

The first thing to think about is that if the peer is accepting the
articles, but you're consistently accumulating a backlog, the peer must
not be able to keep up with what you're offering (because of bandwidth,
disk write speed, intermittent network outages, etc).  So rather than have 
innfeed start dropping stuff at random, it might be better to explain to 
the admin of that site that it's physically not possible to feed whatever 
you currently have configured, and perhaps he should supply additional 
restrictions.  (Better to get all of the groups you care about than to 
have tons of holes in every group, right?  Or maybe there are large 
articles -- perhaps binaries -- in some group that you could filter out 
for him.)

But anyway, you just want enough backlog to hold the articles that
accumulate in whatever length outage you'd like to be prepared for.  Maybe 
you want to be able to build up two days of backlog before you start 
dropping articles.  Then work out how many articles you send it that 
period of time, and what size that would be in bytes.


> but I have another peer which is refusing or rejecting about half of the
> articles sent (this peer's innfeed output file is about half the size of
> the 3-million-liner).  Is there any way that I can determine why this
> peer is refusing/rejecting articles so that I can optimize my newsfeeds
> file?

Well, refusing just means that he already received them from another peer, 
and all you transfered was the Message-ID, which uses hardly any 
bandwidth.  So that's good.  What you *do* want to be optimizing is 
rejections, because for those you had to send the entire article before 
discovering that he didn't want it.

Unfortunately the information that you need is logged on the remote
server.  Assuming he's also running INN, the reason for every rejection is
noted in $pathlog/news.  Something like the following may be a useful way
to analyze that file.

cut -d " " -f 4,5,7- log/news \
  | grep ^- \
  | sort \
  | uniq -c \
  | sort -n \
  | head -n100


> And one last question...backlog-rotate-period is undocumented in the
> innfeed.conf man page.  I'm assuming the value is in seconds, but what
> kind of "rotation" is it controlling?

Hmm, grepping around finds me innfeed/README, which explains:

                - The .output file is allowed to grow for at least 30
                  seconds (or the value defined by the key
                  backlog-rotate-period in the config file). This prevents
                  thrashing of backlog files.

I suppose we ought to document that in innfeed.conf, huh?

-- 
Jeffrey M. Vinocur
jeff at litech.org