innfeed questions/recommendations
Jeffrey M. Vinocur
jeff at litech.org
Tue Sep 30 23:54:30 UTC 2003
On Tue, 30 Sep 2003, Kathryn Hemness wrote:
> Is there a recommended size for backlog-limit? Just looking at my backlog
> directory, I've determined that a 1MB *.output file contains about 300 lines.
> One of my peer.output files contains over 3 million lines.
I've never used the innfeed backlog stuff, but I can try to reason through
a bit.
The first thing to think about is that if the peer is accepting the
articles, but you're consistently accumulating a backlog, the peer must
not be able to keep up with what you're offering (because of bandwidth,
disk write speed, intermittent network outages, etc). So rather than have
innfeed start dropping stuff at random, it might be better to explain to
the admin of that site that it's physically not possible to feed whatever
you currently have configured, and perhaps he should supply additional
restrictions. (Better to get all of the groups you care about than to
have tons of holes in every group, right? Or maybe there are large
articles -- perhaps binaries -- in some group that you could filter out
for him.)
But anyway, you just want enough backlog to hold the articles that
accumulate in whatever length outage you'd like to be prepared for. Maybe
you want to be able to build up two days of backlog before you start
dropping articles. Then work out how many articles you send it that
period of time, and what size that would be in bytes.
> but I have another peer which is refusing or rejecting about half of the
> articles sent (this peer's innfeed output file is about half the size of
> the 3-million-liner). Is there any way that I can determine why this
> peer is refusing/rejecting articles so that I can optimize my newsfeeds
> file?
Well, refusing just means that he already received them from another peer,
and all you transfered was the Message-ID, which uses hardly any
bandwidth. So that's good. What you *do* want to be optimizing is
rejections, because for those you had to send the entire article before
discovering that he didn't want it.
Unfortunately the information that you need is logged on the remote
server. Assuming he's also running INN, the reason for every rejection is
noted in $pathlog/news. Something like the following may be a useful way
to analyze that file.
cut -d " " -f 4,5,7- log/news \
| grep ^- \
| sort \
| uniq -c \
| sort -n \
| head -n100
> And one last question...backlog-rotate-period is undocumented in the
> innfeed.conf man page. I'm assuming the value is in seconds, but what
> kind of "rotation" is it controlling?
Hmm, grepping around finds me innfeed/README, which explains:
- The .output file is allowed to grow for at least 30
seconds (or the value defined by the key
backlog-rotate-period in the config file). This prevents
thrashing of backlog files.
I suppose we ought to document that in innfeed.conf, huh?
--
Jeffrey M. Vinocur
jeff at litech.org
More information about the inn-workers
mailing list