Problem using trn & INN 2.3.1 (fwd)

Thu Dec 7 08:18:55 UTC 2000

This is responso two of two I'm sending to the INN mailing-list.

Patrick.

---------- Forwarded message ----------
Date: Wed, 6 Dec 2000 17:08:24 -0600
From: Mike Castle <dalgoda at ix.netcom.com>
To: trn-users at lists.sourceforge.net
Subject: Re: Problem using trn & INN 2.3.1 (fwd)

I suspect the reason trn asks for things in chunks is to support the
background threading.  In both cases (xover and xhdr), pulling in the
entire range would result in the application blocking until all the data is
received.  Usually THAT is unacceptable.  Remember, rn/trn was written
to be threaded before things like pthreads was popular.

I believe what is happening is this:

trn does it's xover stuff to grab information.  However, because many of
the articles are expired, much is not returned.  Trn is probably then has
not yet assumed that the article is non-existent, but rather than the xover
database is corrupt/out-of-date/or something.  So, later on, when trn tries
to process the information (kill files, display on the screen, etc), it
then falls back to xhdr stuff, only this time it's in a mode where it's
calling one at a time rather than block mode (basically trn is trying to
defer getting the information at this point until it absolutely needs it).

Now, falling back to doing xhdr one at a time does look a little flaky, and
is something trn should probably be improved on.

However, I think part of it is INN's fault.

>From the description given below, it sounds like INN generates the xover
response differently depending on whether you pull the whole range or not.
This is stupid, because it could be that at some point the client and the
server are out of date as to what the entire range is, and the server is
going to generate a large amount of data the very hard way.  Not a very
brilliant design, IMO.

Of course, I don't think the person who responded to you is very
convincing.

The constant, immature spellings of "froup," "swerver," "st00pid," "brane,"
"hertz" for example.

Also, consider the following comments:
> down, I just don't understand *WHY* everyone is complaining that
> it takes forever to retrieve headers from binaries froups when they

> A Real Newsreader[tm] will keep track of the last article you read
> in the froup, and only get information from that point forward.

I assume that the author thinks that a real newsreader (I do wonder what
he considers a real newsreader) should be responsible for duplicating
the information that the server already has.  A client historically has
fewer resources than a server.  But apparently he is expecting the client
to keep track of all the information about the groups inbetween visits.
That can be a substantial amount of resources!  (At least trn DOES keep
the information if you exit a group so that if you immediately reenter
that same group, it is still cached).  Not to mention that one does need
the information for the entire group in order to support proper threading.

That said, here are some things to try:

In rt-ov.ih, change the following line:
#define OV_CHUNK_SIZE   40

Make it something unreasonably large (it appears to only affect the range
of the xover command, not any arrays).

Do something similar for head.h:
#define PREFETCH_SIZE 5

If they are large enough, they may end up doing larger batches of xover
fetches and have less effect on the server.  Of course, you will
essentially make the backgrounded threading useless, as it won't process a
keystroke until those large ranges of data are handled.

Another thing to consider is installing something like nntpcache or
leafnode.  Both will implement utilizing local resources to keep copies
of posts.

mrc

On Wed, Dec 06, 2000 at 04:59:22PM +0100, Patrick Vervoorn wrote:
> Hi,
>
> I got this response from someone in the INN list. Maybe something to keep
> in mind for some changes to trn itself in the future?
>
> Best regards,
>
> Patrick.
>
> ---------- Forwarded message ----------
> Date: Wed, 6 Dec 2000 16:50:37 +0100 (CET)
> From: System Operatour <NewsUser at free-n0rp.NetScum.dk>
> Reply-To: inn-lurkers at netscum.dk
> To: Patrick Vervoorn <vervoorn at cddb.ubicom.tudelft.nl>
> Cc: inn-workers at isc.org
> Subject: Re: Problem using trn & INN 2.3.1
>
> ::   1. Trn asks for the range of valid articles in the newsgroup
> ::   2. The news-server reacts to this with the range.
> ::   3. Trn does an XOVER from the first several hundred articles numbers
> ::      which are inside the range as returned by step 2.
>
> This is ....  This is disgusting.   This is ...  st00pid.  This
> is incredibly stupidly obnoxiously brane-damaged.  This is senseless.
> This is the sort of thing I've blocked people from my swerver for...
>
> :: <215 Newsgroups in form "group high low flags".
> :: <alt.chello.binaries 0001147847 0000806723 y
> :: >XOVER 806723-807042
> :: <224 data follows
> :: >XOVER 807043-807362
> :: <224 data follows
> :: >XOVER 807363-807962
> :: <224 data follows
> :: >XOVER 807963-808642
> :: <224 data follows
> :: etc...
>
> I mean, what *is* the point?  Why do you need all the information
> from all the tens or hundreds of thousands of articles in such a
> froup, every time you enter it?  Rrrgh.  My brane hertz.  And if
> you're pulling all the headers, why not ask for all of them at once
> rather than this back-and-forth that I simply don't understand at
> all.  I mean, you add latency for every time you ask for a partial
> range, and I've optimised my swerver that returning the whole range
> in one response is a trivial no-op, while if you ask for bits and
> pieces, I'm assuming you're a Real Reader with a Real Reason to
> just want part of the data, and I'll do the extra work to make sure
> that data is correct even if it kills me...  grumble...
>
>
> ::   6. After a while, trn starts fetching the headers, using the range as
> ::      returned by the newsserver in step 2.
> ::      For example, if the range as returned by step 2. was 100-60100, then
> ::      trn does:
> ::      XHDR 100
> ::      XHDR 101
> ::      XHDR 102
> ::      etc...
>
> Yeah.  I've seen this too.  WHAT IS THE POINT.  You want a whole
> range, ask for the whole range.  It's usually the case that the
> first article present (when it's still present) of the range of
> hundreds of thousands is a long-lived FAQ, followed by a gap of
> tens of thousands of missing articles that you're pulling by the
> handful, rather than taking the fast easy way out and asking for
> the entire range, first through last...  Sorry, my brane has melted
> down, I just don't understand *WHY* everyone is complaining that
> it takes forever to retrieve headers from binaries froups when they
> are asking for *all* headers, even ones they've seen before...
> Grrr....
>
>
> ::      These also fail (each XHDR command results in an error message from
> ::      the newsserver mentioning the asked article no longer exists), but at
> ::      long last trn in this way reaches articles that _DO_ exist, and
> ::      finally it displays the article overview in the regular fashion.
> ::
> :: This entire process takes several HOURS to complete, and all the while the
> :: load of the news-server (according to the admin) is about 40-50%, which is
> :: the reason he shut-out the machine I'm using trn on.
>
> Yeah.  You're asking for a range of articles that are missing.  The
> `LIST ACTIVE' command only gives you the low- and high-watermarks
> for the froup, not the count of actual articles present, which will
> almost certainly be less, whether due to cancels or gaps in the
> numbering caused by uneven expiry rates (no reason for a FAQ not to
> remain for a month when 25 000 binary posts per day get purged in
> less than a week)...
>
> A Real Newsreader[tm] will keep track of the last article you read
> in the froup, and only get information from that point forward.
> That will usually be enough to get past the gap in numbering that
> you're seeing, until you go on holiday for a week or something, but
> still.  Geez.  I mean, really.
>
> Yeah, this is something I've noticed on my test swerver -- that people
> connect and pull the entire overview data, which means recreating it
> out of the database, and that's pretty expensive.  Or at least it
> looks that way from `top' and looking at the disk activity light on
> the untuned spool and seeing the slowdown of everything else.  Of
> course, I have my ideas about what to do about these sorts of readers
> (`route add <offender>/16 127.0.0.1' is my friend) but these inefficient
> readers really do load down a swerver that could be doing a heck of a
> lot more if it weren't for these poorly-designed reader clients.  Like
> Outhouse Distress, too.  Rrrg.  I feel a rant coming on.  Rrrrr.
>
>
> :: I have done something similar (same newsgroup, but even more unread
> :: articles: >340,000) on another newsserver:
> :: 201 dreader1.news.xs4all.nl NNRP Service Ready -
> :: (This could be a Diablo server?)
>
> Yeah.  A completely different design.  Either you really do have
> all the articles present (and no gaps) or you're not showing the
> entire conversation below:
>
> :: <alt.chello.binaries 0001147847 0000806723 y
> :: >XOVER 806723-807042
> :: <224 data follows
> :: >XOVER 807043-807362
>
> Either you got data, if the articles are present, or you got no
> data, and just a `.' returned if the articles were missing.  I'd
> guess you got data, which would make some difference, since it
> seems you're not prepared to handle the case where you don't get
> data returned that you expect very gracefully.  (Okay, not you,
> I'm not about to bite your head off, substitute `your program'
> where I refer to `you' above except where `you' really means `you'
> and `your program' would make no sense.  And `no' means `maybe'.)
>
>
> :: server differs from this in that when the XOVERs are issued using the
> :: information returned (211 341125 806723 1147847 alt.chello.binaries), an
> :: error is generated for the first XOVER.
>
> Reasonable, if the article(s) you're trying to pull are missing --
> you also have to be prepared to handle the case where the numbers
> you get are correct implying the article was present at time you
> entered the froup, but went missing due to a rogue cancel spree or
> something just before you attempted to get the info for it.
>
>
> I think I need a tank of coffee.
>

-- 
       Mike Castle       Life is like a clock:  You can work constantly
  dalgoda at ix.netcom.com  and be right all the time, or not work at all
www.netcom.com/~dalgoda/ and be right at least twice a day.  -- mrc
    We are all of us living in the shadow of Manhattan.  -- Watchmen