INNd filtering & Log changes for discussion

Russ Allbery rra at stanford.edu
Sun Apr 30 05:08:14 UTC 2006


Bjoern A Zeeb <bzeeb at zabbadoz.net> writes:

> basically I have three things I'd like to discuss with you before I am
> going to read any C code in detail and patch it (or perhaps you will
> do;-)...

> 1) I turned on status: 300 in inn.conf and restarted INN but never go
>    a status file. The problem is that it's hard to debug because Fopen
>    doesn't or log an error.

Fopen itself shouldn't log an error; it should return NULL (which it
does).  However, innd/status.c should log an error when Fopen returns
NULL, but apparently doesn't.  A patch to fix that would be gratefully
accepted, or I'll get around to it at some point.

> 2) I have started to enhanced the filter.perl we use (cleanfeed) to
>    achieve various things - specially per peer statistics gathering
>    (also had to change innreport with these changes but that
>    shouldn't matter for this question).

>    What I was specially interested in was to have the article size
>    of what cleanfeed rejects in the logs (along with the newsgroups).
>    For now I am using the less efficient workaround to calculate
>    length $hdr{__BODY__} because I cannot rely on {Bytes}.

Why would length ($hdr{__BODY__}) be less efficient?  That seems to be
exactly the number that you want, and since Perl uses counted strings
internally, it should be as efficient as you could possibly want.

length in Perl is not a call to strlen.

> 3) We do have plenty of storage on the newsserver itself which
>    generally is not a bad thing;) But as our history is already
>    _big_, we don't want junk, etc. we have turned off remembertrash
>    in inn.conf for some time now. Instead I am now using the
>    filter_messageid API together with cleanfeeds reject function to
>    achieve about the same: every article that gets rejected by cleanfeeds
>    reject routine (from filter_art hook) is stored in something
>    similar to Cleanfeed::Queue. The only differences are that
>    I have a) also added an upper limit of Message-IDs which I
>    have not yet hit btw, and b) I am also storing the original
>    reject message (including the article size, etc. from 2)) so the
>    time the Message-ID gets rejected again later I can log the original
>    reason plus my additional information and an extra hint that it
>    was catched with my logic in filter_messageid.

That's an interesting solution to that problem.  Sure, that should work.

>    The main problem here turned out to be that innd/nc.c doesn't
>    log those refused articles to Log for CHECK or IHAVE.

Good heavens, no.  Any sort of normal server would hammer its logs to
death by logging every refusal.

>    So I am using the INN::syslog callback for now to get them
>    logged but of course I am missing the peer name and NNTP_HAVEIT_VAL
>    resp. NNTP_ERR_GOTID_VAL.

>    So the question is - why does nc.c not log these refused articles
>    and could we (optionally) make it do that?

It would have to be optional.  I guess I'd be willing to accept a clean
patch, but I'm not particularly interested in doing this work myself since
this sounds like a very strange site and a very special case.  I suppose
it might be somewhat useful occasionally for debugging....

>    Is it because people are worried about log file size or because it
>    is in a 'critical performance path'?

It would, for most sites, increase the size of their logs by a factor of
25-50 or more, with a similarly multiplicative factor in the amount of
disk writing that INN does.

>    Alternatively could we make the "additional" information like
>    peername also available to other hooks of filters?

I'd be glad to do this, but it should probably happen as part of a
complete redesign of the filter API that supports languages in a more
separated and pluggable fashion and cleans up a lot of interface ugliness,
particularly around the Perl filters.  This is on the long-term to-do
list, although I'd certainly welcome work from someone else if someone
wanted to tackle it as a project.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>


More information about the inn-workers mailing list