INN commit: branches/2.5 (17 files)

INN Commit rra at isc.org
Tue Aug 18 13:53:55 UTC 2009


    Date: Tuesday, August 18, 2009 @ 06:53:54
  Author: iulius
Revision: 8575

* History retention is now done, when possible, on posting
times and not arrival times. Otherwise, articles may be kept
too long in history.

* HISremember in history API now has a fourth argument
(the article posting date).

* The default value for /remember/ is now 11 (artcutoff + 1
in order to take into account articles posted one day into
the future).

* Documentation rewording and improvements.

Modified:
  branches/2.5/contrib/makeexpctl.in
  branches/2.5/doc/pod/expire.ctl.pod
  branches/2.5/doc/pod/expireover.pod
  branches/2.5/doc/pod/inn.conf.pod
  branches/2.5/doc/pod/install.pod
  branches/2.5/doc/pod/libinnhist.pod
  branches/2.5/doc/pod/news.pod
  branches/2.5/history/his.c
  branches/2.5/history/hisinterface.h
  branches/2.5/history/hisv6/hisv6.c
  branches/2.5/history/hisv6/hisv6.h
  branches/2.5/include/inn/history.h
  branches/2.5/innd/art.c
  branches/2.5/innd/innd.h
  branches/2.5/innd/nc.c
  branches/2.5/innd/util.c
  branches/2.5/samples/expire.ctl

------------------------+
 contrib/makeexpctl.in  |   13 ++++++---
 doc/pod/expire.ctl.pod |   51 +++++++++++++++++++++-----------------
 doc/pod/expireover.pod |    6 ++--
 doc/pod/inn.conf.pod   |   10 +++++--
 doc/pod/install.pod    |   25 +++++++++---------
 doc/pod/libinnhist.pod |   22 +++++++++-------
 doc/pod/news.pod       |   22 ++++++++++++++++
 history/his.c          |    4 +--
 history/hisinterface.h |    2 -
 history/hisv6/hisv6.c  |   40 +++++++++++++++++++++---------
 history/hisv6/hisv6.h  |    2 -
 include/inn/history.h  |    2 -
 innd/art.c             |   33 +++++++++++++++----------
 innd/innd.h            |    2 -
 innd/nc.c              |    4 ++-
 innd/util.c            |    8 ++++--
 samples/expire.ctl     |   62 +++++++++++++++++++++++++++--------------------
 17 files changed, 193 insertions(+), 115 deletions(-)

Modified: contrib/makeexpctl.in
===================================================================
--- contrib/makeexpctl.in	2009-08-18 13:51:41 UTC (rev 8574)
+++ contrib/makeexpctl.in	2009-08-18 13:53:54 UTC (rev 8575)
@@ -37,18 +37,21 @@
 ##	<patterns>:<modflag>:<keep>:<default>:<purge>
 ##  First line gives history retention; other lines specify expiration
 ##  for newsgroups.  Must have a "*:A:..." line which is the default.
-##	<patterns>	wildmat-style patterns for the newsgroups
+##	<patterns>	Wildmat-style patterns for the newsgroups
 ##	<modflag>	Pick one of M U A -- modifies pattern to be only
 ##			moderated, unmoderated, or all groups
-##	<keep>		Mininum number of days to keep article
+##	<keep>		Minimum number of days to keep article
 ##	<default>	Default number of days to keep the article
 ##	<purge>		Flush article after this many days
 ##  <keep>, <default>, and <purge> can be floating-point numbers or the
-##  word "never."  Times are based on when received unless -p is used;
-##  see expire.8
+##  word "never".  Times are based on the arrival time for expire and expireover
+##  (unless -p is used; see expire(8) and expireover(8)), and on the posting
+##  time for history retention.
+##
+##  See the expire.ctl man page for more information.
 
 # How long to remember old history entries for.
-/remember/:2
+/remember/:11
 #
 EOF
 

Modified: doc/pod/expire.ctl.pod
===================================================================
--- doc/pod/expire.ctl.pod	2009-08-18 13:51:41 UTC (rev 8574)
+++ doc/pod/expire.ctl.pod	2009-08-18 13:53:54 UTC (rev 8575)
@@ -28,9 +28,9 @@
     /remember/:<days>
 
 where <days> is a decimal number that specifies the minimum number of days
-a history record for a given message ID is retained, regardless of whether
-the article is present in the spool.  (History entries for articles still
-present in the spool are always retained.)
+a history record for a given message-ID is retained (from its original
+posting time), regardless of whether the article is present in the spool.
+(History entries for articles still present in the spool are always retained.)
 
 The primary reason to retain a record of old articles is in case a peer
 offers old articles that were previously accepted but have already
@@ -38,8 +38,11 @@
 accept the article again and readers would see duplicate articles.
 Articles older than a certain number of days won't be accepted by the
 server at all (see I<artcutoff> in inn.conf(5) and the B<-c> flag in
-innd(8)), and this setting should probably match that time period (10 days
-by default) to ensure that the server never accepts duplicates.
+innd(8)), and this setting should probably match that time period
+to ensure that the server never accepts duplicates.  As the default
+value for I<artcutoff> is C<10> days, it means that C</remember/>
+should be set to C<11> days in order to take into account articles
+whose posting date is one day into the future.
 
 Most of the lines in this file will be in the second format, which
 consists of either four or five colon-separated fields:
@@ -69,15 +72,15 @@
     A   All groups
     X   Remove the article from all groups it appears in
 
-One of M, U, or A must be specified.  X should be used in combination with
-one of the other letters, not by itself.
+One of C<M>, C<U>, or C<A> must be specified.  C<X> should be used in
+combination with one of the other letters, not by itself.
 
 An expiration policy is applied to every article in a newsgroup it
 matches.  There is no way to set an expiration policy for articles
 crossposted to groups you don't carry that's different than other articles
 in the same group.  Normally, articles are not completely deleted until
 they expire out of every group to which they were posted, but if an
-article is expired following a rule where <flag> contains X, it is deleted
+article is expired following a rule where <flag> contains C<X>, it is deleted
 out of all newsgroups to which it was posted immediately.
 
 If I<groupbaseexpiry> is instead set to false, there is no <pattern> and
@@ -90,26 +93,28 @@
 they were posted.
 
 The remaining three fields are the same in either format, and are used to
-determine how long an article should be kept.  Each field should be either
+determine how long an article should be kept from its original arrival
+time (unless the B<-p> flag is passed to expire(8) or expireover(8), in
+which case its original posting time is used).  Each field should be either
 a decimal number of days (fractions like C<8.5> are allowed, but remember
 that articles are only removed when B<expire> or B<expireover> is run,
 normally once a day by B<news.daily>) or the word C<never>.
 
 The middle field, <default>, will be used as the expiration period for
 most articles.  The other two fields, <min> and <max>, only come into
-play if the article requests a particular expiration date with an Expires
-header.  Articles with an Expires header will be expired at the date given
+play if the article requests a particular expiration date with an Expires:
+header.  Articles with an Expires: header will be expired at the date given
 in that header, subject to the constraints that they will be retained at
 least <min> days and no longer than <max> days.
 
 If <min> is set to C<never>, no article matching that line will ever be
 expired.  If <default> is set to C<never>, no article matching that line
-without an explicit Expires header will ever be expired.  If <max> is
-set to C<never>, Expires headers will be honored no matter how far into
+without an explicit Expires: header will ever be expired.  If <max> is
+set to C<never>, Expires: headers will be honored no matter how far into
 the future they are.
 
 One should think of the fields as a lower bound, the default, and an upper
-bound.  Since most articles do not have an Expires header, the second
+bound.  Since most articles do not have an Expires: header, the second
 field is the most important and most commonly applied.
 
 Articles that do not match any expiration rule will not be expired, but
@@ -121,24 +126,24 @@
 file (except for C</remember/>) so that other expiration rules can
 override it.
 
-It is often useful to honor the Expires header in articles, especially
+It is often useful to honor the Expires: header in articles, especially
 those in moderated groups.  To do this, set <min> to zero, <default> to
 whatever normal expiration you wish, and <max> to C<never> or some large
-number, like 365 days for a maximum article life of a year.
+number, like C<365> days for a maximum article life of a year.
 
-To ignore any Expires header, set all three fields to the same value.
+To ignore any Expires: header, set all three fields to the same value.
 
 =head1 EXAMPLES
 
 When I<groupbaseexpiry> is true (the default):
 
-    # Keep expired article history for 10 days, matching artcutoff.
-    /remember/:10
+    # Keep expired article history for 11 days, matching artcutoff plus one.
+    /remember/:11
 
-    # Most articles stay for two weeks, ignoring Expires.
+    # Most articles stay for two weeks, ignoring Expires: headers.
     *:A:14:14:14
 
-    # Accept Expires headers in moderated groups for up to a year and
+    # Accept Expires: headers in moderated groups for up to a year and
     # retain moderated groups for a bit longer.
     *:M:1:30:365
 
@@ -148,8 +153,8 @@
 
 When I<groupbaseexpiry> is false, for class-based expiration:
 
-    # Keep expired article history for 10 days, matching artcutoff.
-    /remember/:10
+    # Keep expired article history for 11 days, matching artcutoff plus one.
+    /remember/:11
 
     # Set a default expiration of seven days.
     *:7:7:7

Modified: doc/pod/expireover.pod
===================================================================
--- doc/pod/expireover.pod	2009-08-18 13:51:41 UTC (rev 8574)
+++ doc/pod/expireover.pod	2009-08-18 13:53:54 UTC (rev 8575)
@@ -15,7 +15,7 @@
 from the overview database mentions of any articles that no longer exist
 in the news spool.
 
-If I<groupbaseexpiry> in I<inn.conf> is true, B<expireover> also removes
+If I<groupbaseexpiry> in F<inn.conf> is true, B<expireover> also removes
 old articles from the news spool according to the expiration rules in
 F<expire.ctl>.  Otherwise it only removes overview entries for articles
 that have already been removed by some other process, and B<-e>, B<-k>,
@@ -93,7 +93,7 @@
 
 B<expireover> normally only checks the existence of articles in the news
 spool if querying the storage method for that article to see if it still
-exists is considered "inexpensive."  To always check the existence of all
+exists is considered "inexpensive".  To always check the existence of all
 articles regardless of how resource-intensive this may be, use the B<-s>
 flag.  See storage.conf(5) for more information about this metric.
 
@@ -103,7 +103,7 @@
 other than the current time.  This is occasionally useful to force groups
 to be expired or not expired without changing F<expire.ctl> for the expire
 run.  I<offset> should be a signed floating point number specifying the
-number of days difference from the current time to use as "now."  This
+number of days difference from the current time to use as "now".  This
 flag is ignored if I<groupbaseexpiry> is false.
 
 =item B<-z> I<rmfile>

Modified: doc/pod/inn.conf.pod
===================================================================
--- doc/pod/inn.conf.pod	2009-08-18 13:51:41 UTC (rev 8574)
+++ doc/pod/inn.conf.pod	2009-08-18 13:53:54 UTC (rev 8575)
@@ -138,10 +138,14 @@
 
 =item I<artcutoff>
 
-Articles older than this number of days are dropped.  This setting should
-probably match the setting on the C</remember/> line in F<expire.ctl>.
-The default value is C<10>.
+Articles older than this number of days are dropped.  The default value
+is C<10>, which means that an incoming article will be rejected if its
+posting date is farther in the past than ten days.
 
+The number on the C</remember/> line in F<expire.ctl> should probably
+be one more than that number in order to take into account articles
+whose posting date is one day into the future.
+
 =item I<bindaddress>
 
 Which IP address innd(8) should bind itself to.  This must be in

Modified: doc/pod/install.pod
===================================================================
--- doc/pod/install.pod	2009-08-18 13:51:41 UTC (rev 8574)
+++ doc/pod/install.pod	2009-08-18 13:53:54 UTC (rev 8575)
@@ -700,7 +700,7 @@
 C<#> symbol are considered comments and are ignored.  Be careful, not all
 files permit comments to begin in the middle of the line.
 
-=head2 inn.conf
+=head2 F<inn.conf>
 
 The first, and most important file is F<inn.conf>.  This file is organized
 as a series of parameter-value pairs, one per line.  The parameter is
@@ -788,7 +788,7 @@
 are only needed for very obscure things, and with more experience running
 your news server the rest will make more sense.
 
-=head2 newsfeeds
+=head2 F<newsfeeds>
 
 F<newsfeeds> determines how incoming articles are redistributed to your
 peers and to other INN processes.  F<newsfeeds> is very versatile and
@@ -966,7 +966,7 @@
 careful:  B<innd> may accept articles faster than B<overchan> can process the
 data.
 
-=head2 incoming.conf
+=head2 F<incoming.conf>
 
 F<incoming.conf> file specifies which machines are permitted to connect to
 your host and feed it articles.  Remote servers you peer with should be
@@ -1016,7 +1016,7 @@
 Note for those familiar with older versions of INN:  this file replaces
 the old F<hosts.nntp> configuration file.
 
-=head2 cycbuff.conf
+=head2 F<cycbuff.conf>
 
 F<cycbuff.conf> is only required if CNFS is used.  If you aren't using
 CNFS, skip this section.
@@ -1124,7 +1124,7 @@
 Finally, you have to create the cycbuffs.  See L<Creating the Article
 Spool (CNFS only)> for more information on how to do that.
 
-=head2 storage.conf
+=head2 F<storage.conf>
 
 F<storage.conf> determines where incoming articles will be stored (what
 storage method, and in the case of CNFS, what metacycbuff).  Each entry in
@@ -1198,20 +1198,20 @@
 newsgroups.  Make note of the storage class IDs you assign as they will be
 needed when you edit F<expire.ctl> a bit later.
 
-=head2 expire.ctl
+=head2 F<expire.ctl>
 
 F<expire.ctl> sets the expiration policy for articles stored on the
 server.  Be careful, since the default configuration will expire most
-articles after 10 days; in most circumstances this deletion is
+articles after 15 days; in most circumstances this deletion is
 I<permanent>, so read this whole section carefully if you want to keep
 local hierarchies forever.  (See archive(8) for a way to automate backups
 of important articles.)
 
 Only one entry is required for all storage classes; it looks like:
 
-    /remember/:10
+    /remember/:11
 
-This entry says how long to keep the Message-IDs for articles that have
+This entry says how long to keep the message-IDs for articles that have
 already expired in the history file so that the server doesn't accept them
 again.  Occasionally, fairly old articles will get regurgitated somewhere
 and offered to you again, so even after you've expired articles from your
@@ -1220,7 +1220,8 @@
 
 INN will reject any articles more than a certain number of days old (the
 I<artcutoff> parameter in F<inn.conf>, defaulting to C<10>); the number on
-the C</remember/> line should match that.
+the C</remember/> line should be one more than that number in order to take
+into account articles whose posting date is one day into the future.
 
 CNFS makes no further use of F<expire.ctl>, since articles stored in CNFS
 buffers expire automatically when the buffer runs out of free space (but
@@ -1237,7 +1238,7 @@
 don't have an Expires: header, those are the only two values that matter.
 For articles with an Expires: header, the other two values come into play;
 the date given in the Expires: header of an article will be honored,
-subject to the contraints set by <keep> and <purge>.  All articles in this
+subject to the constraints set by <keep> and <purge>.  All articles in this
 storage class will be kept for at least <keep> days, regardless of their
 Expires: headers, and all articles in this storage class will be expired
 after <purge> days, even if their Expires: headers specify a longer life.
@@ -1263,7 +1264,7 @@
 indicated expiration status.  All of the other fields have the same
 meaning as above.
 
-=head2 readers.conf
+=head2 F<readers.conf>
 
 Provided that I<noreader> is set to false in F<inn.conf>, any connection
 from a host that doesn't match an entry in F<incoming.conf> (as well as

Modified: doc/pod/libinnhist.pod
===================================================================
--- doc/pod/libinnhist.pod	2009-08-18 13:51:41 UTC (rev 8574)
+++ doc/pod/libinnhist.pod	2009-08-18 13:53:54 UTC (rev 8575)
@@ -45,7 +45,7 @@
 
 B<bool HISwrite(struct history *>I<history>B<, const char *>I<key>B<, time_t >I<arrived>B<, time_t >I<posted>B<, time_t >I<expires>B<, const TOKEN *>I<token>B<);>
 
-B<bool HISremember(struct history *>I<history>B<, const char *>I<key>B<, time_t >I<arrived>B<);>
+B<bool HISremember(struct history *>I<history>B<, const char *>I<key>B<, time_t >I<arrived>B<, time_t >I<posted>B<);>
 
 B<bool HISreplace(struct history *>I<history>B<, const char *>I<key>B<, time_t >I<arrived>B<, time_t >I<posted>B<, time_t >I<expires>B<, const TOKEN *>I<token>B<);>
 
@@ -61,9 +61,9 @@
 
 =head1 DESCRIPTION
 
-These functions provide provide access to the INN history
-database. They maintain key/value pairs in an opaque database whilst
-providing for expiry of outdated information.
+These functions provide access to the INN history database.  They maintain
+key/value pairs in an opaque database whilst providing for expiry of
+outdated information.
 
 The history structure is an opaque handle returned from HISopen.
 
@@ -129,10 +129,11 @@
 
 B<HISremember> writes a new entry to the database I<history>
 associated with I<key>, merely remembering that this I<key> has been
-seen, together with its arrival time I<arrived>. B<HISremember>
-returns B<true> on success, or B<false> on failure. The behaviour when
-I<key> is not unique with respect to the existing entries in
-I<history> is unspecified.
+seen, together with its arrival time I<arrived> and also its posting
+time I<posted>, if known.  (Otherwise, its posting time may be specified
+as <= 0 in case it is absent.)  B<HISremember> returns B<true> on success,
+or B<false> on failure.  The behaviour when I<key> is not unique with
+respect to the existing entries in I<history> is unspecified.
 
 B<HISreplace> replaces an existing entry in the database I<history>,
 associated with I<key>. I<arrived>, I<posted>, I<expired> specify the
@@ -175,11 +176,12 @@
 posting and/or expiry may be zero, but that the token will never be
 B<NULL> (such entries are handled solely via the threshold
 mechanism). The storage token passed to the discrimination function
-may updated if required (for example, as might be needed by a
+may be updated if required (for example, as might be needed by a
 hierachical storage management implementation).
 
-Entries in the database with an arrival time less than I<threshold>
+Entries in the database with a posting time less than I<threshold>
 with no token associated with them are deleted from the database.
+In case the posting time is unknown, the arrival time is used instead.
 
 The parameter I<cookie> is passed to the discrimination function, and
 may be used for any purpose required by the caller.

Modified: doc/pod/news.pod
===================================================================
--- doc/pod/news.pod	2009-08-18 13:51:41 UTC (rev 8574)
+++ doc/pod/news.pod	2009-08-18 13:53:54 UTC (rev 8575)
@@ -52,6 +52,28 @@
 
 =item *
 
+The way the C</remember/> line in F<expire.ctl> works has changed.
+History retention for an article was done according to its original
+arrival time; it is now according to its original posting date.
+Otherwise, unnecessary data may be kept too long in the F<history>
+file.
+
+To achieve that, the HISremember() function in history API now
+expects a fourth parameter:  the article posting time.
+
+Note that article expiration has not changed and is still based
+on arrival time, unless the B<-p> flag is passed to B<expire>
+or B<expireover>, in which case posting time is used.
+
+=item *
+
+The default value for C</remember/> has changed from C<10> to C<11>
+because it should be one more than the I<artcutoff> parameter
+in F<inn.conf>, so that articles posted one day into the future
+are properly retained in history.
+
+=item *
+
 A new B<-L> flag has been added by Jonathan Kamens to B<makehistory>
 so as to specify a load average limit.  If the system load average exceeds
 the specified limit, B<makehistory> sleeps until it goes below the limit.

Modified: history/his.c
===================================================================
--- history/his.c	2009-08-18 13:51:41 UTC (rev 8574)
+++ history/his.c	2009-08-18 13:53:54 UTC (rev 8575)
@@ -276,14 +276,14 @@
 }
 
 bool
-HISremember(struct history *h, const char *key, time_t arrived)
+HISremember(struct history *h, const char *key, time_t arrived, time_t posted)
 {
     bool r;
 
     if (his_checknull(h))
 	return false;
     TMRstart(TMR_HISWRITE);
-    r = (*h->methods->remember)(h->sub, key, arrived);
+    r = (*h->methods->remember)(h->sub, key, arrived, posted);
     if (r == true) {
 	HASH hash;
 

Modified: history/hisinterface.h
===================================================================
--- history/hisinterface.h	2009-08-18 13:51:41 UTC (rev 8574)
+++ history/hisinterface.h	2009-08-18 13:53:54 UTC (rev 8575)
@@ -31,7 +31,7 @@
     bool (*walk)(void *, const char *, void *,
 		 bool (*)(void *, time_t, time_t, time_t,
 			  const struct token *));
-    bool (*remember)(void *, const char *, time_t);
+    bool (*remember)(void *, const char *, time_t, time_t);
     bool (*ctl)(void *, int, void *);
 } HIS_METHOD;
 

Modified: history/hisv6/hisv6.c
===================================================================
--- history/hisv6/hisv6.c	2009-08-18 13:51:41 UTC (rev 8574)
+++ history/hisv6/hisv6.c	2009-08-18 13:53:54 UTC (rev 8575)
@@ -734,9 +734,20 @@
     const char *hashtext = HashToText(*hash);
 
     if (token == NULL) {
-	i = snprintf(s, HISV6_MAXLINE, "[%s]%c%lu%c%c\n",
-		     hashtext, HISV6_FIELDSEP,
-		     (unsigned long)arrived, HISV6_SUBFIELDSEP, HISV6_NOEXP);
+        /* Only a line to remember an article.  We keep its arrival
+         * and posting time. */
+        if (posted <= 0) {
+            i = snprintf(s, HISV6_MAXLINE, "[%s]%c%lu%c%c\n",
+                         hashtext, HISV6_FIELDSEP,
+                         (unsigned long)arrived, HISV6_SUBFIELDSEP,
+                         HISV6_NOEXP);
+        } else {
+            i = snprintf(s, HISV6_MAXLINE, "[%s]%c%lu%c%c%c%lu\n",
+                         hashtext, HISV6_FIELDSEP,
+                         (unsigned long)arrived, HISV6_SUBFIELDSEP,
+                         HISV6_NOEXP, HISV6_SUBFIELDSEP,
+                         (unsigned long)posted);
+        }
     } else {
 	const char *texttok;
 
@@ -881,10 +892,11 @@
 
 
 /*
-**  remember a history entry, key, with arrival time arrived.
+**  Remember a history entry, key, with arrival time, and also
+**  posting time if known.
 */
 bool
-hisv6_remember(void *history, const char *key, time_t arrived)
+hisv6_remember(void *history, const char *key, time_t arrived, time_t posted)
 {
     struct hisv6 *h = history;
     HASH hash;
@@ -892,7 +904,7 @@
 
     his_logger("HISwrite begin", S_HISwrite);
     hash = HashMessageID(key);
-    r = hisv6_writeline(h, &hash, arrived, 0, 0, NULL);
+    r = hisv6_writeline(h, &hash, arrived, posted, 0, NULL);
     his_logger("HISwrite end", S_HISwrite);
     return r;
 }
@@ -1140,18 +1152,24 @@
 	    keep = (*hiscookie->cb.expire)(hiscookie->cookie,
 					   arrived, posted, expires,
 					   t);
-	    /* if the callback returns true, we should keep the
-	       token for the time being, else we just remember
-	       it */
+	    /* If the callback returns true, we should keep the
+	     * token for the time being, else we just remember
+	     * it. */
 	    if (keep == false) {
 		t = NULL;
-		posted = expires = 0;
+		expires = 0;
 	    }
 	} else {
 	    t = NULL;
 	}
+        /* When t is NULL (no token), the message-ID is removed from
+         * history when the posting time of the article is older than
+         * threshold, as set by the /remember/ line in expire.ctl.
+         * We keep the check for the arrival time because some entries
+         * might not have one. */
 	if (hiscookie->new &&
-	    (t != NULL || arrived >= hiscookie->threshold)) {
+	    (t != NULL || posted >= hiscookie->threshold
+             || (posted <= 0 && arrived >= hiscookie->threshold))) {
 	    r = hisv6_writeline(hiscookie->new, hash,
 				 arrived, posted, expires, t);
 	}

Modified: history/hisv6/hisv6.h
===================================================================
--- history/hisv6/hisv6.h	2009-08-18 13:51:41 UTC (rev 8574)
+++ history/hisv6/hisv6.h	2009-08-18 13:53:54 UTC (rev 8575)
@@ -38,7 +38,7 @@
 
 const char *hisv6_error(void *);
 
-bool hisv6_remember(void *, const char *key, time_t arrived);
+bool hisv6_remember(void *, const char *key, time_t arrived, time_t posted);
 
 bool hisv6_ctl(void *, int, void *);
 

Modified: include/inn/history.h
===================================================================
--- include/inn/history.h	2009-08-18 13:51:41 UTC (rev 8574)
+++ include/inn/history.h	2009-08-18 13:53:54 UTC (rev 8575)
@@ -90,7 +90,7 @@
 bool                    HIScheck(struct history *, const char *);
 bool                    HISwrite(struct history *, const char *, time_t,
 				 time_t, time_t, const struct token *);
-bool                    HISremember(struct history *, const char *, time_t);
+bool                    HISremember(struct history *, const char *, time_t, time_t);
 bool                    HISreplace(struct history *, const char *, time_t,
 				   time_t, time_t, const struct token *);
 bool                    HISexpire(struct history *, const char *, const char *,

Modified: innd/art.c
===================================================================
--- innd/art.c	2009-08-18 13:51:41 UTC (rev 8574)
+++ innd/art.c	2009-08-18 13:53:54 UTC (rev 8575)
@@ -869,7 +869,9 @@
 	if (HDR_FOUND(HDR__MESSAGE_ID)) {
             HDR_PARSE_START(HDR__MESSAGE_ID);
             msgid = HDR(HDR__MESSAGE_ID);
-            if (!HIScheck(History, msgid) && !InndHisRemember(msgid))
+            /* The article posting time has not been parsed.  We cannot
+             * give it to InndHisRemember. */
+            if (!HIScheck(History, msgid) && !InndHisRemember(msgid, 0))
                 warn("SERVER cant write %s", msgid);
 	}
     }
@@ -1048,6 +1050,7 @@
   TMRstart(TMR_ARTCLEAN);
   data->Arrived = Now.tv_sec;
   data->Expires = 0;
+  data->Posted = 0;
 
   /* replace trailing '\r\n' with '\0\n' of all system header to be handled
      easily by str*() functions */
@@ -1297,7 +1300,7 @@
       TMRstop(TMR_ARTCNCL);
       return;
     }
-    InndHisRemember(MessageID);
+    InndHisRemember(MessageID, data->Posted);
     snprintf(buff, sizeof(buff), "Cancelling %s",
              MaxLength(MessageID, MessageID));
     ARTlog(data, ART_CANC, buff);
@@ -2038,8 +2041,10 @@
   }
   if (!artclean) {
     ARTlog(data, ART_REJECT, cp->Error);
+    /* If the article posting time has not been properly parsed, data->Posted
+     * will be negative or zero. */
     if (innconf->remembertrash && (Mode == OMrunning) &&
-	!InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	!InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
       syslog(L_ERROR, "%s cant write history %s %m", LogName,
 	HDR(HDR__MESSAGE_ID));
     ARTreject(REJECT_OTHER, cp);
@@ -2072,7 +2077,7 @@
                MaxLength(ME.Exclusions[j], ME.Exclusions[j]));
       ARTlog(data, ART_REJECT, cp->Error);
       if (innconf->remembertrash && (Mode == OMrunning) &&
-	  !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	  !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
 	syslog(L_ERROR, "%s cant write history %s %m", LogName,
 	  HDR(HDR__MESSAGE_ID));
       ARTreject(REJECT_SITE, cp);
@@ -2104,7 +2109,7 @@
              cp->Error);
       ARTlog(data, ART_REJECT, cp->Error);
       if (innconf->remembertrash && (Mode == OMrunning) &&
-	  !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	  !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
 	syslog(L_ERROR, "%s cant write history %s %m", LogName,
 	  HDR(HDR__MESSAGE_ID));
       ARTreject(REJECT_FILTER, cp);
@@ -2135,7 +2140,7 @@
              cp->Error);
       ARTlog(data, ART_REJECT, cp->Error);
       if (innconf->remembertrash && (Mode == OMrunning) &&
-	  !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	  !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
 	syslog(L_ERROR, "%s cant write history %s %m", LogName,
 	  HDR(HDR__MESSAGE_ID));
       ARTreject(REJECT_FILTER, cp);
@@ -2152,7 +2157,7 @@
                MaxLength(HDR(HDR__DISTRIBUTION), HDR(HDR__DISTRIBUTION)));
       ARTlog(data, ART_REJECT, cp->Error);
       if (innconf->remembertrash && Mode == OMrunning &&
-	  !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	  !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
         syslog(L_ERROR, "%s cant write history %s %m", LogName,
 	  HDR(HDR__MESSAGE_ID));
       ARTreject(REJECT_DISTRIB, cp);
@@ -2170,7 +2175,7 @@
                            data->Distribution.List[0]));
 	ARTlog(data, ART_REJECT, cp->Error);
         if (innconf->remembertrash && (Mode == OMrunning) &&
-	    !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	    !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
 	  syslog(L_ERROR, "%s cant write history %s %m",
 	    LogName, HDR(HDR__MESSAGE_ID));
 	ARTreject(REJECT_DISTRIB, cp);
@@ -2277,7 +2282,7 @@
                    MaxLength(p, p));
           ARTlog(data, ART_REJECT, cp->Error);
           if (innconf->remembertrash && (Mode == OMrunning) &&
-              !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+              !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
               syslog(L_ERROR, "%s cant write history %s %m",
                      LogName, HDR(HDR__MESSAGE_ID));
           ARTreject(REJECT_GROUP, cp);
@@ -2318,7 +2323,7 @@
                MaxLength(ngp->Name, ngp->Name));
       ARTlog(data, ART_REJECT, cp->Error);
       if (innconf->remembertrash && (Mode == OMrunning) &&
-	  !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	  !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
 	syslog(L_ERROR, "%s cant write history %s %m", LogName,
 	  HDR(HDR__MESSAGE_ID));
       ARTreject(REJECT_UNAPP, cp);
@@ -2422,7 +2427,8 @@
       ARTlog(data, ART_REJECT, cp->Error);
       if (!innconf->wanttrash) {
 	if (innconf->remembertrash && (Mode == OMrunning) &&
-	  !NoHistoryUpdate && !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	  !NoHistoryUpdate && !InndHisRemember(HDR(HDR__MESSAGE_ID),
+                                               data->Posted))
 	  syslog(L_ERROR, "%s cant write history %s %m",
 	    LogName, HDR(HDR__MESSAGE_ID));
 	ARTreject(REJECT_GROUP, cp);
@@ -2435,7 +2441,8 @@
          * which you explicitly excluded in your active file. */
   	if (!GroupMissing) {
 	  if (innconf->remembertrash && (Mode == OMrunning) &&
-	      !NoHistoryUpdate && !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+	      !NoHistoryUpdate && !InndHisRemember(HDR(HDR__MESSAGE_ID),
+                                                   data->Posted))
 	    syslog(L_ERROR, "%s cant write history %s %m",
 	      LogName, HDR(HDR__MESSAGE_ID));
 	  ARTreject(REJECT_GROUP, cp);
@@ -2472,7 +2479,7 @@
       }
       ARTlog(data, ART_REJECT, cp->Error);
       if (innconf->remembertrash && (Mode == OMrunning) &&
-          !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+          !InndHisRemember(HDR(HDR__MESSAGE_ID), data->Posted))
           syslog(L_ERROR, "%s cant write history %s %m",
                  LogName, HDR(HDR__MESSAGE_ID));
       ARTreject(REJECT_OTHER, cp);

Modified: innd/innd.h
===================================================================
--- innd/innd.h	2009-08-18 13:51:41 UTC (rev 8574)
+++ innd/innd.h	2009-08-18 13:53:54 UTC (rev 8575)
@@ -666,7 +666,7 @@
 extern bool             InndHisWrite(const char *key, time_t arrived,
 				     time_t posted, time_t expires,
 				     TOKEN *token);
-extern bool             InndHisRemember(const char *key);
+extern bool             InndHisRemember(const char *key, time_t posted);
 extern void             InndHisLogStats(void);
 extern bool		FormatLong(char *p, unsigned long value, int width);
 extern bool		NeedShell(char *p, const char **av, const char **end);

Modified: innd/nc.c
===================================================================
--- innd/nc.c	2009-08-18 13:51:41 UTC (rev 8574)
+++ innd/nc.c	2009-08-18 13:53:54 UTC (rev 8575)
@@ -1000,8 +1000,10 @@
         if (innconf->remembertrash && (Mode == OMrunning)
             && HDR_FOUND(HDR__MESSAGE_ID)) {
             HDR_PARSE_START(HDR__MESSAGE_ID);
+            /* The article posting time has not been parsed.  We cannot
+             * give it to InndHisRemember. */
             if (!HIScheck(History, HDR(HDR__MESSAGE_ID))
-                && !InndHisRemember(HDR(HDR__MESSAGE_ID)))
+                && !InndHisRemember(HDR(HDR__MESSAGE_ID), 0))
                 syslog(L_ERROR, "%s cant write history %s %m",
                        LogName, HDR(HDR__MESSAGE_ID));
             HDR_PARSE_END(HDR__MESSAGE_ID);

Modified: innd/util.c
===================================================================
--- innd/util.c	2009-08-18 13:51:41 UTC (rev 8574)
+++ innd/util.c	2009-08-18 13:53:54 UTC (rev 8575)
@@ -353,10 +353,14 @@
     return r;
 }
 
+/*
+**  Remember the rejected article.  Its posting time is negative or zero
+**  in case it is unknown.  Its arrival time is now.
+*/
 bool
-InndHisRemember(const char *key)
+InndHisRemember(const char *key, time_t posted)
 {
-    bool r = HISremember(History, key, Now.tv_sec);
+    bool r = HISremember(History, key, Now.tv_sec, posted);
 
     if (r != true)
 	IOError("history remember", errno);

Modified: samples/expire.ctl
===================================================================
--- samples/expire.ctl	2009-08-18 13:51:41 UTC (rev 8574)
+++ samples/expire.ctl	2009-08-18 13:53:54 UTC (rev 8575)
@@ -1,32 +1,42 @@
-##  $Revision$
-##  expire.ctl - expire control file
+##  $Id$
+##
+##  Sample configuration file for article expiration.
+##
 ##  Format:
-##	/remember/:<keep>
-##	<class>:<min>:<default>:<max>
-##	<wildmat>:<flag>:<min>:<default>:<max>
+##      /remember/:<keep>
+##      <class>:<min>:<default>:<max>
+##      <wildmat>:<flag>:<min>:<default>:<max>
+##
 ##  First line gives history retention; second line specifies expiration
 ##  for classes; third line specifies expiration for group if groupbaseexpiry
-##  is true
-##	<class>		class specified in storage.conf
-##	<wildmat>	wildmat-style patterns for the newsgroups
-##	<min>		Mininum number of days to keep article
-##	<default>	Default number of days to keep the article
-##	<max>		Flush article after this many days
-##  <min>, <default>, and <max> can be floating-point numbers or the
-##  word "never."  Times are based on when received unless -p is used;
-##  see expire.8
+##  is true in inn.conf.
+##      <class>         Class specified in storage.conf.
+##      <wildmat>       Wildmat-style patterns for the newsgroups.
+##      <flag>          Status of the newsgroups.
+##      <keep>          Number of days to retain a message-ID in history.
+##      <min>           Minimum number of days to keep the article.
+##      <default>       Default number of days to keep the article.
+##      <max>           Flush the article after this many days.
+##  <keep>, <min>, <default> and <max> can be floating-point numbers or the
+##  word "never".  Times are based on the arrival time for expire and expireover
+##  (unless -p is used; see expire(8) and expireover(8)), and on the posting
+##  time for history retention.
+##
+##  See the expire.ctl man page for more information.
 
-##  If article expires before 10 days, we still remember it for 10 days in
-##  case we get offered it again.  Depending on what you use for the innd
-##  -c flag and how paranoid you are about old news, you might want to
-##  make this 28, 30, etc, but it's probably safe to reduce it to 7 in most
-##  cases if you want to keep your history file smaller.
-/remember/:10
+##  When an article is rejected or expires before 10 days, we still remember
+##  it for 11 days from its original posting time in case we get offered it
+##  again.  See the artcutoff parameter in inn.conf; it should match this
+##  parameter (/remember/ uses 11 days instead of 10 in order to take into
+##  account articles whose posting date is one day into the future).
+/remember/:11
 
-##  Keep for 1-10 days, allow Expires headers to work.  This entry uses
-##  the syntax appropriate when groupbaseexpiry is true in inn.conf.
-*:A:1:10:never
+##  Keep for 1-15 days, allow Expires: headers to work.  This entry uses
+##  the syntax appropriate when groupbaseexpiry is true in inn.conf.  Times
+##  are based on the arrival time (unless -p is used).
+*:A:1:15:never
 
-##  Keep for 1-10 days, allow Expires headers to work.  This is an entry
-##  based on storage class, used when groupbaseexpiry is false.
-#0:1:10:never
+##  Keep for 1-15 days, allow Expires: headers to work.  This is an entry
+##  based on storage class, used when groupbaseexpiry is false.  Times
+##  are based on the arrival time (unless -p is used).
+#0:1:15:never




More information about the inn-committers mailing list