[InterNetNews/inn] ee8f71: expire: skip per-article SMretrieve (cancel log)

[Kevin Bowling]

  Branch: refs/heads/main
  Home:   https://github.com/InterNetNews/inn
  Commit: ee8f717352483ec53c2883c209d551e927defc92
      https://github.com/InterNetNews/inn/commit/ee8f717352483ec53c2883c209d551e927defc92
  Author: Kevin Bowling <kevin.bowling at kev009.com>
  Date:   2026-05-24 (Sun, 24 May 2026)

  Changed paths:
    M MANIFEST
    M doc/pod/expire.pod
    M doc/pod/expireover.pod
    M doc/pod/inn.conf.pod
    M doc/pod/sm.pod
    M expire/expire.c
    M expire/expireover.c
    M expire/expirerm.in
    M expire/makehistory.c
    M frontends/sm.c
    M include/inn/innconf.h
    M include/inn/ov.h
    M include/inn/storage.h
    A include/inn/tombstone.h
    M innd/art.c
    M lib/innconf.c
    M samples/inn.conf.in
    M storage/Makefile
    M storage/expire.c
    M storage/interface.c
    M storage/ovinterface.h
    A storage/tombstone.c
    M support/mkmanifest
    M tests/Makefile
    M tests/TESTS
    A tests/expire/tombstone-e2e.t
    A tests/expire/tombstone-hisexpire-t.c
    A tests/expire/tombstone-t.c
    A tests/storage/cancel-tombstone-t.c

  Log Message:
  -----------
  expire: skip per-article SMretrieve (cancel log)

Adds an opt-in (expiretombstone in inn.conf, default false)
side-channel that records every cancellation alongside the existing
storage and overview deletions, so a subsequent expire run can drop
matching history entries without doing per-article
SMretrieve(RETR_STAT) syscalls.  At billion-article scale on
tradspool/timehash this turns a billion access(2) calls into a few
thousand hash lookups.

Two log files cooperate:

  ${pathdb}/expireover.tombstone -- written by expireover after each
  successful inline SMcancel; in delayrm mode written up front and
  promoted by expirerm after fastrm succeeds.  Atomic .NEW -> final
  rename under an exclusive non-blocking fcntl POSIX lock.  A
  crashed-or-failed previous run's leftover .NEW is verified
  per-token via SMretrieve and merged into the next run, so partial
  rmfile failures cannot orphan articles on disk.

  ${pathdb}/cancels.tombstone -- appended by innd's ARTcancel and by
  sm -r through a new public SMcanceltombstone() helper.  Shared
  fcntl POSIX lock for appenders, exclusive for expire's snapshot
  rename to .processing; one retry on EAGAIN closes the appender
  vs consumer race window.  POSIX O_APPEND atomicity for sub-
  PIPE_BUF writes keeps single-line tokens from interleaving.

expire reads both files into one hashset, treats every article in
either log as already gone, and falls back to per-article SMretrieve
only for storage methods where SMprobe(SELFEXPIRE) is true (CNFS
wrap-around).  An empty-but-present tombstone correctly says "no
cancels this cycle" and skips SMretrieve for everything; truly absent
files fall back to the slow path.

The format is line-per-token; readers ignore blank and #-prefixed
lines, and expireover writes "# inn-tombstone v1" as a forward-
compatibility header.  Disk and memory footprint scale linearly:
~38 bytes per entry on disk, ~50 bytes in expire's hash table.

Tests cover the public library (tombstone_hash_create, _read,
_rename_for_processing, _present), SMcanceltombstone branches
including expiretombstone=false bypass and unwritable-pathdb
non-crash, two-file merge with dedup, comment-line skip,
HISexpire integration with a callback that mirrors EXPdoline's
full decision tree (kept / tombstoned / selfexpire-gone /
selfexpire-alive), and an end-to-end shell test that exercises
sm + cancels.tombstone + expiretombstone toggling.

Default is false; sites should opt in after validating the
option's behavior against their workload.

see #340



To unsubscribe from these emails, change your notification settings at https://github.com/InterNetNews/inn/settings/notifications