innduct - a replacement for innfeed/nntpsend, 0.1 alpha

Ian Jackson ijackson at
Sun May 23 00:33:31 UTC 2010

I've been running nntpsend for quite a while.  I wanted to run
realtime feeds but I have a problem with innfeed from a reliability
point of view[1].

In particular, the program feed protocol spoken between innd and
innfeed is lossy: if innfeed dies unexpectedly, articles which innd
has written to the pipe to innfeed will be skipped.  innd has no way
of telling which articles those are, no useful records, and no code to
retry.  I can see no sensible way to solve this problem without using
a different approach to getting articles from innd.

 [1] By "reliable" I mean that articles will be propagated unless
     either (a) there is some deliberate policy or technical reason
     why they are not or (b) transient problems prevent the
     propagation from succeeding before the article expires.

I looked at editing innfeed to make it tail the feed file with
inotify/kqueue but innfeed is an very large program for what it does -
nearly 25kloc.

So I set out to write something similar but smaller and reliable.
I think I've succeeded.  The resulting program is 4.5kloc, under a
fifth of the size of innfeed.

Since this is rather an alpha release, I'm not providing a tarball or
(heaven forbid) binary packages or anything.  But you can get the
source with:
  git clone git://

Comments appreciated.

Good points:

 * It's fairly small and straightforward to run.

 * There's a comprehensive manpage.

Bad points:

 * Portability isn't wonderful.  It definitely needs GCC and a fairly
   modern libc; there are some things which configure should handle
   which it doesn't (eg, finding your libinn etc.); currently it only
   supports Linux's inotify, not BSD's kqueue, although the latter
   should be easy enough to plumb in and in the absence of the feature
   it will happily poll.

 * You really need the lib/remopen.c patch I posted earlier or you get
   no useful error message if the domain name you specify is wrong.

 * There isn't a config file reader or anything like that and it runs
   just one site at a time.  You have to invoke one instance per site
   (although this is easily done out of cron in a "restart it if not
   running" mode).

For your edification, I have included a plain text formatting of the


INNDUCT(8)                                                          INNDUCT(8)

       innduct - quickly and reliably stream Usenet articles to remote site

       innduct [options] site [fqdn]

       innduct  implements  NNTP  peer-to-peer news transmission including the
       streaming extensions, for sending news articles to a remote  site.   It
       is intended as a replacement for innfeed or nntpsend and innxmit.

       You  need  to  run one instance of innduct for each peer site.  innduct
       manages its interaction with  innd,  including  flushing  the  feed  as
       appropriate,  etc.,  so that articles are transmitted quickly, and man-
       ages the retransmission of its own backlog.  innduct includes the lock-
       ing necessary to avoid multiple simutaneous invocations.

       By  default,  innduct  reads  the default feedfile corresponding to the
       site site (ie pathoutgoing/site) and feeds it via  NNTP,  streaming  if
       possible,  to  the host fqdn.  If fqdn is not specified, it defaults to

       innduct daemonises after argument parsing, and all  logging  (including
       error messages) are sent to syslog (facility news).

       The  best  way to run innduct is probably to periodically invoke it for
       each feed (e.g. from cron), passing the -q option to arrange that  inn-
       duct silently exits if an instance is already running for that site.

              Specifies  the  feedfile  to  read, and indirectly specifies the
              paths to be  used  for  various  associated  files  (see  FILES,
              below).  If specified as DIR/ it is taken as a directory to use,
              and the actual feed file used is path/site.  If PATH or DIR does
              not  start  with a /, it is taken to be relative to pathoutgoing
              from inn.conf.  The default is site.

              Makes innduct silently exit (with status 0) if  another  innduct
              holds  the  lock  for the site.  Without -q, this causes a fatal
              error to be logged and a nonzero exit.

              Do not daemonise.  innduct runs in the foreground, but otherwise
              operates normally (logging to syslog, etc.).

              Do  not  daemonise.  innduct runs in the foreground and all mes-
              sages (including all  debug  messages)  are  written  to  stderr
              rather than syslog.  A control command line is also available on

              Do not try to use the streaming extensions to NNTP (for  use  eg
              if the peer can't cope when we send MODE STREAM).

              Do  not  try  to use the file change monitoring support to watch
              for writes by innd to the feed file; poll it instead.  (If  file
              monitoring  is  not compiled in, this option just downgrades the
              log message which warns about this situation.)

              Read FILE instead of the default inn.conf.

              Connect to port PORT at the remote site rather than to the  NNTP
              port (119).

              Change  directory to pathrun at startup.  The default is pathrun
              from inn.conf.

              Listen for control command line connections on CLI-DIR/site  (if
              the  value ends with a /) or CLI-PATH (if it doesn't).  See CON-
              TROLLING INNDUCT, below.  Note that  there  is  a  fairly  short
              limit  on the lengths of AF_UNIX socket pathnames.  If specified
              as CLI-DIR/, the directory will be created with mode 700 if nec-
              essary.   The  default  is  innduct/  which means to create that
              directory in PATHRUN and listen on PATHRUN/innduct/site.

       --help Just print a brief usage message and list of the options to std-

       See TUNING OPTIONS below for more options.

       If you tell innd to drop the feed, innduct will (when it notices, which
       will normally be the next time it decides to flush) finish up the arti-
       cles  it  has in hand now, and then exit.  It is harmless to cause innd
       to flush the feed (but innduct won't notice and flushing won't start  a
       new feedfile; you have to leave that to innduct).

       If  you  want to stop innduct you can send it SIGTERM or SIGINT, or the
       stop control command, in which case it will report  statistics  so  far
       and  quickly  exit.  If innduct receives SIGKILL nothing will be broken
       or corrupted; you just won't see some of the article stats.

       innduct  listens  on  an  AF_UNIX  socket  (by  default,   pathrun/inn-
       duct/site),  and provides a command-line interface which can be used to
       trigger various events and for debugging.  When a  connection  arrives,
       innduct  writes  a  prompt, reads commands a line at a time, and writes
       any output back to the caller.  (Everything uses  unix  line  endings.)
       The  cli can most easily be accessed with a program like netcat-openbsd
       (eg nc.openbsd -U /var/run/news/innduct/site) or socat.  The prompt  is

       The following control commands are supported:

       h      Print a list of all the commands understood.  This list includes
              undocumented commands which mess with innduct's  internal  state
              and  should  only  be used by a developer in conjuction with the
              innduct source code.

       flush  Start a new feed file and trigger a flush  of  the  feed.   (Or,
              cause the FLUSH-FINISH-PERIOD to expire early, forcibly complet-
              ing a previously started flush.)

       stop   Log statistics and exit.  (Same effect as SIGTERM or SIGINT.)

       dump q|a
              Writes information about innduct's state to a  plain  text  file
              feedfile_dump.  This overwrites any previous dump.  innduct does
              not ever delete these  dump  files.   dump  q  gives  a  summary
              including  general  state and a list of connections; dump a also
              includes information about each article innduct is dealing with.

       next blscan
              Requests  that  innduct rescan for new backlog files at the next
              PERIOD poll.  Normally innduct assumes that  any  backlog  files
              dropped  in  by the administrator are not urgent, and it may not
              get around to noticing them for BACKLOG-SCAN-PERIOD.

       next conn
              Resets the connection startup delay counter so that innduct  may
              consider making a new connection to the peer right away, regard-
              less of the setting of RECONNECT-PERIOD.  A  connection  attempt
              will  still only be made if innduct feels that it needs one, and
              innduct may wait up  to  PERIOD  before  actually  starting  the

       You should not normally need to adjust these.  Time intervals may spec-
       ified in seconds, or as a number  followed  by  one  of  the  following
       units: s m h d, sec min hour day, das hs ks Ms.

              Restricts  the  maximum  number of simultaneous NNTP connections
              per peer to max.  There is no global limit on the number of con-
              nections  used  by  all innducts, as the instances for different
              sites are entirely independent.  The default is 10.

              Restricts the maximum number of outstanding articles  queued  on
              any  particular  connection  to max.  (Non-streaming connections
              can only handle one article at a time.)  The default is 200.

              Restricts the maximum number articles read into  core  from  any
              one input file to max.  The default is twice per-conn-max.

              Specifies  that  innduct  should  flush the feed and start a new
              feedfile when the existing  feedfile  size  exceeds  bytes;  the
              effect  is that the innduct will try to avoid the various batch-
              files growing much beyond this size.  The default is 100000.

              Specifies wakup interval and period granularity.  innduct  wakes
              up  every  PERIOD-INTERVAL  to  do  various housekeeping checks.
              Also, many of the timeout and rescan intervals (those  specified
              in this manual as PERIOD) are rounded up to the next multiple of
              PERIOD-INTERVAL.  The default is 30s.

              How long to allow for a connection setup attempt  before  giving
              up.  The default is 200s.

              How  long  to wait for innd to respond to a flush request before
              giving up.  The default is 100s.

              How often to poll the feedfile for new articles written by  innd
              if  file  monitoring  (inotify  or equivalent) is not available.
              (When file monitoring is available, there is no need  for  peri-
              odic  checks  and  we  wake immediately up whenever the feedfile
              changes.)  The default is 5s.

              If the moving  average  of  the  proportion  of  articles  being
              accepted  (rather than declined) by the peer exceeds this value,
              innduct uses "no check mode" - ie it just  sends  the  peer  the
              articles  with  TAKETHIS  rather  than checking first with CHECK
              whether the article is wanted.  This only affects streaming con-
              nections.  The default is 95 (ie, 95%).

              The  moving  average  mentioned above is an alpha-smoothed value
              with a half-life of ARTICLES.  The default is 100.

              Limits initiation of new  connections  to  one  each  RECONNECT-
              PERIOD.   This  applies  to  reconnections  if the peer has been
              down, and also to ramping up the number of  connections  we  are
              using  after  startup  or  in response to an article flood.  The
              default is 1000s.

              If our attempt to flush the feed failed (usually  this  will  be
              because  innd  is  not  running),  try  again after PERIOD.  The
              default is 1000s.

              When the peer responds to our offer of an article with a 431  or
              436  NNTP response code, indicating that the article has already
              been offered to it by another of its peers, and that  we  should
              try again, we wait at least PERIOD.  before offering the article
              again.  The default is 100s.

              We scan the directory containing feedfile for backlog  files  at
              least  every  BACKLOG-SCAN-PERIOD, in case the administrator has
              manually dropped in a file there for processing.  The default is

              We flush the feed and start a new feedfile at least every PERIOD
              even if the current instance of the feedfile has not reached the
              size threshold.  The default is 100000s.

              If  we  flushed FLUSH-FINISH-PERIOD ago, and are still trying to
              finish processing articles that were written  to  the  old  feed
              file, we forcibly and violently make sure that we can finish the
              old feed file: we abandon and defer all the work, which includes
              unceremoniously  dropping  any  connections  on which we've sent
              some of those articles but not yet had replies, as they're prob-
              ably stuck somehow.  The default is 2000s.

              Connections  which  have  had  no  activity  for  PERIOD will be
              closed.  This includes connections where we have  sent  commands
              or  articles  but  have  not yet had the responses, so this same
              value doubles as the timeout after which we  conclude  that  the
              peer  is  unresponsive or the connection has become broken.  The
              default is 1000s.

       --low-volume-thresh=WIN-THRESH --low-volume-window=PERIOD
              If innduct has only one connection to the  peer,  and  has  pro-
              cessed  fewer  than  WIN-THRESH  articles in the last PERIOD and
              also no articles in the last PERIOD-INTERVAL it will  close  the
              connection  quickly.   That is, innduct switches to a mode where
              it opens a connection for each article (or, perhaps, each  hand-
              ful  of articles arriving together).  The default is to close if
              fewer than 3 articles in the last 1000s.

              We tolerate up to this proportion of  badly-formatted  lines  in
              the  feedfile and other input files.  Every badly-formatted line
              is logged, but if there are too many we conclude that  the  cor-
              ruption  to  our  on-disk data is too severe, and crash; to suc-
              cessfully restart, administrator intervention will be  required.
              This avoids flooding the logs with warnings and also arranges to
              abort earlyish if an attempt is made to process a  file  in  the
              wrong  format.  We need to tolerate a small proportion of broken
              lines, if for no other reason than that a crash  might  leave  a
              half-blanked-out entry.  The default is 1 (ie, 1%).

              Additionally,  we  tolerate this number of additional badly-for-
              matted lines, so that if the badly-formatted lines are a few but
              at  the  start  of  the  file,  we don't crash immediately.  The
              default is 30 (which would suffice to ignore one  whole  corrupt
              4096-byte  disk  block  filled  with random data, or one corrupt
              1024-byte disk block filled with an inappropriate text file with
              a mean line length of at least 35).

              does  roughly  the  same  thing as innduct.  However, the way it
              receives information from innd can result in articles being lost
              (not  offered to peers) if innfeed crashes for any reason.  This
              is an inherent defect in the innd channel feed  protocol.   inn-
              duct  uses  a file feed, constantly "tailing" the feed file, and
              where implemented uses inotify(2) to reduce  the  latency  which
              would  come  from having to constantly poll the feed file.  inn-
              duct is  much  smaller  and  simpler,  at  <5kloc  to  innfeed's
              ~25kloc.   innfeed  needs  a  separate  helper script or similar
              infrastructure (of which there is an example  in  its  manpage),
              whereas  innduct  can be run directly and doesn't need help from
              shell scripts.  However, innfeed is capable of feeding  multiple
              peers  from a single innfeed instance, whereas each innduct pro-
              cess handles exactly one peer.

              processes feed files in batch mode.  That is, you have to  peri-
              odically  invoke  nntpsend, and when you do, the feed is flushed
              and articles which arrived before the  flush  are  sent  to  the
              peer.  This introduces a batching delay, and also means that the
              NNTP connection to the peer needs to be remade  at  each  batch.
              nntpsend  (which  uses innxmit) cannot make use of multiple con-
              nections to a single peer site.  However, nntpsend automatically
              find  which  sites  need  feeding  by  looking  in pathoutgoing,
              whereas the administrator needs to  arrange  to  invoke  innduct
              separately for each peer.

              is the actual NNTP feeder program used by nntpsend.

                                   innfeed   innduct   nntpsend/innxmit
       realtime feed               Yes       Yes       No
       reliable                    No        Yes       Yes
       source code size            24kloc    4.6kloc   1.9kloc
       invoke once for all sites   Yes       No        Yes
       number of processes         one       1/site    2/site, intermittently

       0      An  instance of innduct is already running for this feedfile and
              -q was specified.

       4      The feed has been dropped by innd, and we (or previous innducts)
              have successfully offered all the old articles to the peer site.
              Our work is done.

       8      innduct was invoked with bad options or command line  arguments.
              The  error  message  will be printed to stderr, and also (if any
              options or arguments were passed at all) to syslog with severity

       12     Things  are  going  wrong,  hopefully shortage of memory, system
              file table entries; disk  IO  problems;  disk  full;  etc.   The
              specifics  of  the  error will be logged to syslog with severity
              err (if syslog is working!)

       16     Things are going badly wrong in an unexpected way: system  calls
              which are not expected to fail are doing so, or the protocol for
              communicating  with  innd  is  being  violated,  or  some  such.
              Details  will  be  logged with severity crit (if syslog is work-

       24-27  These exit statuses are used by children forked  by  innduct  to
              communicate to the parent.  You should not see them.  If you do,
              it is a bug.

       innduct dances a somewhat complicated dance with innd to make sure that
       everything  goes  smoothly  and  that there are no races.  (See the two
       ascii-art diagrams in innduct.c for details of the protocol.)   Do  not
       mess  with  the  feedfile  and  other  associated  files, other than as
       explained here:

              Default current working directory for innduct, and also  default
              grandparent directory for the command line socket.

              Default feedfile.

              Main  feed  file  as  specified in newsfeeds(5).  This and other
              batchfiles used by innduct contains lines each of  which  is  of
              the  form     token messageid where token is the inn storage API
              token.  Such lines can be written by Tf,Wnm  in  a  newsfeeds(5)
              entry.  During processing, innduct overwrites lines in the batch
              files which correspond to articles it has processed:  each  such
              line  is  replaced  with  one containing only spaces.  Only innd
              should create feedfile, and only innduct should remove it.

              Lockfile, preventing multiple innduct invocations for  the  same
              feed.   A  process holds this lock after it has opened the lock-
              file, made an fcntl F_SETLK call, and then checked with stat and
              fstat that the file it now has open and has locked still has the
              name feedfile_lock.  (Only) the lockholder may delete the  lock-
              file.   For your convenience, after the lockfile is locked, inn-
              feed's pid, the site, feedfile and fqdn are all written  to  the
              lockfile.   NB  that  stale  lockfiles may contain stale data so
              this information should not be relied on other  than  for  trou-

              Batch  file:  the  main  feedfile is renamed to this filename by
              innduct before it asks inn to  flush  the  feed.   Only  innduct
              should create, modify or remove this file.

              Batch file containing details of articles whose transmission has
              very recently been deferred at  the  request  of  the  recipient
              site.  Created, written, read and removed (only) by innduct.

              Batch file containing details of articles whose transmission has
              less recently been deferred at  the  request  of  the  recipient
              site.   Created  by  innduct, and will also be read, updated and
              removed by innduct.  However you (the  administrator)  may  also
              safely remove backlog files.

              Batch file manually provided by the administrator.  innduct will
              automatically find, read and process any file matching this pat-
              tern  (blanking  out entries for processed articles) and eventu-
              ally remove it.  something may not contain # ~ or /.

              Be sure to have finished writing the file before you  rename  it
              to match the pattern feedfile_backlog*, as otherwise innduct may
              find and process the file, and even think it  has  finished  it,
              before  you have written the complete file.  You may also safely
              remove backlog files.

              Default AF_UNIX listening socket for the control  command  line.
              See CONTROLLING INNDUCT, above.

              On  request  via a control connection innduct dumps a summary of
              its state to this text file.  This is mostly useful  for  debug-

              Used  for  pathoutgoing (to compute default feedfile and associ-
              ated paths), pathrun  (to  compute  default  PATHRUN  and  hence
              effective default CLI-DIR and CLI-PATH), for finding how to com-
              municate with innd, and also for sourceaddress and/or  sourcead-

       Written by Ian Jackson <ijackson at>

       inn.conf(5), innd(8), newsfeeds(5)


More information about the inn-workers mailing list