mail to news and vice versa

[Russ Allbery]

Matthew Harrell <mharrell at bittwiddlers.com> writes:

> Well I was trying to make it transparent so you could either respond to
> the news article or to respond to the email and appropriate action would
> be taken.

Bill's approach should do that.  I do bidirectional gatewaying here at
Stanford, quite a bit of it, and it works fairly well, but you have to
think it through pretty thoroughly before doing it.

Here's some stuff I wrote for USEFOR a while back that may be useful.  I
think I wrote something else more detailed about what we do here at some
point, but I can't find it quickly.

8.7.  Duties of a Gateway

   A Gateway transforms an article into the native message format of
   another medium, or translates the messages of another medium into news
   articles.  Encapsulation of a news article into a message of MIME type
   application/news-transmission, or the subsequent undoing of that
   encapsulation, is not gatewaying, since it involves no transformation
   of the article.

   There are two basic types of Gateway, the Outgoing Gateway that
   transforms a news article into a different type of message, and the
   Incoming Gateway that transforms a message from another medium into a
   news article and injects it into a netnews system.  These are handled
   separately below.

   The primary dictate for a gateway is:

        Above all, prevent loops.

   Transformation of an article into another medium stands a very high
   chance of discarding or interfering with the protection inherent in the
   news system against duplicate articles.  The most common problem caused
   by gateways is "spews," gateway loops that cause previously posted
   articles to be reinjected repeatedly into Usenet.  To prevent this, a
   gateway MUST take precautions against loops, as detailed below.

   If bidirectional gatewaying (both an incoming and an outgoing gateway)
   is being set up between netnews and some other medium, the incoming and
   outgoing gateways SHOULD be coordinated to avoid reinjection of gated
   articles.  Circular gatewaying (gatewaying a message into another
   medium and then back into netnews) SHOULD NOT be done; encapsulation of
   the article SHOULD be used instead where this is necessary.

   A second general principal of gatewaying is that the transformations
   applied to the message SHOULD be as minimal as possible while still
   accomplishing the gatewaying.  Every change made by a gateway
   potentially breaks a property of one of the media or loses information,
   and therefore only those transformations made necessary by the
   differences between the media should be applied.

8.7.1.  Duties of an Outgoing Gateway

   From the perspective of netnews, an Outgoing Gateway is just a special
   type of news reader.  The exact nature of what the outgoing gateway
   will need to do to articles depends on the medium to which the articles
   are being gated.  The operation of the outgoing gateway is only subject
   to additional constraints in the presence of one or more corresponding
   incoming gateways back from that medium to netnews, since this opens
   the possibility of loops.

   It is recommended, however, that the following practices be followed by
   all outgoing gateways regardless of whether there is known to be a
   related incoming gateway, both as a precautionary measure and as
   quality of implementation guidelines.

   1. Only the minimal necessary changes should be made, as stated above.

   2. The Message-ID of the news article should be preserved if at all
      possible, preferably as or within the corresponding unique
      identifier of the other medium, but if not at least as a comment in
      the message.  This helps greatly with preventing loops.

   3. The Date of the news article should also be preserved if possible,
      for similar reasons.

   4. The message should be tagged in some way so as to prevent it's
      reinjection into netnews.  This may be impossible to do without
      knowledge of potential incoming gateways, but it's better to try to
      provide some indication even if not successful; at the least, a
      human-readable indication that the article should not be gated back
      to netnews can help a human locate problems.

   5. News control messages should not be gated to other media unless they
      would somehow be meaningful in that medium.

8.7.2.  Duties of an Incoming Gateway

   The incoming gateway has the serious responsibility of ensuring that
   all of the requirements of this standard are met by the articles that
   it forms.  In addition to its special duties as a gateway, it bears all
   of the duties and responsibilities of an injector as well, and
   additionally has the same responsibility of a relaying agent to reject
   articles that it has already gatewayed.

   An incoming gateway MUST NOT gate the same message twice.  It may not
   be possible to ensure this in the face of mangling or modification of
   the message, but at the very least a gateway, when given a copy of a
   message it has already gated identical except for trace headers (like
   Received in e-mail or Path in netnews) MUST NOT gate the message again.
   An incoming gateway SHOULD take precautions against having this rule
   bypassed violated by modifications of the message that can be
   anticipated.

   News articles prepared by gateways MUST be legal news articles.  In
   particular, they MUST include all of the mandatory headers and MUST
   fully conform to the restrictions on said headers.  This often requires
   that a gateway function not only as a relayer, but also partly as a
   posting agent, aiding in the synthesis of a conforming article from
   non-conforming input.

   Incoming gateways MUST NOT pass control messages (articles containing a
   Control header) without removing or renaming that header.  Gateways
   MAY, however, generate their own cancel messages, under the general
   allowance for injectors cancelling their own messages.  If a gateway
   receives a message that it can determine is a valid equivalent of a
   cancel message in the medium it is gatewaying, it SHOULD discard that
   message without gatewaying it, generate a corresponding cancel message,
   and inject that cancel message.

   Incoming gateways MUST NOT inject control messages other than cancels.
   Encapsulation SHOULD be used instead of gatewaying, when direct posting
   is not possible or desirable.

        NOTE: It is not unheard of for mail-to-news gateways to be used to
        post control messages, but encapsulation should be used for these
        cases instead.  Gateways by their very nature are particularly
        prone to loops.  Spews of normal articles are bad enough; spews of
        control messages with special significance to the news system,
        possibly resulting high processing load or even e-mail sent for
        every message received, are catastrophic.  It is far preferable
        to construct a system specifically for posting control messages
        that can do appropriate consistency checks and authentication of
        the originator of the message.

   If there a message identifier that fills a role similar to that of
   Message-ID in news, it SHOULD be used in the formation of the
   Message-ID of the news article, perhaps with transformations required
   to meet the uniqueness requirement of netnews.  This transformation
   SHOULD be designed so that two messages with the same identifier
   receive the same Message-ID.

        NOTE: Message-ID serves a central role in the prevention of
        duplicates, and its correct use by gateways will do much to
        prevent loops.  Netnews does, however, require that Message-ID be
        unique, and therefore message identifiers from other media may not
        be suitable for use without modification.  A balance must be
        struck by the gateway between preserving information used to
        prevent loops and generating unique Message-IDs.

   Exceptionally, if there are multiple incoming gateways for a particular
   set of messages, each incoming gateway SHOULD generate a Message-ID
   unique to that gateway.  Each incoming gateway nonetheless MUST ensure
   that it does not gate the same message twice.

        NOTE: Consider the example of two gateways of a given mailing list
        into world-wide Usenet newsgroups, both of which preserve the mail
        Message-ID.  Each newsgroup may receive only 50% of the messages.
        In these cases, where there is no one "official" gateway, some
        other method of generating Message-IDs has to be used to avoid
        collisions.  It would obviously be preferable for there to be
        only one gateway which crossposts, but this may not be possible to
        coordinate.

   If no date information is available, the gateway MAY supply a Date
   header with the gateway's current date.  If only partial information is
   available (e.g. date but not time), this SHOULD be fleshed out to a
   full Date header by adding default values rather than discarding this
   information.  Only in very exceptional circumstances should Date
   information be discarded, as it plays an important role in preventing
   reinjection of old messages.

   An incoming gateway MUST add a Sender header to the news article it
   forms containing the mailbox of the administrator of the gateway.
   Problems with the gateway may be reported to this address.  The display
   name portion of this mailbox SHOULD indicate that the entity
   responsible for injection of the message is a gateway.  If the original
   message already had a Sender header, it SHOULD be renamed so that its
   contents can be preserved.

8.7.3.  Moderators as Incoming Gateways

   This standard requires that submissions to moderators be sent in the
   form of encapsulated articles, but this is a change from existing
   practice before this standard.  Previously, news systems transformed
   proto-articles into mail messages and sent them to the moderation
   submission address.  Moderators SHOULD therefore be prepared to serve
   as gateways until news systems have been made to conform with this
   standard.  A message requiring gatewaying can be identified by the lack
   of an application/news-transmission Content-Type.

   Some news systems did encapsulate news articles, but did so without
   MIME labeling.  Moderators may therefore want to treat messages where
   the body of the message starts with valid news headers, including a
   Newsgroups header containing the name of the moderated group, as
   encapsulated news messages, particularly if the main mail headers are
   lacking required news headers such as Subject.

   It is worth noting that safe bidirectional gatewaying between a mailing
   list and newsgroup is far easier if the newsgroup is moderated.  Posts
   to the moderated group and submissions to the mailing list can then go
   through a single point that does the necessary gatewaying and then
   sends the message out to both the newsgroup and the mailing list at the
   same time, eliminating most of the possibility of loops.  Bidirectional
   gatewaying between a mailing list and an unmoderated newsgroup, in
   contrast, is difficult to do correctly and is far more fragile.

   Newsgroups intended to be bidirectionally gated to a mailing list
   SHOULD therefore be moderated where possible, even if the moderator is
   a simple gateway and injector that correctly handles crossposting to
   other moderated groups and otherwise passes all traffic.

8.7.4.  Example

   To illustrate the type of precautions that should be taken against
   loops, here is an example of the measures taken by one particular
   combination of mail-to-news and news-to-mail gateways at Stanford
   University designed to handle bidirectional gatewaying between mailing
   lists and unmoderated groups.

   1. The outgoing gateway preserves the Message-ID of the news article in
      generated mail message.  The incoming gateway likewise preserves the
      mail Message-ID provided that it is syntactically valid for netnews.
      This allows the news system's built-in suppression of duplicates to
      serves as the first line of defense against loops.

   2. The outgoing gateway adds an X-Gateway header to all messages it
      generates.  The incoming gateway discards any incoming messages
      containing this header.  This is robust against mailing list
      managers that replace the message ID, and against any number of mail
      hops, provided that the other message headers are preserved.

   3. The incoming gateway adds the host name from which it received the
      mail message to the Path tail.  The outgoing gateway refuses to
      gateway any message that contains the list server name in the Path
      tail.  This is robust against any amount of munging of the message
      headers by the mailing list, provided that the mail only goes
      through one hop.

   4. The incoming gateway is designed to never generate bounces to the
      envelope sender.  Instead, articles that are rejected by the news
      server (for reasons not warranting silent discarding of the message)
      result in a bounce message sent to an errors address known not to
      forward to any mailing lists, so that they can be handled by the
      news administrators.

   These precautions have proven effective in practice at preventing loops
   for this particular application (bidirectional gatewaying between
   mailing lists and locally distributed newsgroups where both gateways
   can be designed together).  General gatewaying to world-wide newsgroups
   poses additional difficulties; one must be very wary of strange
   configurations, such as a newsgroup gated to a mailing list which is in
   turn gated to a different newsgroup.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>