tinyfeed miniature transit server

[Russ Allbery]

Christoph Biedl <Cbiedl at gmx.de> writes:

> Using a number will cause trouble if there's one than one instance of
> tinyfeed running on the same spool; or do you take precautings against a
> race, say, locking the file?

It doesn't take any precautions against that sort of thing; it's very
simple and stupid.

> Furthermore, if "unsigned long" is 32 bit and I transmit a !binary full
> feed via tinyproxy with some 400k articles every day, I will run into an
> overflow in a quite forseeable time.

Point.

> May I therefore suggest to use the md5 hash of the Message-ID as file
> name? This will also allow to reply with a "Duplicate" if the file is
> already present; or even "Try again later" under certain circumstances
> (file present but locked/size 0/not ending with "\n.\n").

Sure, that's a good idea.  The additional work (duplicate, etc.) I don't
really have the time to do just at the moment (although patches are
certainly welcome!), but switching the file name generation over to using
an MD5 checksum is trivial.  (One of the nice things about using libinn,
that.)

> I understand that this might lead to files remaining unprocessed after a
> crash?  Perhaps tinyfeed should check for files present in the spool at
> startup and re-offer them to the processor.

I had some vague notion originally that you could point tinyfeed at a
directory that contained files other than articles and it would just cope,
but perhaps that isn't actually that useful of a concept.  You're right
that such a restart process would be very useful.

>> EXAMPLE
>>     Suppose that you want to archive news articles on a particular host
>>     (like the FTP server for a newsgroup archive) where you don't want
>>     the overhead of running a full-blown news server.  Write a program
>>     that reads one line at a time from standard input and treats
>>     everything before the first space as the filename of a news article
>>     to archive. Each time the program reads a line, it should archive
>>     that file and then delete it, and it should exit when it gets end
>>     of file on standard input.

> Hm, that's quite close to archive which will also create an according
> directory structure - but on the same host only and without a "you've
> got new articles".

Yeah, tinyfeed is basically intended to provide the world's smallest
channel feed.  It provides Tc,Wnm format, except the n is actually a file
name rather than a storage API token.  That means that many of INN's
existing feed backends can be used with it if you want, except that they
don't delete files after they've been processed.

I have it feeding into a couple of somewhat more complex scripts right
now; one of them checks the PGPMoose signature of news.announce.newgroups
messages and then archives them if it's valid, and the other one processes
incoming control messages to maintain the ftp.isc.org active and
newsgroups files and the control message archive.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.