CNFS and FreeBSD

[Craig Norborg]

         After struggling with getting CNFS up and running on a FreeBSD 
box, it all came down to a fairly easy thing that I thought you might want 
to at least elude to in your "INSTALL" file.

         The paragraph that states:
"For each metacycbuff, you now need to determine how many cycbuffs will 
make up the metacycbuff, the size of those cycbuffs and where they will be 
stored. Some OSs do not support files larger than 2 GB which will force all 
of your cycbuffs to be < 2GB (even if they are stored on raw disk 
partitions). Linux is known to have this limitation, FreeBSD does not. If 
in  doubt, keep your cycbuffs smaller than 2 GB. Also when laying out 
your  cycbuffs you will want to try to arrange them across as many physical 
disks as possible (or use a striped disk array and put them all on that)."

while it is true that FreeBSD does not have the 2Gb partition, you will 
continually get the error messages in your /var/log/messages:

innd: CNFS-sm: CNFSinitdisks: mmap for /var/news/spool/buffers/buf06 offset 
0 len 500006912 failed: Cannot allocate memory
innd: SM storage method 'cnfs' failed initialization
innd: SM one or more storage methods failed initialization
innd: SERVER cant initialize the storage subsystem one or more storage 
methods failed initialization

         After trying different things, I finally traced the message down 
in the code, and its a problem in a "mmap" call.  In the FreeBSD man pages 
under mmap:
"BUGS
      len is limited to 2GB.  Mmapping slightly more than 2GB doesn't work, 
but
      it is possible to map a window of size (filesize % 2GB) for file 
sizes of
      slightly less than 2G, 4GB, 6GB and 8GB.

      The limit is imposed for a variety of reasons. Most of them have to do
      with FreeBSD not wanting to use 64 bit offsets in the VM system due to
      the extreme performance penalty. So FreeBSD uses 32bit page indexes and
      this gives FreeBSD a maximum of 8TB filesizes. It's actually bugs in the
      filesystem code that causes the limit to be further restricted to 1TB
      (loss of precision when doing blockno calculations).

      Another reason for the 2GB limit is that filesystem metadata can reside
      at negative offsets."



         So, even though FreeBSD doesn't have the filesize limitation, it 
does have a problem with mmap'ing a file over 2Gb in size.  Rather than 
mess around with "mapping a window" or trying to convert the kernel to 
64bit offsets in the VM system, I decided that recreating the buffers as 
<2Gb files would probably be easier...  Everything worked like a champ 
right away...
         Now to see how this puppy works...  Not worrying about expiring 
articles will be a strange thing....
Craig Norborg
craig at norb.org