no largefile support for Solaris character devices!

[James Ralston]

I wrote:

> Even though passing --with-largefiles to INN 2.2.2's configure
> causes storage/cnfs/cnfs.c to be compiled with the right options to
> enable largefile support on Solaris 2.6 (that is,
> -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64), my 8GB CNFS buffer
> failed with seek errors when it hit 2GB.

I figured out my own answer: yes, this is a Solaris limitation.

Under Solaris 2.6 (and possibly other versions of Solaris), you cannot
mmap() a disk block device when the "offset" argument to mmap() is
greater than 2GB.

# /tmp/mmap-test /dev/dsk/c1t0d0s0
mmap-test: debug: _LARGEFILE_SOURCE is defined
mmap-test: debug: _FILE_OFFSET_BITS = 64
mmap-test: debug: sizeof(int) = 4
mmap-test: debug: sizeof(off_t) = 8
mmap-test: debug: sizeof(long) = 4
mmap-test: debug: sizeof(off64_t) = 8
mmap-test: debug: sizeof(long long) = 8
mmap-test: debug: sizeof(offset) = 8
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 0
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 268435456
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 536870912
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 805306368
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 1073741824
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 1342177280
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 1610612736
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 1879048192
mmap-test: attempting to mmap 1 byte of /dev/dsk/c1t0d0s0 at offset 2147483648
mmap-test: mmap failed: Invalid argument

But:

# newfs -v /dev/dsk/c1t0d0s0
# mount /dev/dsk/c1t0d0s0 /mnt
# dd if=/dev/zero of=/mnt/testfile bs=1024k count=4096
# mmap-test /mnt/testfile
mmap-test: debug: _LARGEFILE_SOURCE is defined
mmap-test: debug: _FILE_OFFSET_BITS = 64
mmap-test: debug: sizeof(int) = 4
mmap-test: debug: sizeof(off_t) = 8
mmap-test: debug: sizeof(long) = 4
mmap-test: debug: sizeof(off64_t) = 8
mmap-test: debug: sizeof(long long) = 8
mmap-test: debug: sizeof(offset) = 8
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 0
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 268435456
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 536870912
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 805306368
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 1073741824
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 1342177280
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 1610612736
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 1879048192
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 2147483648
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 2415919104
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 2684354560
mmap-test: attempting to mmap 1 byte of /mnt/testfile at offset 2952790016
mmap-test: mmap testing complete

I'll bug Sun about this (we have a support contract), but I suspect
they don't actually have a patch to work around the limitation.

As Paul Theodoropoulos observed:

> If you have a small number of disks on a feed-exchanger, I'd
> recommend trying out the 'SEQUENTIAL' mode modifier in cycbuff.conf.
> the 'waiting on I/O' statistic on my server dropped from an average
> of 16% down to 8% by making that change.  I believe it's because the
> disk no longer has to seek constantly across the platters to write
> articles as it would in the default round-robin mode.

So, this means people using Solaris 2.6 systems who want to do this
(like me) are stuck: either they have to use a single large file in a
filesystem, and incur filesystem overhead, or else use a series of
round-robin CNFS buffers, and incur seek time overhead.

The SEQUENTIAL mode modifier would be a good work-around, but the
server I'm setting up will be our new main feed-exchanger server, and
I don't want it to be on the cutting edge of the code base...

James

-- 
James Ralston, Information Technology
Software Engineering Institute
Carnegie Mellon University, Pittsburgh, PA, USA