Request for Comments: load balancing my CNFS buff files....

Russ Allbery rra at stanford.edu
Fri Mar 31 11:16:42 UTC 2000


John Kozubik <john_kozubik at hotmail.com> writes:

> I have 15 disks for my spool area, and I am using cnfs.

> da1 - da7 are on one SCSI host adaptor, and da8-da15 are on a different
> SCSI host adaptor.

> I created only two meta cycbuffs - one called 'BINARIES' and one called
> 'ALLDREST' - I am seeking any and all comments on something I did in
> creating these two meta cycbuffs.

> Basically, it occurred to me that my BINARIES meta cycbuff might have a
> very different disk/host adapter utilization pattern than my ALLDREST
> meta cycbuff.  So what I did, is instead of giving all the disks on SCSI
> host adaptor #1 (7 disks in all) to BINARIES and all the disks on SCSI
> host adaptor #2 (8 disks in all) to ALLDREST, is I alternated one disk
> from one host adaptor and one disk from the other host adaptor, until
> all were used up.

[...]

> metacycbuff:ALLDREST:1,8,2,9,3,10,4,11
> metacycbuff:BINARIES:5,12,6,13,7,14,15

> Note that the numbers are NOT consecutive - the buffer named '1' is the
> first disk on the first host adaptor, but the buffer named '8' is the
> first disk on the second host adaptor.

> What I hoped to gain:

> Basically, I have no idea what the utilization pattern of the BINARIES
> meta cycbuff vs. the ALLDREST meta cycbuff is - what I _do_ know is that
> they are _different_ and therefore one of them must utilize a SCSI host
> adaptor differently than the other, and therefore, although I am
> degrading performance for the meta cycbuff that utilizes its controller
> the least, over the aggregate of _BOTH_ meta cycbuff files, I am
> decreasing SCSI host adaptor congestion, and thus receiving higher
> overall disk performance.

> Any comments?

My guess is that this won't make much difference, primarily because INN is
single-threaded and for the single-threaded case, CNFS is designed to
significantly reduce the differences in performance between large and
small article writes.

Here's my analysis.

INN uses cycbuffs by writing data to the end of the current cycbuff and
then moving to the next cycbuff in a metacycbuff for the next article that
falls into it (in normal mode).  The primary difference between large
articles and small articles is that in the former case, INN writes more
data before moving on to the next article, but it's only ever writing out
one article at a time so writing a bunch of data to one disk or the same
amount of data split across two disks doesn't make all that much
difference (provided that the size of the article is smaller than the
writeback cache on the disk, if any).  Small articles are a bit more
seeking per article write to update the bitmap, and large articles are a
larger chunk of data.

innfeed or nnrpd, when reading the articles back, will do a lot more
seeking on the small article metacycbuffs, but it will tie up the large
article metacycbuffs with longer data transfers and multiple innfeeds will
be working at the same time.  Feels like six of one, half-dozen of another
to me.

I personally for transit servers just dump everything into one huge
metacycbuff if I have a lot of spindles, since it maximizes the chance
that two innfeeds feeding different articles will be hitting different
spindles.  If there are multiple metacycbuffs and the number of articles
going into one is significantly lower than the number going into another,
it may reduce the effective spread across the spindles.

But overall, I don't see much real room for optimization here; CNFS's disk
access pattern is already pretty efficient.  Using sequential rather than
round-robin appears to help when you have only a few spindles, but in my
experience the real performance gain comes from just having a huge
write-back memory cache for your disks.

Multithreading INN's disk writes would also help a lot, but that's a
pretty hard problem.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>



More information about the inn-workers mailing list