makedbz memory footprint

[Richard Todd]

In message <01JMAAY7LXQK8WWFZ0 at OREGON.UOREGON.EDU>, Joe St Sauver writes:
>During a recent makedbz, I noticed that makedbz can grow to a 
>rather astronomical size for large history files -- e.g.,
>we were seeing:
>
>  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
>20009 root       1  60    0  703M  537M sleep  55:58  0.53% makedbz
>
>for a history file that was 2147483647 in size, and arguments
>of makedbz -s35000000 -o -f /news/history
>
>[Yes, I know that 35000000 lines of history is quite a bit, but
>if you figure 600K articles/day and spools that are running
>100GB on a routine basis these days, well, if you don't carry a 
>lot of warez and mp3's, it is amazing what you can do.]
>
>Have I accidentally triggered some undetected limit that's making
>the makedbz thrash, or is that just the sort of memory footprint
>I should expect?

Are you using tagged hash or not?  I'll work out both ways what the expected
memory usage for a dbz rebuild should be with each one, though as I personally
only run the tagged-hash case, I can't guarantee that I've not screwed up
the calculation for non-tagged-hash.

Basically, for either tagged hash or non-tagged hash, the primary thing that
uses up memory is the mmap()ed index file(s).  How big that file is is 
determined by the size parameter as follows.

Case 1: Tagged Hash

The size you gave was 35 million entries.  dbz attempts to keep the hash table
2/3rds full, so for 35 million entries to be in there, the hash table total
size needs to be 3/2*35*10^6 or 52,500,000 slots in the table.  Each slot
holds a 4-byte file offset, so the total table size is 4*52,500,000
= 210,000,000 bytes.  

  [Digression: the entries in the hash table are always 4 bytes, even if
  you are on an OS where off_t is 8 bytes.  This is intentional. I know,
  I'm the guy who made the code this way.  The theory is that people on
  tagged-hash systems are tight on memory and probably don't have
  history files bigger than 4G anyway, so don't need to waste 2x the
  "usual" amount of memory.  This is why even if you give the
  --largefiles (or whatever it is) config option, tagged-hash dbz won't
  like history files above 4G (or possibly even 2G), which I think someone
  was asking about here recently.  If you really really need tagged hash *and*
  large history files, you might try changing the #define of of_t in lib/dbz.c.
  This is highly untested, though, so if there are still problems with big
  history files, don't say you weren't warned. :-) ]

Anyway, back to the analysis.  Since you're using more than 210M, I'm
assuming you're doing non tagged hash dbz.  

Case 2: Non Tagged Hash.

Both the .index and .hash files are mmaped() by makedbz.  Both are hash
tables, similar to the one in the tagged-hash case.  The hash tables 
are kept 50% full, so each one has 70 million slots.  Each slot is 
sizeof(off_t) bytes in size.  How much RAM makedbz will want depends on
how big your off_t is.

   Case 2a: sizeof(off_t) = 4.
	Then each of the hash tables is 4*70 million = 280,000,000 bytes, for
        a total usage of 560,000,000 bytes (yow!)

   Case 2b: sizeof(off_t) = 8.
	Double all the previous figures; the total usage is 1,120,000,000
	bytes. (Yow!)

The 560,000,000 works out to be 534M, so the 537M RSS figure seems
reasonable.  The 703M total memory size doesn't seem to fit too well
with that, though.  Unfortunately, it does fit with the possibility
that you have ~534M of dbz index and about another 160M of RAM being
leaked somewhere in the makedbz process, which is rather worrisome.  I
certainly haven't seen anything like that in my experience.  Any suggestions
from anyone as to where that 160M discrepancy is coming from?