I hate makedbz.

[Ron Jarrell]

At 09:26 AM 7/27/01 +0100, Alex Kiernan wrote:

>Ron Jarrell <jarrell at solaris.cc.vt.edu> writes:
>
>> Ok, so this is the first time I've had to regenerate a history file
>> since 2.2.  What is *with* makedbz?  I've got an 853 meg history
>> file after makehistory ran.  The old file had a bit over 16m entries
>> in it, so a did a makedbz -s 20000000 -f history.n (leaving room for
>> the fact that the server's been down now since early wed.  morning,
>> so I'm going to have to deal with a huge surge of backlog).
>> 
>> Doing the makehistory, re-reading the entire spool, and building the
>> ovdb files took 6 hours.  In 20 more minutes the makedbz, which just
>> has to build the offset index into the history.n file, is going to
>> hit 24 hours, and still going strong.  The io rates are very
>> reasonable, there's about 3.5 gig of memory free on the system, and
>> the other two processors are twiddling their thumbs.
>> 
>
>What size are the history.n.hash and history.n.index - if they're
>anything other than 0 you've spilled to disk and you should just kill
>the makedbz & start again with a bigger -s.

Oh, yea, they're non zero.  They were non zero in the first
10 minutes.

It finished last night at about 2am, 10 minutes shy of 25 hours.

Currently; 
solaris{root}121% ls -l history*
-rw-r--r--   1 news     news     919985441 Jul 27 15:04 history
-rw-r--r--   1 news     news           48 Jul 27 15:04 history.dir
-rw-r--r--   1 news     news     80999988 Jul 27 15:04 history.hash
-rw-r--r--   1 news     news     107999984 Jul 27 15:04 history.index
-rw-rw-r--   1 news     news     497852416 Jul 27 12:50 history.n
-rw-rw-r--   1 news     news           48 Jul 27 03:02 history.n.dir
-rw-rw-r--   1 news     news     35999022 Jul 27 12:50 history.n.hash
-rw-rw-r--   1 news     news     47998696 Jul 27 12:50 history.n.index
solaris{root}122% cat history.dir
dbz 6 750000 14 66
12791549 0 0 0 0 0 0 0 0 0 0
solaris{root}123% cat history.n.dir
dbz 6 750000 14 66
0 12159388 0 0 0 0 0 0 0 0 0
solaris{root}124% wc -l history
 12793609 history



Expire is currently running, and has been since 3am.  It typically runs
for about 48-60 hours, and now I know why; 24 of that has been the
raw generation.

Now I did the makedbz, as I said, with a -s 20000000.  Given that the 
history file is only 12793609, shouldn't that have been *plenty*?

Got a guess as to how big I have to make it to get performance like you are?


>Here's a box I'm running here:
>
>bash-2.03$ ls -l history
>-rw-rw-r--   1 news     news     785561421 Jul 27 08:23 history
>bash-2.03$ time ~news/bin/makedbz -s 20000000 
>
>real    1m19.316s
>user    0m50.840s
>sys     0m15.980s
>
>Its not quite as big a history as yours, but thats the kind of run
>time you should be looking at.
>
>-- 
>Alex Kiernan, Principal Engineer, Development, Thus PLC