BerkeleyDB slower than tradindexed?

Heath Kehoe heath.kehoe at intermec.com
Mon Jul 31 16:09:55 UTC 2000


>
>Was messing around with my test box on Thursday, and decided to drop
>waaaay back to BerkeleyDB 2.7.7 (having already tried 3.x+ in the past
>with abysmal results).
>
>Now normally, I get around 126GB/day, with most of that coming from
>Cidera, and BBN filling in the blanks.
>
>I normally run with tradindexed for overview, on a 4 disk RAID 0 array.
>
>So I wipe out the traditional index, compile with  Berkeley DB 2.7.7, 
>bump the # of files to 128 in inn.conf (can't remember the setting, but
>it's the one for the # of database files), and turn it loose...
>


You set numdbfiles to 128?  Hmm... I've never tried it with that many.
I wonder if having that many DBs open at once is causing problems for
the DB library.


>About 6 hours later, I get a call from Skycache/Cidera, telling me I'm
>dropping articles (about 1/4th to 1/3rd), and lo and behold, I only
>processed about 95GB of news during the day...
>
>Hmph...  Tried another day.  Same thing.  Hmph.  Switched back to
>traditional.  Much better.
>
>Here's the part that concerns me:
>
>here's a log entry from the timer stats with BerkeleyDB:
>
>overview write   08:14:15.967  34.9%     577645     0.000     51.339
>36534.000
>
>36 seconds to write overview?!?!?!?!?
>
>And a 51ms average?  Ouch.
>
>
>What kind of numbers are other people seeing?  Outside of these numbers,
>everything looks fine, there's lots of idletime, the box has plenty of
>oomph left, low load average, CNFS buffers, and only 1 outbound feed, and
>1 or 2 readers (usually me, testing).
>


I've never seen it that bad... My server does about 450000 overview writes
per day, and averages 10ms per write.

Did you happen to look at how busy the disks were while the server was
running on ovdb?  Was it thrashing the overview disk?  If not, I can
only guess that BerkeleyDB was getting deadlocked for some reason.
But that doesn't make sense, because normally innd is the only process
that writes to the database (except when expireover runs).

Perhaps your cachesize was set too low.  If you left it at the default,
it was *definitely* too low.  And with your numdbfiles set that high,
you'll need even more cache.

Should you decide to try ovdb again, I recommend using a lower setting
for numdbfiles (say, 64) and a larger cachesize (at least 20000).

-heath




More information about the inn-workers mailing list