Comparing query rates (was Re: 9.2.5 db causes high cpu?)

[Jim Reid]

>>>>> "Brad" == Brad Knowles <brad at stop.mail-abuse.org> writes:

    Brad> 	The tests were specifically designed to make the
    Brad> servers thrash on .tv, and none of the servers should have
    Brad> thrashed on the root zone.  I specifically chose those two
    Brad> particular zones for that reason.  I wanted to show the
    Brad> relative performance of the software in question when the
    Brad> zone could be guaranteed to completely fit into memory, and
    Brad> when the zone could be guaranteed to greatly exceed the
    Brad> memory available by at least an order of magnitude.  Both
    Brad> extremes are useful when trying to compare the performance
    Brad> of different software.

I fail to understand your logic. If the name server thrashes, it's
self-evident that perfomance will be at least an order of magnitude
worse because of disk I/O. What next, will you conduct a test to prove
what bears do in the woods?

Furthermore, no sane person would run a production name server in an
environment which has been deliberately set up to make the server
thrash. This is like taking the wheels off a car and then measuring
its 0-60 time: what's the point?

    Brad> 	But until you or someone else can come up with
    Brad> suitable comparative numbers, I absolutely will not stop
    Brad> referring to my own test results, and no amount of badgering
    Brad> from you or anyone else is going to make me stop.

I am not badgering you at all. I'm just asking that you try to avoid
misleading people by quoting numbers out of context. At the very least
you should describe your hardware platform, test methodology and
explain that your results come with a whole lot of qualifiers that are
unrepresentative of the typical DNS installation: a 133MHz Pentium
laptop with 40 MB of RAM. Just saying name server X does Y qps helps
no-one. Especially if X is operating in an environment that's nothing
like the ones people use for their production name servers.

    >> FYI, I just did a crude test on an elderly 300 Mhz Pentium II
    >> running BIND 9.3. It handles ~2000 qps -- 20 times what your
    >> test found -- when answering for in-core authoritative data.

    Brad> 	Right, but what about the latest versions of BIND-8,
    Brad> ANS/CNS, and NSD on the same hardware?  

You miss my point. I wasn't comparing the performance of these
implementations. I was showing just how misleading your data is. The
modest hardware I used got 20 times the performance of the even more
modest hardware you used. And since my server is old and atypical of
today's name servers, this shows just how far your testbed was even
more removed from reality. Which is/was something you freely admitted
when you've given the presentation,

    Brad> Now that you've done
    Brad> the trivial in-core test, how about doing a test that is
    Brad> explicitly designed to exceed the amount of memory available
    Brad> by at least an order of magnitude?

This won't tell us anything we don't already know, so why bother?