Comparing query rates (was Re: 9.2.5 db causes high cpu?)
brad at stop.mail-abuse.org
Wed Feb 23 08:53:33 UTC 2005
At 4:38 AM +0000 2005-02-23, Jim Reid wrote:
> Furthermore, no sane person would run a production name server in an
> environment which has been deliberately set up to make the server
> thrash. This is like taking the wheels off a car and then measuring
> its 0-60 time: what's the point?
We show the best and worst case performance in the test
environment, and then assuming that the reader is using a similar
configuration, they should be able to expect that their performance
will fall somewhere between those extremes.
The bigger issue is to establish two points of reference for each
piece of software that is tested, so as to make relative comparisons
at least a little bit more meaningful.
> I am not badgering you at all.
It sure felt like badgering.
> I'm just asking that you try to avoid
> misleading people by quoting numbers out of context.
They should be able to get that by reading the intro to the
referenced talk. Moreover, while quoting performance numbers that
were seen, my primary point was (and always has been) to show the
relative performance differences between the software in question.
> At the very least
> you should describe your hardware platform, test methodology and
> explain that your results come with a whole lot of qualifiers that are
> unrepresentative of the typical DNS installation: a 133MHz Pentium
> laptop with 40 MB of RAM.
All of that is detailed in the slides for the talk, as well you know.
> Just saying name server X does Y qps helps
> no-one. Especially if X is operating in an environment that's nothing
> like the ones people use for their production name servers.
Everyone has a different environment. If your requirement for
quoting numbers is that the test environment has to perfectly match
the one that is used for the production for a given reader, then we'd
have to run the test on every single machine and possible combination
of machines on the planet.
Not only do I not have access to every single machine and
possible combination of machines on the planet, there is not enough
time in the Universe to run the sequence of tests on all those
systems, because by the time you've completed one run there will be a
whole host of new machines to test.
I detailed my testing environment and methods ad nauseum in the
talk. It's all there on the slides.
> You miss my point. I wasn't comparing the performance of these
Which is the *only* thing I was doing!
> I was showing just how misleading your data is. The
> modest hardware I used got 20 times the performance of the even more
> modest hardware you used.
The raw data has never been the point, it was not the point, and
it will never be the point.
The point was, is, and always will be, the relative performance
of the various pieces of software in question.
> And since my server is old and atypical of
> today's name servers, this shows just how far your testbed was even
> more removed from reality. Which is/was something you freely admitted
> when you've given the presentation,
Thank you for finally acknowledging that fact.
> Brad> Now that you've done
> Brad> the trivial in-core test, how about doing a test that is
> Brad> explicitly designed to exceed the amount of memory available
> Brad> by at least an order of magnitude?
> This won't tell us anything we don't already know, so why bother?
I think they do tell us something useful. I think a surprising
number of people run nameservers that are horribly misconfigured and
under-specified, and knowing the worst case performance is very
useful for doing capacity planning.
Knowing that server A performs okay when everything is completely
in memory but turns into a dead dog when you run out of memory is
very useful when comparing that against server B whose performance
may not be quite as high when everything is in memory, but whose
performance does not degrade nearly so badly when the data exceeds
the amount of memory available.
Brad Knowles, <brad at stop.mail-abuse.org>
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."
-- Benjamin Franklin (1706-1790), reply of the Pennsylvania
Assembly to the Governor, November 11, 1755
SAGE member since 1995. See <http://www.sage.org/> for more info.
More information about the bind-workers