Comparing query rates (was Re: 9.2.5 db causes high cpu?)

Brad Knowles brad at stop.mail-abuse.org
Wed Feb 23 08:53:33 UTC 2005


At 4:38 AM +0000 2005-02-23, Jim Reid wrote:

>  Furthermore, no sane person would run a production name server in an
>  environment which has been deliberately set up to make the server
>  thrash. This is like taking the wheels off a car and then measuring
>  its 0-60 time: what's the point?

	We show the best and worst case performance in the test 
environment, and then assuming that the reader is using a similar 
configuration, they should be able to expect that their performance 
will fall somewhere between those extremes.

	The bigger issue is to establish two points of reference for each 
piece of software that is tested, so as to make relative comparisons 
at least a little bit more meaningful.

>  I am not badgering you at all.

	It sure felt like badgering.

>                                 I'm just asking that you try to avoid
>  misleading people by quoting numbers out of context.

	They should be able to get that by reading the intro to the 
referenced talk.  Moreover, while quoting performance numbers that 
were seen, my primary point was (and always has been) to show the 
relative performance differences between the software in question.

>                                                       At the very least
>  you should describe your hardware platform, test methodology and
>  explain that your results come with a whole lot of qualifiers that are
>  unrepresentative of the typical DNS installation: a 133MHz Pentium
>  laptop with 40 MB of RAM.

	All of that is detailed in the slides for the talk, as well you know.

>                             Just saying name server X does Y qps helps
>  no-one. Especially if X is operating in an environment that's nothing
>  like the ones people use for their production name servers.

	Everyone has a different environment.  If your requirement for 
quoting numbers is that the test environment has to perfectly match 
the one that is used for the production for a given reader, then we'd 
have to run the test on every single machine and possible combination 
of machines on the planet.

	Not only do I not have access to every single machine and 
possible combination of machines on the planet, there is not enough 
time in the Universe to run the sequence of tests on all those 
systems, because by the time you've completed one run there will be a 
whole host of new machines to test.


	I detailed my testing environment and methods ad nauseum in the 
talk.  It's all there on the slides.

>  You miss my point. I wasn't comparing the performance of these
>  implementations.

	Which is the *only* thing I was doing!

>                    I was showing just how misleading your data is. The
>  modest hardware I used got 20 times the performance of the even more
>  modest hardware you used.

	The raw data has never been the point, it was not the point, and 
it will never be the point.

	The point was, is, and always will be, the relative performance 
of the various pieces of software in question.

>                            And since my server is old and atypical of
>  today's name servers, this shows just how far your testbed was even
>  more removed from reality. Which is/was something you freely admitted
>  when you've given the presentation,

	Thank you for finally acknowledging that fact.

>      Brad> Now that you've done
>      Brad> the trivial in-core test, how about doing a test that is
>      Brad> explicitly designed to exceed the amount of memory available
>      Brad> by at least an order of magnitude?
>
>  This won't tell us anything we don't already know, so why bother?

	I think they do tell us something useful.  I think a surprising 
number of people run nameservers that are horribly misconfigured and 
under-specified, and knowing the worst case performance is very 
useful for doing capacity planning.

	Knowing that server A performs okay when everything is completely 
in memory but turns into a dead dog when you run out of memory is 
very useful when comparing that against server B whose performance 
may not be quite as high when everything is in memory, but whose 
performance does not degrade nearly so badly when the data exceeds 
the amount of memory available.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

   SAGE member since 1995.  See <http://www.sage.org/> for more info.


More information about the bind-workers mailing list