[bind10-dev] benchmarking

[Shane Kerr]

Jeremy,

Great, thanks for this! Some comments below...

On Monday, 2012-03-05 21:09:14 -0600, 
"Jeremy C. Reed" <jreed at isc.org> wrote:
> The wikipage at http://bind10.isc.org/wiki/DnsBenchmarks will
> document our official standard and specific DNS benchmarks we will
> use for measuring BIND 10 performance improvements.

I think we need to provide links to the actual data and scripts used
for the benchmarking. Probably we should include the scripts in the
BIND 10 source code, but I can understand if the data is perhaps too
big.

As with all of BIND 10, the goal should be for anyone to be able to
independently reproduce these on their own stuff, and also for the
developers to use these benchmarks to test things as they are coding.
Sometimes counter-intuitive results occur, especially with the complex
interactions between memory, CPUs, networks, and operating systems.

> The wikipage at http://bind10.isc.org/wiki/DnsBenchmarkIdeas is a
> free list of ideas and suggestions of what to research. This list is
> based on ideas shared at various meetings. Feel free to add your
> thoughts.

Looks good. I find having a collection place for these things is
helpful. We can expand our benchmarking as it becomes more useful.

> With the many variables (label size, response size, number of zones, 
> size of zones, DO bit, UDP/TCP, on and on) we could create hundreds
> of different reports based on similar date with slight differences of 
> testing.

Right. Perhaps we'll do that someday, although we'd probably want some
web 2.0-style interface if we do that. Or maybe just in text files....

> Our goal has been initially to be as fast as BIND 9. 

"No slower than BIND 9". ;)
 
> I will add a third trac wiki page to document the existing setup. I
> have done many tests of old BIND 9.6, old BIND 8, other DNS servers,
> 50+ date-based snapshots of BIND 10. I will do some recent BIND 9
> versions soon.

Also note that we are not planning on publishing any benchmarks versus
non-ISC software. If someone wants to do that, even mentioning it on
this list, then that is fine. We just don't think it is appropriate for
ISC to do that. 

Cheers,

--
Shane