[bind10-dev] auth testing

[Jeremy C. Reed]

I have questions at the bottom of this email.

Goals for BIND 10 development include having performance and resource 
usage targets. Also during development and design hopefully do 
benchmarking of our components and microbenchmarking of specific functions 
and tasks. To help define benchmark targets, I am working on gathering 
some data.

We now have a dedicated system for doing performance and benchmark 
testing. I don't know what type of hardware it is. I think it has two 
physical CPUs. The dmesg says:

 Memory: 4032428k/5242880k available (2493k kernel code, 152780k reserved, 
    1263k data, 200k init
 Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz stepping 05
 Brought up 8 CPUs

The operating system is CentOS 5.3 running a x86_64 Linux 
2.6.18-128.7.1.el5 with SMP.

I installed a few DNS servers from source:

bind-8.4.7
bind-9.6.1
pdns-2.9.22
nsd-3.2.3
(and a simple 40 line python script that provides a fake auth DNS server).

(I didn't realize I didn't have openssl headers installed yet until I nsd 
which requires them. So my bind8 and bind9 are built without SSL. I will 
redo my tests for them later with ssl. powerdns doesn't use ssl, so 
doesn't matter.)

I made simple configurations for testing authoritative, recursion turned 
off, and running on alternative port. (For pdns, I use the "bind" 
backend.)

I have a simple zone with three records: SOA, NS, and A (for that NS 
target).

My script to gather resource usage stores the /proc/$PID/status, stat, and 
statm files, and ps output like:

%CPU  C     ELAPSED     TIME   TIME - -  UTIME %MEM   RSS   RES    SZ - 
START    VSZ COMMAND
40.0 40       00:01 00:00:00   0:00 - -      -  0.0  1808     -  1512 - 
12:10   5736 named

The queryperf I used for all of them is from bind-9.6.1/contrib. I use no 
special options for it. My queryperf input file is 30,000 entries long and 
just repeats the queries for the three available records. I am running 
queryperf on the same system.

I wrote some scripts to start each server, record some resource usage, dig 
one time (prime it?), record resource usage, queryperf, record resource 
usage, stop server.

I have lots of data stored, but here is a summary:

Server		Queries/second	Virtual Size	cputime/realtime
------		--------------	------------	----------------
bind-8.4.7	77582 qps	5736 kB		40.0%
bind-9.6.1	37745 qps	8536 kB		79.0%
pdns-2.9.22	89884 qps	148364 kB	0.0%
nsd-3.2.3	83938 qps	14228 kB	0.0%
python-test	29812 qps	104184 kB	101%

(I don't know why the %CPU is 0.0 sometimes. And I don't know how it is 
101% either, must be very inefficient? Also I saw pdns reach 213900 kB 
VSIZE and 32.0 PCPU. And I have see nsd reach 90450 qps. I did several 
runs, but above summary is not an average.)

I didn't do any special configurations to help with performance or memory 
usage. As for pdns, I couldn't get it to even run when I tried to ulimit 
-v 100000.

I plan on doing various other tests too (such as huge zones and many 
zones) and also adding tinydns and maybe a server based on perl 
Net::DNS::Server to the mix. (Months ago, I did some ACL performance and 
resource usage tests. I may also do them again.)

Later, I will also do same tests on a 233MHz StrongARM system with 32M (or 
64M) memory running NetBSD. (I don't have the hardware yet.)

My questions:

1) Any types of tests you'd like me to run?

2) Any configurations you'd like me to use?

3) Other server software?

4) Any queryperf patches or options I should use?

5) Any queryperf input data files I should use?

6) What other data should I collect?

7) Any suggestions or pointers for creating graphs? What do you want to 
see? How should it be presented? Should I average results?