BIND 10 #1264: Design document for DHCP benchmarking utility

[BIND 10 Development]

#1264: Design document for DHCP benchmarking utility
-------------------------------------+-------------------------------------
                   Reporter:         |                 Owner:  johnd
  stephen                            |                Status:  reviewing
                       Type:  task   |             Milestone:  Sprint-
                   Priority:  major  |  DHCP-20111026
                  Component:  dhcp   |            Resolution:
                   Keywords:         |             Sensitive:  0
            Defect Severity:  N/A    |           Sub-Project:  DHCP
Feature Depending on Ticket:         |  Estimated Difficulty:  0
        Add Hours to Ticket:  0      |           Total Hours:  0
                  Internal?:  0      |
-------------------------------------+-------------------------------------

Comment (by tomek):

 Replying to [comment:7 stephen]:
 > Replying to [ticket:1263#comment:2 tomek]
 >
 > > I have couple of comments.
 > Hmm... this is a definition of "couple" of which I was previously
 unaware :-)
 I planed this comment to be short, but it kind of got carried away.

 > > Regarding the -r option, it is useful, but it is not enough.
 > > :
 > > To meet those usages, -time (or -t) option should be added that
 specifies duration...
 > Agreed.  Since r * t = n, the command parser should accept any two and
 calculate the third (objecting if all three are given).  I would suggest
 that the default for r be something
 No necessarily object. In stability or stress testing, these can be
 considered exit criteria. Tool should conclude test whichever comes first.
 "Try to get 1 million leases, but I don't want this test to last longer
 than 1 hour" is a valid test scenario.

 like 10/second; if neither t nor n is specified, assume a value of n equal
 to a 2^32^ - 1, i.e. essentially unlimited.
 >
 > (As an aside, allowing very large values for n complicates the mapping
 of packet ID to information about the exchange as a simple pre-allocated
 array cannot be used.  However some form of double-buffer - where a buffer
 can be reused once a the time equal to (time last packet using this buffer
 was sent + packet drop time) has passed - should work.)
 We may ignore this aspect for our first release, but this tool should have
 reasonable memory requirements. When used in stability test (e.g. hammer
 our server over a weekend), it should not take much more memory than
 during relatively short test. Some form of sliding window should be used.

 > > Besides of using dhcperf as manual tool, it will also be used as
 automated test. In that case it should have clearly state if specified
 pass criteria are met or not. Something that could be easily parsed by
 automated environments.
 > >
 > > Make sure that return code will specify status.
 > This sounds useful, although I'm not clear what you mean here.  In any
 case, I think it is something that can be added later.  Could you raise a
 ticket for it?
 I believe there is no ticket necessary. That is really simple. dhcperf
 should print out TEST PASSED or TEST FAILED as one of the last line. That
 will be easy to parse by a script or execution environment of some sort.

 Note that just giving PASS/FAIL status without explanation is not enough.
 If test fails, it should explain why. For example "Maximum allowed dropped
 packets:1 actually dropped: 5". That is another suggestion from
 experience. In time, we will use this tool to run many different test
 scenarios in automated environment somewhat similar to our build bot. When
 we spot that test failed, the first question will always be "why?". Having
 the ability to just scroll down the log file and get first rough idea
 about what went wrong is very useful.

 > for now, as suggested above, the return code should be 1 if any packets
 were dropped.
 Sounds good.

 > See above when the "-o" option is specified.  Do you think there is a
 need to echo them if used interactively, at a terminal?
 Yes. People often redirect terminal output to a file. Or even copy-paste
 from terminal if something breaks. Printing out a single line does not
 make the log much bigger, and it will spare us troubles.

-- 
Ticket URL: <http://bind10.isc.org/ticket/1264#comment:9>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development