DHCP Validation project - testing framework

[Marcin Siodelski]

On 11/13/2012 06:00 PM, Włodzimierz Wencel wrote:
> Hello,
> 
> It is my first mail here and also first request for review. So it is
> good time to introduce myself. As you all aware (or not) there is DHCP
> validation project http://bind10.isc.org/wiki/DhcpTesting
> I'm Włodek Wencel, studying at Gdansk University of Technology I have
> opportunity to work for ISC creating testing environment  for DHCPv6 as
> my thesis. I hope that in next half year, with your support, I will
> create good piece of code ;)
> 
> I'm asking for review over design document of framework.
> http://bind10.isc.org/wiki/frameworkv6
> Because it's my first project of that kind I ask you for any hints,
> corrections and tips or new features, every response will be much
> appreciated.
> 
> 
> Regards,
> 
> Włodzimierz Wencel
> 
> 
> 
> _______________________________________________
> bind10-dhcp mailing list
> bind10-dhcp at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind10-dhcp
> 

Hi,

A few quick comments following the other thread that you have with
Stephen....

*Design*
I think that *Requirements* will be a better name for the introductory
section like this. Design is more about how you want to fulfil requirements.

Extensibility: It is indeed not for DHCPv6 only. It should work for
whatever DHCP implementation.

*Test Schema*
I am seeeing some weakness in the example you have given. In fact, from
the client perspective the server behaves correctly: it sent a couple of
solicits and got no reponse which suggests that server complies with the
RFC. However, if the server crashed after receiving first solicit
without DUID (because programmer made a wrong assumption that DUID
should be there and tried to read it without checking that it is
present) test should be marked as FAILED. If it is marked as PASSED and
you fire-up another tests using a dead server you end-up having all
subsequent tests failed while in fact only the first one was FAILED.
This brings another question, how do you validate that the server is
running before you run each test case. You should probably send some
valid message and see if it replies. This makes simple test a bit more
complex but the good think about it is that you can use the same check
methodology for each test.

I think we can have multiple tests covering specific requirements within
a section. Thus if you want to run the test by specifying "RFC section"
you have to be prepared that multiple tests will have to be executed.

*Running tests*
The --SAVE option makes the script "save the user configuration" but it
is not explain what the configuration is. Is it a dump of the command
line? How do I reuse this dumped configuration when running test again.

I can see that there is a note about "config files" under "modes:"
bullet but it would be nice to have an example or a template of a config
file. This is a good occasion to think through what you really want to
make configurable for the user.

Since you already refer to the specific options that are exposed by the
command line, maybe it is worth to add separate section describing the
(preliminary) command line in the form of "man page" or output from
"--help command"?

*Adding new tests*
What atcually it is to "generate test file with blank space"? Is this a
piece of code being generated like "function name, parameters and empty
body" and the function body is where you have to put the test logic in
python? I am wondering if it is really needed feature given that there
are large number of tests to be written and those obviously have higher
priority.

'self-test': can this be called unit-tests?

Marcin