DHCP Validation project - testing framework

[Włodzimierz Wencel]

>
> With DHCP, a test may well require multiple clients.  (And multiple
> servers if you are testing DHCP failover.) In this case, it would be
> good to avoid the need to log into multiple systems to start the tests.
>
> I think that a central console that connects to the servers would be
> the way to go.  It does not have to be exclusively interactive: it
> could also operate as a program taking arguments, e.g.
>
>     $ ./test-console
>     test-console> start test_45
>     Starting test_45 on server1 ... started
>     Starting test_45 on client1 ... started
>     Starting test_45 on client2 ... started
>     test-console>
>
> or
>
>     $ ./test-console "start test_45"
>     Starting test_45 on server1 ... started
>     Starting test_45 on client1 ... started
>     Starting test_45 on client2 ... started
>     $
for now I prefer second option ;)
> I would advise doing something quick and easy first and getting a
> first version of the framework working before doing anything clever
> like an interactive creator.  This would suggest some form of file.
I need to learn sth about lettuce and then I will be able to create and 
present first version. (for now I have just part that controls DHCP 
server and generator for fake massages)
> Perhaps another use-case is "What needs to be in a test".  Taking a
> simple test, we might need the following sequence of steps:
>
>      install server(commit id 1234...) on server1
>      install client(commit id 1234...) on client1
>      install server-configuration(test45) on server1
>      install client-configuration(test45) on server2
>      start server(using test45 server command-line switches)
>      start client(using test45 client command-line switches)
>      wait 10 minutes
>      kill client
>      kill server
>      copy client-logs to repository
>      copy server-logs to repository
>
> This seems to suggest that even a simple test is going to require some
> form of sequencing.  It also suggests that a test may comprise a
> series of steps, and each step itself may comprise a series of
> substeps (e.g. "install server" may include copying the tarball to the
> server, unpacking it, configuring it, running "make", then installing it.)
>
Apart from the actual details of the program, first assumption, made by 
Tomek Mrugalski was:

    starting server on one virtual machine
    on the other machine program generate fake messages (using scapy)
    and check if answer is correct.
    testing DHCP don't bring real client to process (client like dhclient)

Is it necessary to bring clients to testing environment? It would be 
easier to make new tests, but in that case lettuce/scapy are useless 
(and in fact that would change my thesis little bit to much). I think 
that server validation should be performed on messages level not on 
logs. And it's much more faster (worst case scenario we wait just time 
of max_retry_attempts * timeout_value)

I didn't think that installation should be performed by program, but 
it's good idea. Also concept provides that it will be only one 
configuration file (or several) and test that needs different config 
will just contain something to add to config file. Not one test = one 
config file.
> In fact, for a given commit ID, the test may not be unique.  It could
> be that the user wants to re-run a test on the same commit ID (perhaps
> anomalous results were obtained and it is suspected that something
> else was running on the test machine at the same time).  Under these
> circumstances a date/time needs to be included.
That was reason for making two different files. Tomorrow I will make 
some 'example test report', it will be easier.
> This is beginning to suggest that using a version number, commit ID or
> a date/time for a test identification may not be the right way to go.
>   Perhaps a unique number (results-1, results-2 ...) with an associated
> database/file listing the attributes of the tests?  When accessing a
> test by version/date/commit-id, a query is made on the database to
> identify the relevant directory/directories.
>
>
> This would be the advantage of XML - the information about the test
> could also be placed in the XML file, and what is displayed depends on
> the XSLT used.
So it's time to think again about details, and rewrite some parts of 
concept.

Włodek Wencel

-- 
Goal of Open Source is to *Share *
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind10-dhcp/attachments/20121116/830c9d24/attachment.html>