BIND 10 #1290: initial setup of system tests with cucumber

[BIND 10 Development]

#1290: initial setup of system tests with cucumber
-------------------------------------+-------------------------------------
                   Reporter:  jelte  |                 Owner:  stephen
                       Type:  task   |                Status:  reviewing
                   Priority:  major  |             Milestone:
                  Component:         |  Sprint-20111108
  Unclassified                       |            Resolution:
                   Keywords:         |             Sensitive:  0
            Defect Severity:  N/A    |           Sub-Project:  DNS
Feature Depending on Ticket:         |  Estimated Difficulty:  9
        Add Hours to Ticket:  0      |           Total Hours:  0
                  Internal?:  0      |
-------------------------------------+-------------------------------------
Changes (by jelte):

 * owner:  jelte => stephen


Comment:

 Replying to [comment:10 stephen]:
 > Reviewed commit 46adf014f18c6b3f9a685b8f0fdd0775a583a7c5.
 >
 > First off, well done.  I'm impressed by the amount that's been covered
 in this ticket and think it a very good first step in getting a
 comprehensive system test suite together.
 >

 Thanks :)


 > '''src/bin/bindctl/bindcmd.py'''[[BR]]
 > run(): the first "if" check includes a "return" without a value although
 all other return statements return integers.
 >

 ack, fixed

 > '''tests/lettuce/README'''[[BR]]
 > > The bind10 main script, bindctl script, and dig must all be in the
 default search path of your environment, and BIND 10 must not be running
 if you use the installed version when you run the tests.
 >
 > What is the "bind10 main script"?
 >

 '<prefix>/bin/bind10'. Change it to 'executable'?


 > > If you want to test an installed version of bind 10, just run
 'lettuce' in this directory.
 > But where is BIND 10 installed?  Perhaps we should mandate some
 environment variables, e.g.
 >
 > * BIND10_TOP - the directory tree holding the installed version of BIND
 10. If not defined, the version used is the one in the tree in which the
 tests are located.
 > * BIND9_TOP - holds the (build) source tree of BIND 9.  (I add this as
 for the moment, some of the interoperability tests (i.e. IXFR) require use
 of another nameserver.)  If defined, "dig" is obtained from here,
 otherwise it is sought from the standard search path.
 >

 I must say I don't really see why. I know we have something like this
 for current systests, but why not 'just' take whatever the environment
 is already setup to use? (e.g. 'just take the bind10 from the users
 path settings. If user wants different bind10, let him change his
 settings', also implying 'test the system by default exactly how it
 would be called outside of this environment'. And of course provide
 some helper scripts to set it up for some special occasions, like the
 current setup script to run from source). But maybe I'm just too
 used to running different versions software from non-standard paths,
 and switching between them. But see below.


 > > We have set up the tests to assume that lettuce is run from this
 directory, so even if you specify a specific feature file, you should do
 it from this directory.
 > Comment: At some time in the future we may want to be able to run the
 tests from another directory.
 >

 we can put in a run script here as well that just changes cwd to this
 directory, maybe that could also translate those _TOP thingies. But
 i'm not sure whether this would not be more confusing, really.

 > > These directories are currently not divided further; we may want to
 consider this as the set grows. Due to a (current?) limitation of Lettuce,
 for feature files this is currently not possible; the python files
 containing steps and terrain must be below or at the same level of the
 feature files.
 > Git allows symbolic links to be stored in the repository, so we could
 consider some form of hierarchy with symbolic links from the location
 expected by Lettuce.
 >

 ack. Not that I have taken any action in that directory, but I prefer
 to improve lettuce to support a base environment for (all or specific)
 tests :)

 > > as that way we might run into a 'symmetric bug'
 > I assume that you mean that to do the test we will be relying on code
 that we are testing?
 >

 yes

 >
 > '''tests/lettuce/README.tutorial'''[[BR]]
 > The content is there although I think it could be phrased better in a
 few cases.  However, I think it is fine for now; as we get more experience
 with Lettuce we may need to revisit this documentation. A few points:
 >
 > When explaining the starting of BIND 10 (the paragraph that
 > first includes the phrase "python decorator"), the various arguments
 > are mentioned.  It would be helpful to illustrate this with an
 > example statement staring BIND 10 and specify a different port and
 > process name.
 >

 ok, added an example


 > Also worth noting that when defining a step, Lettuce is searching
 > each line in the scenario for the defined phrase and that the extra
 > words are just for readability (e.g in the last example, "Given"
 > could be replaced by "After" with no effect on the test).
 >

 added


 > '''tests/lettuce/configurations/example.org.config.orig'''[[BR]]
 > '''tests/lettuce/configurations/example2.org.config'''[[BR]]
 > '''tests/lettuce/configurations/no_db_file.config'''[[BR]]
 > These would be easier to read if the configuration were split across
 several lines.
 >

 done


 > '''tests/lettuce/features/example.feature'''[[BR]]
 > Typo - "existance" should be spelt "existence".
 >

 oopsie, fixed


 > '''tests/lettuce/features/terrain/bind10_control.py'''[[BR]]
 > This is Python code and even though it includes Lettuce decorators,
 > it should still be properly commented - including ISC header,
 > function headers etc.  In particular, the parameters expected for
 > each step should be listed (e.g. "set bind10 configuration (\S+) to
 > (.*)" - is the first parameter the optional name of which bind10 I am
 > referring to?)
 >

 I've documented them all as function parameters, where if it is a
 step, the step 'part' is it is taken from is described as well, and
 whether it is optional. I'm not sure whether this is the best approach
 in the end but it is all i could come up with right now.

 > check_lines() is more really "find_line".  Also, the function should end
 "return None" to make it clearer what is returned if the line is not
 found.
 >

 actually, it wasn't even used anymore, so I removed it completely.

 > How exact is the match of the phrase?  In particular, I am thinking of
 initial capital letters: for example, do both
 > {{{
 > Then set bind10 configuration to x
 > Set bind10 configuration to x
 > }}}
 > ... match the phrase required to execute set_config_command().  If not,
 is is possible to be flexible and let each step start with either a
 capital or non-capital letter?
 >

 It is completely case-insensitive when matching, but any part we
 pull out and use as arguments are taken as-is (e.g. for
 'with configuration <file>', the file will be case-sensitive).

 Added something about this to the readme.

 > We probably need some documentation as to what phrases are available.
 >

 Hmm, yes, probably. Not sure how to do it though, I don't really want
 to retype the entire function description yet again (which will go
 stale).

 >
 > Not clear from the documentation how you check a combination of flags in
 a query response.  Must the order you give them in your question be the
 same as they appear in the "dig" output?
 >

 Added a line to the example feature, and to the initial comments for
 queryresult class.

-- 
Ticket URL: <http://bind10.isc.org/ticket/1290#comment:11>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development