BIND 10 #1290: initial setup of system tests with cucumber

[BIND 10 Development]

#1290: initial setup of system tests with cucumber
-------------------------------------+-------------------------------------
                   Reporter:  jelte  |                 Owner:  jelte
                       Type:  task   |                Status:  reviewing
                   Priority:  major  |             Milestone:
                  Component:         |  Sprint-20111108
  Unclassified                       |            Resolution:
                   Keywords:         |             Sensitive:  0
            Defect Severity:  N/A    |           Sub-Project:  DNS
Feature Depending on Ticket:         |  Estimated Difficulty:  9
        Add Hours to Ticket:  0      |           Total Hours:  0
                  Internal?:  0      |
-------------------------------------+-------------------------------------
Changes (by stephen):

 * owner:  stephen => jelte


Comment:

 Reviewed commit 46adf014f18c6b3f9a685b8f0fdd0775a583a7c5.

 First off, well done.  I'm impressed by the amount that's been covered in
 this ticket and think it a very good first step in getting a comprehensive
 system test suite together.

 '''src/bin/bindctl/bindcmd.py'''[[BR]]
 run(): the first "if" check includes a "return" without a value although
 all other return statements return integers.

 '''tests/lettuce/README'''[[BR]]
 > The bind10 main script, bindctl script, and dig must all be in the
 default search path of your environment, and BIND 10 must not be running
 if you use the installed version when you run the tests.

 What is the "bind10 main script"?

 > If you want to test an installed version of bind 10, just run 'lettuce'
 in this directory.
 But where is BIND 10 installed?  Perhaps we should mandate some
 environment variables, e.g.

 * BIND10_TOP - the directory tree holding the installed version of BIND
 10. If not defined, the version used is the one in the tree in which the
 tests are located.
 * BIND9_TOP - holds the (build) source tree of BIND 9.  (I add this as for
 the moment, some of the interoperability tests (i.e. IXFR) require use of
 another nameserver.)  If defined, "dig" is obtained from here, otherwise
 it is sought from the standard search path.

 > We have set up the tests to assume that lettuce is run from this
 directory, so even if you specify a specific feature file, you should do
 it from this directory.
 Comment: At some time in the future we may want to be able to run the
 tests from another directory.

 > These directories are currently not divided further; we may want to
 consider this as the set grows. Due to a (current?) limitation of Lettuce,
 for feature files this is currently not possible; the python files
 containing steps and terrain must be below or at the same level of the
 feature files.
 Git allows symbolic links to be stored in the repository, so we could
 consider some form of hierarchy with symbolic links from the location
 expected by Lettuce.

 > as that way we might run into a 'symmetric bug'
 I assume that you mean that to do the test we will be relying on code that
 we are testing?


 '''tests/lettuce/README.tutorial'''[[BR]]
 The content is there although I think it could be phrased better in a few
 cases.  However, I think it is fine for now; as we get more experience
 with Lettuce we may need to revisit this documentation. A few points:

 When explaining the starting of BIND 10 (the paragraph that first includes
 the phrase "python decorator"), the various arguments are mentioned.  It
 would be helpful to illustrate this with an example statement staring BIND
 10 and specify a different port and process name.

 Also worth noting that when defining a step, Lettuce is searching each
 line in the scenario for the defined phrase and that the extra words are
 just for readability (e.g in the last example, "Given" could be replaced
 by "After" with no effect on the test).

 '''tests/lettuce/configurations/example.org.config.orig'''[[BR]]
 '''tests/lettuce/configurations/example2.org.config'''[[BR]]
 '''tests/lettuce/configurations/no_db_file.config'''[[BR]]
 These would be easier to read if the configuration were split across
 several lines.

 '''tests/lettuce/features/example.feature'''[[BR]]
 Typo - "existance" should be spelt "existence".

 '''tests/lettuce/features/terrain/bind10_control.py'''[[BR]]
 This is Python code and even though it includes Lettuce decorators, it
 should still be properly commented - including ISC header, function
 headers etc.  In particular, the parameters expected for each step should
 be listed (e.g. "set bind10 configuration (\S+) to (.*)" - is the first
 parameter the optional name of which bind10 I am referring to?)

 check_lines() is more really "find_line".  Also, the function should end
 "return None" to make it clearer what is returned if the line is not
 found.

 How exact is the match of the phrase?  In particular, I am thinking of
 initial capital letters: for example, do both
 {{{
 Then set bind10 configuration to x
 Set bind10 configuration to x
 }}}
 ... match the phrase required to execute set_config_command().  If not, is
 is possible to be flexible and let each step start with either a capital
 or non-capital letter?

 We probably need some documentation as to what phrases are available.


 '''tests/lettuce/features/terrain/querying.py'''[[BR]]
 Needs ISC header.

 Functions/methods need headers - examples would be useful.

 Not clear from the documentation how you check a combination of flags in a
 query response.  Must the order you give them in your question be the same
 as they appear in the "dig" output?

 '''tests/lettuce/features/terrain/steps.py'''[[BR]]
 '''tests/lettuce/features/terrain/terrain.py'''[[BR]]
 Needs ISC header.

 Functions/methods need headers

-- 
Ticket URL: <http://bind10.isc.org/ticket/1290#comment:10>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development