[bind10-dev] Ask some questions for cucumber

[Jeremy C. Reed]

On Thu, 21 Apr 2011, qiaojing wrote:

> I saw the discussion about cucumber being validated to be used in bind10
> test.
>  
> After my investigation of this tool, some questions come up as follows:
>  
> * As I know, cucumber is a tool for BDD, and its obvious benefit is for
> design.
>    So I wonder what is the advantage of using it in bind10 as the codes are
> already there?
>    Or just to use its automated test mechanism?

We need a way to do functional testing where we can have reusable parts 
(like starting, configuring, queries, transfers, stopping, etc) and can 
generate useful reports. Cucumber may be nice since we can describe the 
behavior is plain text and write (hopefully) simple steps in Python.

We do have some of BIND 9's test framework also ported, with the new 
"systest" target. (We still need to automate this on one of our build 
farm machines.)  I don't know what is better for this (redoing using 
cucumber style or extending our systest).

Another goal is for it to be DNS-server agnostic. We hope to be able to 
use the suite of tests we develop against other DNS implementations too.

> * If it is chosen, I think it will be a time-consuming work to write the
> features and steps 
>    from scratch. Or if there is way to move the use cases from ready-made
> ones? And Where
>    are they? 

Yes, it will be time consuming. As for reusing use cases, I think we 
will need to rewrite all.

> * About the test code, it seems the step definition is like white-box test,
> it often directly
>    calls the lib of source code. So I think the test programmer should be
> familiar with the
>    implementation very much. 

I don't think we will use cucumber at that level. For that we will 
continue to use pytest and googletest.

> * As cucumber, ruby is the best language, and I see other languages
> might not be well-supported,
>    such as python and shell. And whether it should be the same language used
> in implementation?  

I hope we don't need to introduce another dependency (ruby) on all our 
build systems.

> * Is there a tester called 'dns cucumber'? Where shall I get it? And what is
> the three clone of cucumber
>    you discussed?

For the existing work, Michael Graff had done some experiments in ruby 
around a year ago. I don't know more about that.

As for python implementations of cucumber:

Freshen
https://github.com/rlisagor/freshen
(scroll down)

Lettuce
http://lettuce.it/

(As for three implementations: I was confused. I found another that was 
just an idea and not completed at
http://www.matthanger.net/2010/10/no-cucumber-for-python.html
and then also python examples for ruby cucumber at
http://github.com/aslakhellesoy/cucumber/wiki/Python .)

A major part of this effort is defining all the test cases (even if not 
writing the corresponding code). I started doing this for DNSSEC a few 
weeks ago, but got delayed by other work (botan, new web/git/trac 
server, issues with production auth server, log4cplus, etc. :)