How to implement SIGHUP tests

Marcin Siodelski marcin at isc.org
Tue May 27 12:38:32 UTC 2014



On 27/05/14 14:11, Stephen Morris wrote:
> On 27/05/14 10:19, Marcin Siodelski wrote:
> 
>> 1. Shell is there on every OS and doesn't cause any additional 
>> dependencies for Kea to be tested, whilst this is not true for
>> Python. Even if we can detect that there is no Python on the system
>> and not run these tests, the down side is that we don't run these
>> tests.
> 
> That's true.
> 
> 
>> 2. One of the tasks for 0.9 is to implement a script that controls
>> the startup of Kea. We might not have decided on what language it
>> will be written in, but I also don't think anybody wanted Python as
>> we are in the process of removing Python-based code from Kea. The
>> shell script makes a lot of sense to me and if so, I would like to
>> test Kea startup and reconfiguration using the same approach as
>> this script would use.
> 
> We would want to test that the script starts and stops Kea; we don't
> need to test Kea in the same way.
> 

Sure, but the more real life test scenario it is, the better. When you
test the script and it fails you don't know whether it is a bug in the
script or in the server. Having a test that validates the server
directly eliminates the influence of the script and makes it easier to
identify the issue.

>>
>> 3. As Tomek pointed out, the shell scripts don't provide any
>> framework for testing (nothing similar to gtest) which makes it
>> more reasonable to implement our tests in C++ and within gtest
>> framework. But, python based tests that we already have don't have
>> any framework either and we used to live with it. So, is lack of
>> framework really a problem?
> 
> I thought Python did have a unit test framework:
> 
> https://docs.python.org/release/2.6/library/unittest.html
> 
> ... and that BIND 10 used it.
> 

Ok, point taken.

>>
>> 4. Implementing tests in C++ would certainly make tests consistent
>> with other tests we have and I am in general in favor of that. But,
>> C++ is not the easiest environment to deal with the type of tests
>> we are talking about. Dealing with spawning background processes,
>> grepping log files, creating configuration files is something that
>> is natural in shell because shell is basically for this purpose
>> mainly. I don't disagree that it is doable in C++: you can fork,
>> execlp, read file in C++ (harder to grep but doable) but it doesn't
>> make code implementation and maintenance any easier.
> 
> I agree that searching log files is awkward, but there is no need to
> do that.  The logging framework permits log messages to be sent to
> stdout (or stderr) and that can be read by the parent process via a pipe.

Whether I get it through a pipe or by reading a file it still seems that
it is easier to parse the output with a script, rather than with C++.
This includes things like counting how many occurrences of some log
message appeared so far - e.g. how many reconfigurations.

> 
> I would think a C++ class for handing subprocesses that has methods like:
> 
> <constructor>(command)
> - start a process in the background and store the PID. Connect stdout
> of the output process to a pipe between it and the parent process.
> 
> isSubrocessRunning()
> - return true or false if the background process is running.
> 
> searchOutput(text, timeout)
> - wait a maximum of "timeout" for the specified text to be output by
> the background process. As each log message has a unique ID, passing
> that ID as an argument to this method is a way of checking whether the
> message has been output.
> 
> sendSignal(signal)
> - send the specified signal to the background process.
> 
> waitForProcessTermination(timeout)
> - wait up to "timeout" for the process to terminate.
> 
> ... would be sufficient to do the testing

It is sufficient, unless we will come up with some more sophisticated
test scenarios in the future it will will just make it harder and harder
for unit testing.

> 
> We would only have to write this once and the advantage about C++ is
> that it would naturally fit into the existing unit test framework.
> There are already some unit testing utilities in Kea, in
> src/lib/util/unittests - this would just be another.
> 
>> Tomek's major objections to this seem to be:
>>
>> 1. We already have tests in Python and they should be portable
>> with Python 2.x if people hate Python 3 and its incompatibilities
>>
>> 2. If you run unit tests you're probably a developer and you can
>> afford installing Python 2.x.
>>
>> 3. There is no test framework for shell but there is for C++. This
>> is in favor of writing a test in C++.
>>
>> 4. We are already using Forge that is written in Python so we could
>> move some tests between Kea and Forge.
>>
>> And Tomek, has a list of preferred languages for these tests: 1.
>> C++ 2. Python 3. Shell
> 
> I agree with Tomek here.
> 
> One of the problems with BIND 10 was the use of two languages because
> few people were masters of both.  Unfortunately, we can't get rid of
> Python code altogether: Lettuce is written in Python.  However,
> Lettuce is all about writing unit tests in a natural form of English.
> To my mind, that far outweighs the inconvenience of maintaining the
> Python needed to implement Lettuce features.

What do we use lettuce for? I know we do use lettuce in Forge. But, what
in Kea is tested with lettuce?

If we want to write a script controlling Kea in shell, we will be using
shell anyway. So, it seems that knowledge of shell is sort of important
for Kea team anyway.

> 
>> I would really like to have input from others why you like or
>> dislike a certain approach because it may affect the way we develop
>> similar tests in the future. Apart from 3 test cases we are talking
>> about, there will be 6 additional test cases for other components:
>> 3 for D2 and 3 for v4. It is better to alter the course now, not
>> when they are implemented. Having said that, I should have
>> initiated this discussion prior to implementing tests for v6, but
>> better now than later.
> 
> In summary, I think if we were to implement a C++ class for handling
> subprocesses that output to stdout, we could do a lot of this type of
> testing in the C++ unit tests.

If people feel this is a general consensus, that's fine. I'll still wait
for additional input from other mail list members.

Marcin


More information about the kea-dev mailing list