How to implement SIGHUP tests

[Stephen Morris]

On 27/05/14 10:19, Marcin Siodelski wrote:

> 1. Shell is there on every OS and doesn't cause any additional 
> dependencies for Kea to be tested, whilst this is not true for
> Python. Even if we can detect that there is no Python on the system
> and not run these tests, the down side is that we don't run these
> tests.

That's true.


> 2. One of the tasks for 0.9 is to implement a script that controls
> the startup of Kea. We might not have decided on what language it
> will be written in, but I also don't think anybody wanted Python as
> we are in the process of removing Python-based code from Kea. The
> shell script makes a lot of sense to me and if so, I would like to
> test Kea startup and reconfiguration using the same approach as
> this script would use.

We would want to test that the script starts and stops Kea; we don't
need to test Kea in the same way.

> 
> 3. As Tomek pointed out, the shell scripts don't provide any
> framework for testing (nothing similar to gtest) which makes it
> more reasonable to implement our tests in C++ and within gtest
> framework. But, python based tests that we already have don't have
> any framework either and we used to live with it. So, is lack of
> framework really a problem?

I thought Python did have a unit test framework:

https://docs.python.org/release/2.6/library/unittest.html

... and that BIND 10 used it.

> 
> 4. Implementing tests in C++ would certainly make tests consistent
> with other tests we have and I am in general in favor of that. But,
> C++ is not the easiest environment to deal with the type of tests
> we are talking about. Dealing with spawning background processes,
> grepping log files, creating configuration files is something that
> is natural in shell because shell is basically for this purpose
> mainly. I don't disagree that it is doable in C++: you can fork,
> execlp, read file in C++ (harder to grep but doable) but it doesn't
> make code implementation and maintenance any easier.

I agree that searching log files is awkward, but there is no need to
do that.  The logging framework permits log messages to be sent to
stdout (or stderr) and that can be read by the parent process via a pipe.

I would think a C++ class for handing subprocesses that has methods like:

<constructor>(command)
- start a process in the background and store the PID. Connect stdout
of the output process to a pipe between it and the parent process.

isSubrocessRunning()
- return true or false if the background process is running.

searchOutput(text, timeout)
- wait a maximum of "timeout" for the specified text to be output by
the background process. As each log message has a unique ID, passing
that ID as an argument to this method is a way of checking whether the
message has been output.

sendSignal(signal)
- send the specified signal to the background process.

waitForProcessTermination(timeout)
- wait up to "timeout" for the process to terminate.

... would be sufficient to do the testing

We would only have to write this once and the advantage about C++ is
that it would naturally fit into the existing unit test framework.
There are already some unit testing utilities in Kea, in
src/lib/util/unittests - this would just be another.

> Tomek's major objections to this seem to be:
> 
> 1. We already have tests in Python and they should be portable
> with Python 2.x if people hate Python 3 and its incompatibilities
> 
> 2. If you run unit tests you're probably a developer and you can
> afford installing Python 2.x.
> 
> 3. There is no test framework for shell but there is for C++. This
> is in favor of writing a test in C++.
> 
> 4. We are already using Forge that is written in Python so we could
> move some tests between Kea and Forge.
> 
> And Tomek, has a list of preferred languages for these tests: 1.
> C++ 2. Python 3. Shell

I agree with Tomek here.

One of the problems with BIND 10 was the use of two languages because
few people were masters of both.  Unfortunately, we can't get rid of
Python code altogether: Lettuce is written in Python.  However,
Lettuce is all about writing unit tests in a natural form of English.
To my mind, that far outweighs the inconvenience of maintaining the
Python needed to implement Lettuce features.

> I would really like to have input from others why you like or
> dislike a certain approach because it may affect the way we develop
> similar tests in the future. Apart from 3 test cases we are talking
> about, there will be 6 additional test cases for other components:
> 3 for D2 and 3 for v4. It is better to alter the course now, not
> when they are implemented. Having said that, I should have
> initiated this discussion prior to implementing tests for v6, but
> better now than later.

In summary, I think if we were to implement a C++ class for handling
subprocesses that output to stdout, we could do a lot of this type of
testing in the C++ unit tests.

Stephen