How to implement SIGHUP tests

[Marcin Siodelski]

Hi,

In the course of preparing Kea 0.9 release Tomek has implemented the
switchable configuration backends. Currently there are two: Bundy - the
one that allows to configure Kea using the former BIND10 infrastructure
and JSON, which allows specifying a configuration file (with a full
configuration) from the command line and run Kea standalone.

In order to preserve the ability to dynamically reconfigure the DHCP
server in case of use of the JSON backend, we decided that we had to
implement SIGHUP signal handler for all three Kea processes: DHCPv4 &
DHCPv6 server and DDNS module. The handler should trigger process
reconfiguration when the SIGHUP is received.

The ticket #3406 introduces SIGHUP handler for the DHCPv6 server
process. It also introduces a handler for SIGINT and SIGTERM which shuts
down the server gracefully.

The changes in the server's logic are pretty straightforward. The more
involved part of this ticket is about testing the change. I think there
is no argument about the test logic itself, but during ticket's review
Tomek raised concerns about the language that I've picked: shell script.
We both have strong opinions about it and it seems we reached the point
when it is useful to have input from other people.

Please see comments in the #3406 and code on trac3406 branch for details.

One of the tests I have implemented follows this logic (other tests
follow similar logic but check if appropriate messages are logged when
the SIGTERM or SIGINT is received and whether he process really shutdown):
- Create a file with the full Kea6 configuration
- Set logging to a file by setting the environmental variable
- Start Kea6 process in background using a config file specified with a
command line option
- Wait for a Kea6 process to start by monitoring the contents of the log
file (if anything shows up in the log file) the process has started.
- Get pid of the Kea6 process
- Grep the log file for the message which tells that the configuration
has been successful (guarded with timeout)
- Create invalid configuration in the file being used by Kea
- Send SIGHUP signal
- Monitor the log file to check if the message indicating configuration
failure occurs (guarded with a timeout)
- If the configuration failure occurs, proceed to next step
- Restore a good configuration in the configuration file being used by Kea
- send SIGHUP signal again
- Monitor the log file to see if it reports successful reconfiguration
(guarded by timeout)
- Kill Kea process

So, the test presented above requires spawning a new process, sending
signals and monitoring whether it reacts in an expected way. A couple of
reasons why I think it should be written in shell (not in C++, not in
Python):

1. Shell is there on every OS and doesn't cause any additional
dependencies for Kea to be tested, whilst this is not true for Python.
Even if we can detect that there is no Python on the system and not run
these tests, the down side is that we don't run these tests.

2. One of the tasks for 0.9 is to implement a script that controls the
startup of Kea. We might not have decided on what language it will be
written in, but I also don't think anybody wanted Python as we are in
the process of removing Python-based code from Kea. The shell script
makes a lot of sense to me and if so, I would like to test Kea startup
and reconfiguration using the same approach as this script would use.

3. As Tomek pointed out, the shell scripts don't provide any framework
for testing (nothing similar to gtest) which makes it more reasonable to
implement our tests in C++ and within gtest framework. But, python based
tests that we already have don't have any framework either and we used
to live with it. So, is lack of framework really a problem?

4. Implementing tests in C++ would certainly make tests consistent with
other tests we have and I am in general in favor of that. But, C++ is
not the easiest environment to deal with the type of tests we are
talking about. Dealing with spawning background processes, grepping log
files, creating configuration files is something that is natural in
shell because shell is basically for this purpose mainly. I don't
disagree that it is doable in C++: you can fork, execlp, read file in
C++ (harder to grep but doable) but it doesn't make code implementation
and maintenance any easier.

Tomek's major objections to this seem to be:

1. We already have tests in Python and they should be portable with
Python 2.x if people hate Python 3 and its incompatibilities

2. If you run unit tests you're probably a developer and you can afford
installing Python 2.x.

3. There is no test framework for shell but there is for C++. This is in
favor of writing a test in C++.

4. We are already using Forge that is written in Python so we could move
some tests between Kea and Forge.

And Tomek, has a list of preferred languages for these tests:
1. C++
2. Python
3. Shell

See http://kea.isc.org/ticket/3406#comment:5

I would really like to have input from others why you like or dislike a
certain approach because it may affect the way we develop similar tests
in the future. Apart from 3 test cases we are talking about, there will
be 6 additional test cases for other components: 3 for D2 and 3 for v4.
It is better to alter the course now, not when they are implemented.
Having said that, I should have initiated this discussion prior to
implementing tests for v6, but better now than later.

Thanks,
Marcin