How to implement SIGHUP tests

Thomas Markwalder tmark at isc.org
Tue May 27 11:11:31 UTC 2014


On 5/27/14, 5:19 AM, Marcin Siodelski wrote:
> Hi,
>
> In the course of preparing Kea 0.9 release Tomek has implemented the
> switchable configuration backends. Currently there are two: Bundy - the
> one that allows to configure Kea using the former BIND10 infrastructure
> and JSON, which allows specifying a configuration file (with a full
> configuration) from the command line and run Kea standalone.
>
> In order to preserve the ability to dynamically reconfigure the DHCP
> server in case of use of the JSON backend, we decided that we had to
> implement SIGHUP signal handler for all three Kea processes: DHCPv4 &
> DHCPv6 server and DDNS module. The handler should trigger process
> reconfiguration when the SIGHUP is received.
>
> The ticket #3406 introduces SIGHUP handler for the DHCPv6 server
> process. It also introduces a handler for SIGINT and SIGTERM which shuts
> down the server gracefully.
>
> The changes in the server's logic are pretty straightforward. The more
> involved part of this ticket is about testing the change. I think there
> is no argument about the test logic itself, but during ticket's review
> Tomek raised concerns about the language that I've picked: shell script.
> We both have strong opinions about it and it seems we reached the point
> when it is useful to have input from other people.
>
> Please see comments in the #3406 and code on trac3406 branch for details.
>
> One of the tests I have implemented follows this logic (other tests
> follow similar logic but check if appropriate messages are logged when
> the SIGTERM or SIGINT is received and whether he process really shutdown):
> - Create a file with the full Kea6 configuration
> - Set logging to a file by setting the environmental variable
> - Start Kea6 process in background using a config file specified with a
> command line option
> - Wait for a Kea6 process to start by monitoring the contents of the log
> file (if anything shows up in the log file) the process has started.
> - Get pid of the Kea6 process
> - Grep the log file for the message which tells that the configuration
> has been successful (guarded with timeout)
> - Create invalid configuration in the file being used by Kea
> - Send SIGHUP signal
> - Monitor the log file to check if the message indicating configuration
> failure occurs (guarded with a timeout)
> - If the configuration failure occurs, proceed to next step
> - Restore a good configuration in the configuration file being used by Kea
> - send SIGHUP signal again
> - Monitor the log file to see if it reports successful reconfiguration
> (guarded by timeout)
> - Kill Kea process
>
> So, the test presented above requires spawning a new process, sending
> signals and monitoring whether it reacts in an expected way. A couple of
> reasons why I think it should be written in shell (not in C++, not in
> Python):
>
> 1. Shell is there on every OS and doesn't cause any additional
> dependencies for Kea to be tested, whilst this is not true for Python.
> Even if we can detect that there is no Python on the system and not run
> these tests, the down side is that we don't run these tests.
>
> 2. One of the tasks for 0.9 is to implement a script that controls the
> startup of Kea. We might not have decided on what language it will be
> written in, but I also don't think anybody wanted Python as we are in
> the process of removing Python-based code from Kea. The shell script
> makes a lot of sense to me and if so, I would like to test Kea startup
> and reconfiguration using the same approach as this script would use.
>
> 3. As Tomek pointed out, the shell scripts don't provide any framework
> for testing (nothing similar to gtest) which makes it more reasonable to
> implement our tests in C++ and within gtest framework. But, python based
> tests that we already have don't have any framework either and we used
> to live with it. So, is lack of framework really a problem?
>
> 4. Implementing tests in C++ would certainly make tests consistent with
> other tests we have and I am in general in favor of that. But, C++ is
> not the easiest environment to deal with the type of tests we are
> talking about. Dealing with spawning background processes, grepping log
> files, creating configuration files is something that is natural in
> shell because shell is basically for this purpose mainly. I don't
> disagree that it is doable in C++: you can fork, execlp, read file in
> C++ (harder to grep but doable) but it doesn't make code implementation
> and maintenance any easier.
>
> Tomek's major objections to this seem to be:
>
> 1. We already have tests in Python and they should be portable with
> Python 2.x if people hate Python 3 and its incompatibilities
>
> 2. If you run unit tests you're probably a developer and you can afford
> installing Python 2.x.
>
> 3. There is no test framework for shell but there is for C++. This is in
> favor of writing a test in C++.
>
> 4. We are already using Forge that is written in Python so we could move
> some tests between Kea and Forge.
>
> And Tomek, has a list of preferred languages for these tests:
> 1. C++
> 2. Python
> 3. Shell
>
> See http://kea.isc.org/ticket/3406#comment:5
>
> I would really like to have input from others why you like or dislike a
> certain approach because it may affect the way we develop similar tests
> in the future. Apart from 3 test cases we are talking about, there will
> be 6 additional test cases for other components: 3 for D2 and 3 for v4.
> It is better to alter the course now, not when they are implemented.
> Having said that, I should have initiated this discussion prior to
> implementing tests for v6, but better now than later.
>
> Thanks,
> Marcin
> _______________________________________________
> kea-dev mailing list
> kea-dev at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-dev
First, to say there's no framework for shell is not particularly
important in my view.  It is entirely possible to create scripts that
provide common functions akin to libraries in C++ or modules in Python. 
All one needs to do is source
these subscript(s) into any other script that needs them.  There is no
reason shell cannot be refactored into common pieces like any other
language.   If there are already common pieces that should be done now.

Shell is terrific for interacting with the OS for manipulating
processes, files, and so forth.   These are the types of things for
which it is intended.   

We will very likely find that much of what needs to be done for these
tests also needs to be done for our nanny process so there will likely
be large chunks that are directly reusable in the nanny.   The converse
would be true so it
could be that after the nanny is created these tests are retrofitted
with bits of that.

In interests of time and getting 0.9 finished I would suggest using
shell for these tests for now and as all of this evolves we can, if we
feel its beneficial, move to a C++ mechanism.    This would be my
recommendation for the nanny process as well.   For 0.9 do it in shell
so we can get it off the ground.   Then we improve it over time either
by refining the shell script of replacing with it a C++ nanny.

Short term and long term I think I would prefer Python the least 
primarily because:

a. there's no direct tie between what we do in Python and C++/gtest world
b. getting Python all squared away for a particular environment always
seems to be an issue.  It is getting more difficult to write Python that
is both 2.6, 2.7 and 3.x compatible (Yes there is a gap between 2.6 and 2.7)
c. The nanny is most assuredly not going to be Python so there would be
no reuse of code or skill
d. I really hate leading white space for logic levels, that's almost as
bad as tabs in make files

If we do it something other than shell, I would say long term do it in C++.



















More information about the kea-dev mailing list