How to implement SIGHUP tests

[Tomek Mrugalski]

On 27.05.2014 11:19, Marcin Siodelski wrote:
> So, the test presented above requires spawning a new process, sending
> signals and monitoring whether it reacts in an expected way. A couple of
> reasons why I think it should be written in shell (not in C++, not in
> Python):
> 
> 1. Shell is there on every OS and doesn't cause any additional
> dependencies for Kea to be tested, whilst this is not true for Python.
> Even if we can detect that there is no Python on the system and not run
> these tests, the down side is that we don't run these tests.
True, but not shells are equal. Most linuxes use bash, but some BSDs use
tcsh, zsh, ash or one of several other shells. Sure, we can dodge the
problem by declaring that our environment requires bash to run, but this
is similar to declaring that we require specific version of python.

> 2. One of the tasks for 0.9 is to implement a script that controls the
> startup of Kea. We might not have decided on what language it will be
> written in, but I also don't think anybody wanted Python as we are in
> the process of removing Python-based code from Kea. The shell script
> makes a lot of sense to me and if so, I would like to test Kea startup
> and reconfiguration using the same approach as this script would use.
> 
> 3. As Tomek pointed out, the shell scripts don't provide any framework
> for testing (nothing similar to gtest) which makes it more reasonable to
> implement our tests in C++ and within gtest framework. But, python based
> tests that we already have don't have any framework either and we used
> to live with it. So, is lack of framework really a problem?
It's not a big problem. I think that over time, our build farm should
start collecting information about number of unit-tests being executed.
Let's assume that a commit broke something in configure.ac and
--with-gtest and --with-gtest-source switches no longer work. Our
current build farm wouldn't pick it. make & make check would complete
just fine. As I said, it's not a big issue, just something we should
keep in mind. The code you wrote in dhcp6_test_func.sh could be a start
of such framework. It doesn't have to be fancy: just uniform printing of
test name and test result will do the trick. The code currently does
that. The (minor) problem is that there's no way to enforce that
behavior in shell tests.

> 4. Implementing tests in C++ would certainly make tests consistent with
> other tests we have and I am in general in favor of that. But, C++ is
> not the easiest environment to deal with the type of tests we are
> talking about. Dealing with spawning background processes, grepping log
> files, creating configuration files is something that is natural in
> shell because shell is basically for this purpose mainly. I don't
> disagree that it is doable in C++: you can fork, execlp, read file in
> C++ (harder to grep but doable) but it doesn't make code implementation
> and maintenance any easier.
> 
> Tomek's major objections to this seem to be:
> 
> 1. We already have tests in Python and they should be portable with
> Python 2.x if people hate Python 3 and its incompatibilities
As I said before, this is a somewhat weak argument. We already have 7
python tests for v4 and 7 for v6. Regardless if we decide to tweak it to
run on python 2.x or rewrite them in shell, we'll have to spend some
time on it.

> 2. If you run unit tests you're probably a developer and you can afford
> installing Python 2.x.
I think this is quite important point. The fuss about python3 nonsense
was mostly from users perspective. It is obvious that developer-specific
features (like tests) has (and will have) extra dependencies.

> 4. We are already using Forge that is written in Python so we could move
> some tests between Kea and Forge.
I think that over time some of the more advanced "unit" tests will no
longer be unit-tests, they'll just become full featured system tests. If
you don't think this is the case, please check the code that starts the
server with various parameters. The unit here is the DHCPv6 server.

> And Tomek, has a list of preferred languages for these tests:
> 1. C++
> 2. Python
> 3. Shell
That's what I initially wrote, but after giving it more thought I'd like
to amend that list:
1. C++
2. Some scripting language (one of: Python or shell)

I marginally prefer python to shell, as it gives you more flexibility
(e.g. when you want to generate something, like a config with 1000
subnets). Sure, you can do it in shell, but it's just easier in python.
But I won't object is we decide to write some tests in shell. But I will
strongly insist that we should not mix them (some tests in shell and
some in python). If we chose python, I'm 100% that it must not require 3.x.

I think that python has additional benefit. Forge is written in Python.
So if a unit-test grow too large, we can move it to Forge. Also, Wlodek
is quite skilled with python, so we could ask him one day to help us
with unit-tests.

As for Tom's comment that we may do something in 0.9 timeframe and then
possibly revisit our decision later... well, I don't buy it. There will
never be a time when we would say "ok, let's spend 2 weeks rewriting our
unit-tests, because we'd like to have them in a different language".
There will always be tasks that give you better bang for your buck.

But I'm not insisting on it. If we collectively decide that shell is the
way to go, we'll just have a bit more python crap to get rid of :)

Tomek