How to implement SIGHUP tests

Marcin Siodelski marcin at isc.org
Tue May 27 14:19:06 UTC 2014



On 27/05/14 15:59, Tomek Mrugalski wrote:
> On 27.05.2014 11:19, Marcin Siodelski wrote:
>> So, the test presented above requires spawning a new process, sending
>> signals and monitoring whether it reacts in an expected way. A couple of
>> reasons why I think it should be written in shell (not in C++, not in
>> Python):
>>
>> 1. Shell is there on every OS and doesn't cause any additional
>> dependencies for Kea to be tested, whilst this is not true for Python.
>> Even if we can detect that there is no Python on the system and not run
>> these tests, the down side is that we don't run these tests.
> True, but not shells are equal. Most linuxes use bash, but some BSDs use
> tcsh, zsh, ash or one of several other shells. Sure, we can dodge the
> problem by declaring that our environment requires bash to run, but this
> is similar to declaring that we require specific version of python.

Yes, there are differences between shells. But I had this in mind
writing unit tests that you reviewed and it didn't blow up anywhere yet.

> 
>> 2. One of the tasks for 0.9 is to implement a script that controls the
>> startup of Kea. We might not have decided on what language it will be
>> written in, but I also don't think anybody wanted Python as we are in
>> the process of removing Python-based code from Kea. The shell script
>> makes a lot of sense to me and if so, I would like to test Kea startup
>> and reconfiguration using the same approach as this script would use.
>>
>> 3. As Tomek pointed out, the shell scripts don't provide any framework
>> for testing (nothing similar to gtest) which makes it more reasonable to
>> implement our tests in C++ and within gtest framework. But, python based
>> tests that we already have don't have any framework either and we used
>> to live with it. So, is lack of framework really a problem?
> It's not a big problem. I think that over time, our build farm should
> start collecting information about number of unit-tests being executed.
> Let's assume that a commit broke something in configure.ac and
> --with-gtest and --with-gtest-source switches no longer work. Our
> current build farm wouldn't pick it. make & make check would complete
> just fine. As I said, it's not a big issue, just something we should
> keep in mind. The code you wrote in dhcp6_test_func.sh could be a start
> of such framework. It doesn't have to be fancy: just uniform printing of
> test name and test result will do the trick. The code currently does
> that. The (minor) problem is that there's no way to enforce that
> behavior in shell tests.

Shell allows for libraries. The proposed tests use a common shell
function to print test name. The function can be tweaked at any point to
make it uniform with whatever format we want.

> 
>> 4. Implementing tests in C++ would certainly make tests consistent with
>> other tests we have and I am in general in favor of that. But, C++ is
>> not the easiest environment to deal with the type of tests we are
>> talking about. Dealing with spawning background processes, grepping log
>> files, creating configuration files is something that is natural in
>> shell because shell is basically for this purpose mainly. I don't
>> disagree that it is doable in C++: you can fork, execlp, read file in
>> C++ (harder to grep but doable) but it doesn't make code implementation
>> and maintenance any easier.
>>
>> Tomek's major objections to this seem to be:
>>
>> 1. We already have tests in Python and they should be portable with
>> Python 2.x if people hate Python 3 and its incompatibilities
> As I said before, this is a somewhat weak argument. We already have 7
> python tests for v4 and 7 for v6. Regardless if we decide to tweak it to
> run on python 2.x or rewrite them in shell, we'll have to spend some
> time on it.

I was under the impression that we wanted to remove python from Kea.
This means to me that these tests should be replaced anyway. But, maybe
I was wrong. Don't we really want to remove python dependency from Kea
completely? What are the pieces that we MUST keep?

> 
>> 2. If you run unit tests you're probably a developer and you can afford
>> installing Python 2.x.
> I think this is quite important point. The fuss about python3 nonsense
> was mostly from users perspective. It is obvious that developer-specific
> features (like tests) has (and will have) extra dependencies.
> 

I disagree. I don't know the developer's who would like to install more
than necessary.

>> 4. We are already using Forge that is written in Python so we could move
>> some tests between Kea and Forge.
> I think that over time some of the more advanced "unit" tests will no
> longer be unit-tests, they'll just become full featured system tests. If
> you don't think this is the case, please check the code that starts the
> server with various parameters. The unit here is the DHCPv6 server.

I know there are such tests. I believe they are important. But, I'd
rather see them written in shell.

> 
>> And Tomek, has a list of preferred languages for these tests:
>> 1. C++
>> 2. Python
>> 3. Shell
> That's what I initially wrote, but after giving it more thought I'd like
> to amend that list:
> 1. C++
> 2. Some scripting language (one of: Python or shell)
> 
> I marginally prefer python to shell, as it gives you more flexibility
> (e.g. when you want to generate something, like a config with 1000
> subnets). Sure, you can do it in shell, but it's just easier in python.
> But I won't object is we decide to write some tests in shell. But I will
> strongly insist that we should not mix them (some tests in shell and
> some in python). If we chose python, I'm 100% that it must not require 3.x.

I was rather thinking to migrate out of python completely. But, it seems
that there are bits that we want to keep in Python. I am just not sure
which bits and why.

> 
> I think that python has additional benefit. Forge is written in Python.
> So if a unit-test grow too large, we can move it to Forge. Also, Wlodek
> is quite skilled with python, so we could ask him one day to help us
> with unit-tests.

There is a set of unit tests that should always be provided along with
the product (Kea). I don't see any point in moving tests around between
one project and another. Forge as a whole is a great tool that allows
remote execution, crafting packets etc. but all this is not necessary to
test DHCP daemons startup etc. I don't see any reason why such tests
should be moved outside of Kea. They test very basic functionality.

Also, such decisions shouldn't be made on the assumption that someone
feels better in whatever language, because people move around.

> 
> As for Tom's comment that we may do something in 0.9 timeframe and then
> possibly revisit our decision later... well, I don't buy it. There will
> never be a time when we would say "ok, let's spend 2 weeks rewriting our
> unit-tests, because we'd like to have them in a different language".
> There will always be tasks that give you better bang for your buck.

I think this is correct. If we want to move to something else, let's
spend some extra time on this now.

> 
> But I'm not insisting on it. If we collectively decide that shell is the
> way to go, we'll just have a bit more python crap to get rid of :)
> 

I am not sure who is going to have final word on this.

Marcin


More information about the kea-dev mailing list