[bind10-dev] Testing interface detection

[Tomek Mrugalski]

On 29.11.2011 11:19, Shane Kerr wrote:
> Tomek mentioned some issues with testing interface detection (currently
> needed by the DHCP side, but it will eventually be needed by the DNS
> side as well).
> 
> I thought it might be worthwhile to discuss here.
> 
> On Mon, 2011-11-28 at 19:56 +0100, Tomek Mrugalski wrote:
>> - IfaceMgr is now part of libdhcp. (It is no longer used by dhcp6 only,
>>   but both dhcp4 and dhcp6).
>> - Added OS detection capability to build system. Some elements may
>>   now be OS-specific (e.g. interface detection).
>> - I have implemented a prototype of Linux interface detection (#1237).
>>   It works (detects interfaces and IPv4 and IPv6 addresses), but the
>>   code needs some work and it is currently completely untested. I just
>>   don't have any good ideas how to test this without reimplementing the
>>   detection itself. I was thinking about comparing detected interfaces
>>   to output of ifconfig (note that ifconfig output syntax is different,
>>   writing tests in a portable way will be tricky).
> 
> This is indeed tricky.
There are several problems.

1. Interface detection is very OS-specific. We can write tests that are
OS-agnostic, but they would not be strictly unit-tests, but rather
component tests, i.e. they would check if interface detection works, not
that each function works properly. Otherwise, we need to implement the
same OS-specific code twice and check that both implementations do the
same. That's way too much overhead and it does not give us  anything
useful. I think that the reasonable approach is that we need to
acknowledge that some aspects are just not testable in automated manner
(or more precisely: the return of investment on work required to
automate it is so minimal that it is not worth the effort). That is a
reasonable assumption that if interface detection works, its component
functions also work.

2. Some tests have to be run on a per-OS basis. Interface detection is a
good example. Due to various reasons, we made a goal of running DHCP
servers on Linux by end of 2011. That means that interface detection
will not work on other OSes. Currently we have a stub detection that
reads interface info from a text file. Non-Linux systems will use that
stub for a while.

3. Some tests need root privileges. Changing interface and verifying
that it was detected is a notable example. This has a problem of
requiring a test machine to have a spare interface that could be messed
up with. This is not as simple as it looks. Many OSes drop routing
entries for interface that goes down. This means that routing table has
to be saved somehow. If there are any existing connections to a machine
during a test, they will be dropped. If the test crashes during
interface-down phase, we may be out of luck. That's why a dedicated
interface would be preferred.

Alternative approach is to try to use something that is common on all
OSes. But no such thing exists as far as I know. I used to think that
loopback interface is such a thing, but it is called lo in Linux and lo0
in BSD systems. There are even some messy systems that don't have
loopback at all (Windows, I'm looking at you). And there may be side
effects if test just brings loopback down.

Also see below for Shane's comment about 2nd style of tests.

> I think writing non-portable tests is probably the way to go. It should
> not be too bad, since the interface detection is also inherently
> non-portable. Your idea of using ifconfig output makes sense to me.
That is a compromise between amount of time invested and simplicity. My
idea is to run "ifconfig -a" (I think -a is supported on all systems)
and then check out if detected interfaces by B10 code are indeed
reported by ifconfig. As ifconfig output varies from OS to OS, that
comparison will be rather dumb - perhaps a substring search? Or maybe
ifconfig output could be split into sections specific for each interface
and then addresses could also be verified?

Alternatively, for each interface detected by our code, "ifconfig
name-of-the-interface" should return list of configured addresses. It
should return error for non-existing interfaces. We will be able to test
interface names, assigned addresses and perhaps link-layer addresses. I
don't we will be able to parse flags or ifindex.

That will still require some hacks. For example Linux returns that lo
has MAC address 00:00:00:00:00:00. Of course ifconfig filters that data
and does not print MAC address at all for loopback. Perhaps our code
should do the same for loopback interface.

> Perhaps we can have 2 different styles of test:
> 
>      1. Something akin to a unit tests, that simply detects interfaces
>         and confirms that the ifconfig output matches.
Agree. That was my idea. This test could be generic and be run on all
OSes that have interface detection implemented. That would be Linux only
for now, but once detection on BSD is implemented, that could cover
FreeBSD, NetBSD and OpenBSD. (There's common code in Dibbler that run on
all 3 OSes, so that is very doable).

>      2. A test that actually changes the interface status and checks
>         that the detection matches.
> 
> The 2nd test would probably be something that requires root permissions,
> so not run by default. We could run it when porting to a new system, or
> if we have errors reported by users. We would probably enable it on our
> build farms as well, to detect regressions.
Perhaps we need a way to designate that some tests require root
permission? b10-dhcp6 and b10-dhcp4 components both open privileged
sockets, so they need root access. Even when we migrate to socket
creator, we will still need root permission.

> The idea of the 2nd test also allows us to test the system behavior when
> interfaces come online or go offline during runtime. This is something
> we want to handle gracefully at some point.
That's true. Although mainly useful from dhcp client perspective
(interface went up, let's configure it), that may be used on server side
as well. Currently the code detects existing state of interfaces and
does not monitor it. Once we decide that it is not enough, we can
periodically pool and check if there are any differences.

In theory it is also possible to monitor link state changes, but it is a
messy area. I don't remember the exact details, but I read about 4
different techniques to monitor or detect link changes *in Linux only*.

On a related note, my changes to configure.ac about OS detection are
generic and may be used by other components, not just DHCP. I'll ask for
a broader review once the code reaches review stage.

Tomek