[bind10-dev] Parallel tests?

[Michal 'vorner' Vaner]

Hello

Our tests run longer and longer, mostly because sleeps in them. I don't think we
can get rid of them completely, but waiting for the tests to finish is something
I don't really like and it slows the development down.

So, I ask, would it make sense to try and run the tests in parallel?

Running the actual tests in parallel in the test binary could be done (gtest
seem to note something about it), but it doesn't help much, since there are only
few sleeps in each binary, but are in many binaries.

Parallel make won't work, because auto* tools don't like to run parallel across
directory boundaries and we have our tests (and sleeps in them) scattered all
over the place.

What I propose is creating a „manifest“ file in each test directory. It would
only list which test binaries are in the test directory and which of them can
run in parallel with other tests. Then there would be a simple script (either
shell or python), which would:
• Find all the manifests
• Run the tests (most of them in parallel, then the sequential ones which don't
  like being run together with something else). It would store their outputs in
  memory and output them one by one, so they don't mix into each other.
• It could make sure if anything fails, the failing binary's output is the last
  one output, so it is easier to read the problem.

It should probably have a way to run only in some subdirectory, called from
within the make check in a directory (when developing, I like to run only the
local tests).

Do you think something like this makes sense? Any comments? Should I create a
ticket for it?

Thank you

-- 
In the name of kernel, compiler and holy penguin

Michal 'vorner' Vaner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind10-dev/attachments/20110713/807398ab/attachment.bin>