[bind10-dev] failing time related unittests

[Jeremy C. Reed]

Here are the googletest filters I have in place:

NetBSD5-amd64 (on xen):
GTEST_FILTER=-BenchMarkTest.run

FreeBSD8-i386 (on vmware):
TESTS_ENVIRONMENT='GTEST_FILTER=-RecursiveQueryTest.forwardClientTimeout:
RecursiveQueryTest.forwardLookupTimeout:BenchMarkTest.run'

(But never saw RecursiveQueryTest.forwardClientTimeout there?)

FreeBSD8-i386 distcheck (on vmware):
TESTS_ENVIRONMENT='GTEST_FILTER=-RecursiveQueryTest.forwardClientTimeout:
RecursiveQueryTest.forwardLookupTimeout:BenchMarkTest.run:
IntervalTimerTest.overwriteIntervalTimer'

(But never saw RecursiveQueryTest.forwardClientTimeout there?)

FreeBSD8-amd64-GCC (on vmware):
TESTS_ENVIRONMENT='GTEST_FILTER=-BenchMarkTest.run'

But we have related failures on non-virtual machines too. (My logs show 
240 total failures.)

I list all failures below, but prepend an asterisk for those that I 
think are timing related. I put a dollar sign for those that I later 
added filters.

--------
CentOS5-x86_64-GCC	(real hardware)
--------
IOServiceTest.unavailableAddress

--------
Debian5Linux-i686	(real hardware)
--------
*ASIOLinkTest.forwardClientTimeout	(renamed?)
*ASIOLinkTest.forwardLookupTimeout	(renamed?)
*BenchMarkTest.run
DataSrcTest.AuthDelegation
DataSrcTest.Cname
DataSrcTest.CnameExt
DataSrcTest.CnameInt
DataSrcTest.Delegation
DataSrcTest.Dname
DataSrcTest.DS
DataSrcTest.NSQuery
DataSrcTest.Nxdomain
DataSrcTest.NxRRset
DataSrcTest.Query
DataSrcTest.Wildcard
DNSSECTimeTest.toText
*IntervalTimerTest.startIntervalTimer
Rdata_NSEC3_Test.badText
Rdata_NSEC3_Test.createFromWire
Rdata_NSEC3_Test.toText
Rdata_NSEC3_Test.toWireRenderer
Rdata_NSEC_Test.createFromWire_NSEC
*RecursiveQueryTest.forwardLookupTimeout

--------
FreeBSD7-amd64		(this builder no longer exists)
--------
DataSrcTest.AuthDelegation
DataSrcTest.Cname
DataSrcTest.CnameExt
DataSrcTest.CnameInt
DataSrcTest.Delegation
DataSrcTest.Dname
DataSrcTest.DS
DataSrcTest.NSQuery
DataSrcTest.Nxdomain
DataSrcTest.NxRRset
DataSrcTest.Query
DataSrcTest.Wildcard
Rdata_NSEC3_Test.badText
Rdata_NSEC3_Test.createFromWire
Rdata_NSEC3_Test.toText
Rdata_NSEC3_Test.toWireRenderer
Rdata_NSEC_Test.createFromWire_NSEC

--------
FreeBSD8-amd64-GCC	(vmware)
--------
$*BenchMarkTest.run
get_sock.udp6_create

--------
FreeBSD8-i386		(vmware)
--------
*ASIOLinkTest.forwardClientTimeout	(renamed?)
*ASIOLinkTest.forwardLookupTimeout	(renamed?)
$*BenchMarkTest.run
get_sock.udp6_create
$*IntervalTimerTest.overwriteIntervalTimer
*IntervalTimerTest.startIntervalTimer
$*RecursiveQueryTest.forwardLookupTimeout

--------
FreeBSD8-i386-Clang	(vmware)
--------
*ASIOLinkTest.forwardClientTimeout	(renamed?)
*ASIOLinkTest.forwardLookupTimeout	(renamed?)
$*BenchMarkTest.run
get_sock.udp6_create
$*IntervalTimerTest.overwriteIntervalTimer
*IntervalTimerTest.startIntervalTimer
$*RecursiveQueryTest.forwardLookupTimeout

--------
MacOSX10.6-x86_64-Clang		(real hardware)
--------
DNSSECTimeTest.fromText
UDPSocket.SequenceTest

--------
MacOSX10.6-x86_64-GCC		(real hardware)
--------
UDPSocket.SequenceTest

--------
NetBSD5-amd64			(xen)
--------
$*BenchMarkTest.run
get_sock.udp6_create
*IntervalTimerTest.overwriteIntervalTimer
*IntervalTimerTest.startIntervalTimer
IOFetchTest.UdpSendReceive
*RecursiveQueryTest.forwardClientTimeout
*RecursiveQueryTest.forwardLookupTimeout

--------
NetBSD5-i386			(real hardware)
--------
(none, not automated, not enough runs.)

--------
Solaris10-sparc			(real hardware)
(this was renamed to Solaris10-sparc-GCC)
--------
*ASIOLinkTest.forwardClientTimeout	(renamed?)
*ASIOLinkTest.forwardLookupTimeout	(renamed?)
*BenchMarkTest.run
DataSrcTest.Nxdomain
DNSSECTimeTest.toText
Rdata_NSEC3_Test.badText
Rdata_NSEC3_Test.createFromWire
Rdata_NSEC3_Test.toText
Rdata_NSEC3_Test.toWireRenderer

--------
Solaris10-sparc-GCC		(real hardware)
(previously known as Solaris10-sparc)
--------
*ASIOLinkTest.forwardClientTimeout	(renamed?)
*ASIOLinkTest.forwardLookupTimeout	(renamed?)
*BenchMarkTest.run

--------
Solaris10-sparc-Sunstudio	(real hardware)
--------
*BenchMarkTest.run


It was suggested I make a trac ticket for every known test failure so we 
can carefully evaluate.

If a timing test fails, then is it really testing something that may 
cause a failure on a production system?

Another suggestion was to propose minimal hardware we support. I don't 
agree with that based on our existing systems' failures -- because all 
these tests systems are plenty powerful enough to serve probably 99% of 
normal DNS server needs.