<div dir="ltr">Hello again,<div><br></div><div>today I reinstalled bind 9.9.5 without threads, but i still have the same behavior.</div><div><br></div><div>My environment was rhel v6.4, bind 9.9.3-P3 and samba v4.1.</div><div>
I upgraded to rhel v6.5, bind 9.9.5 and samba v4.1.5 but the problem remains:</div><div><br></div><div>after a while bind 9 becomes unresponsive.</div><div><br></div><div>I was not able to use gdb, probably I can do it on Monday, in the meanwhile this is my named.conf, named -V and pstack output captured the moment bind was unresponsive.</div>
<div><br></div><div><div>named-checkconf -px</div><div>================</div><div>options {</div><div><span class="" style="white-space:pre"> </span>directory "/var/named";</div><div><span class="" style="white-space:pre"> </span>dump-file "/var/named/data/cache.dump.db";</div>
<div><span class="" style="white-space:pre"> </span>listen-on port 53 {</div><div><span class="" style="white-space:pre"> </span>"any";</div><div><span class="" style="white-space:pre"> </span>};</div><div><span class="" style="white-space:pre"> </span>listen-on-v6 {</div>
<div><span class="" style="white-space:pre"> </span>"none";</div><div><span class="" style="white-space:pre"> </span>};</div><div><span class="" style="white-space:pre"> </span>statistics-file "/var/named/data/named.stats.txt";</div>
<div><span class="" style="white-space:pre"> </span>tkey-gssapi-keytab "/usr/local/samba/private/dns.keytab";</div><div><span class="" style="white-space:pre"> </span>version "Bind";</div><div><span class="" style="white-space:pre"> </span>allow-recursion {</div>
<div><span class="" style="white-space:pre"> </span><a href="http://10.0.0.0/8">10.0.0.0/8</a>;</div><div><span class="" style="white-space:pre"> </span><a href="http://10.1.4.0/24">10.1.4.0/24</a>;</div><div><span class="" style="white-space:pre"> </span>"localhost";</div>
<div><span class="" style="white-space:pre"> </span>"localnets";</div><div><span class="" style="white-space:pre"> </span>};</div><div><span class="" style="white-space:pre"> </span>auth-nxdomain yes;</div><div>
<span class="" style="white-space:pre"> </span>empty-zones-enable no;</div><div><span class="" style="white-space:pre"> </span>max-cache-size 4294967296;</div><div><span class="" style="white-space:pre"> </span>allow-query {</div>
<div><span class="" style="white-space:pre"> </span>"any";</div><div><span class="" style="white-space:pre"> </span>};</div><div><span class="" style="white-space:pre"> </span>allow-transfer {</div><div><span class="" style="white-space:pre"> </span>"none";</div>
<div><span class="" style="white-space:pre"> </span>};</div><div><span class="" style="white-space:pre"> </span>allow-update {</div><div><span class="" style="white-space:pre"> </span><a href="http://127.0.0.1/32">127.0.0.1/32</a>;</div>
<div><span class="" style="white-space:pre"> </span>"localhost";</div><div><span class="" style="white-space:pre"> </span><a href="http://10.0.0.0/8">10.0.0.0/8</a>;</div><div><span class="" style="white-space:pre"> </span>};</div>
<div><span class="" style="white-space:pre"> </span>forwarders {</div><div><span class="" style="white-space:pre"> </span>8.8.8.8;</div><div><span class="" style="white-space:pre"> </span>8.8.4.4;</div><div><span class="" style="white-space:pre"> </span>};</div>
<div><span class="" style="white-space:pre"> </span>notify no;</div><div>};</div><div>controls {</div><div><span class="" style="white-space:pre"> </span>inet 127.0.0.1 port 953 allow {</div><div><span class="" style="white-space:pre"> </span><a href="http://127.0.0.1/32">127.0.0.1/32</a>;</div>
<div><span class="" style="white-space:pre"> </span>} keys {</div><div><span class="" style="white-space:pre"> </span>"domain1";</div><div><span class="" style="white-space:pre"> </span>};</div><div>};</div><div>
acl "trusted" {</div><div><span class="" style="white-space:pre"> </span><a href="http://10.1.4.0/24">10.1.4.0/24</a>;</div><div><span class="" style="white-space:pre"> </span><a href="http://127.0.0.1/32">127.0.0.1/32</a>;</div>
<div>};</div><div>logging {</div><div><span class="" style="white-space:pre"> </span>channel "basic" {</div><div><span class="" style="white-space:pre"> </span>file "/var/log/named/named.log" versions 3 size 20971520;</div>
<div><span class="" style="white-space:pre"> </span>severity debug 1;</div><div><span class="" style="white-space:pre"> </span>print-time yes;</div><div><span class="" style="white-space:pre"> </span>print-severity yes;</div>
<div><span class="" style="white-space:pre"> </span>print-category yes;</div><div><span class="" style="white-space:pre"> </span>};</div><div><span class="" style="white-space:pre"> </span>category "default" {</div>
<div><span class="" style="white-space:pre"> </span>"basic";</div><div><span class="" style="white-space:pre"> </span>};</div><div>};</div><div>statistics-channels {</div><div><span class="" style="white-space:pre"> </span>inet 0.0.0.0 port 8060 allow {</div>
<div><span class="" style="white-space:pre"> </span>"trusted";</div><div><span class="" style="white-space:pre"> </span>};</div><div>};</div><div>key "domain1" {</div><div><span class="" style="white-space:pre"> </span>algorithm "hmac-md5";</div>
<div><span class="" style="white-space:pre"> </span>secret "????????????????????????????????????????????";</div><div>};</div><div>zone "." {</div><div><span class="" style="white-space:pre"> </span>type hint;</div>
<div><span class="" style="white-space:pre"> </span>file "named.root";</div><div>};</div><div>zone "localhost" {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "master/localhost.zone";</div>
<div>};</div><div>zone "0.0.127.in-addr.arpa" {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "master/0.0.127.zone";</div>
<div>};</div><div>zone "ait.nkm" IN {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "dynamic/fz.db.ait.nkm";</div><div>
};</div><div>zone "aka.nkm" IN {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "dynamic/fz.db.aka.nkm";</div><div>};</div>
<div>zone "axa.nkm" IN {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "dynamic/fz.db.axa.nkm";</div><div>};</div><div>
zone "2.1.10.in-addr.arpa" IN {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "dynamic/rz.db.2.1.10";</div><div>};</div>
<div>zone "7.1.10.in-addr.arpa" IN {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "dynamic/rz.db.7.1.10";</div><div>
};</div><div>zone "6.1.10.in-addr.arpa" IN {</div><div><span class="" style="white-space:pre"> </span>type master;</div><div><span class="" style="white-space:pre"> </span>file "dynamic/rz.db.6.1.10";</div>
<div>};</div><div>dlz "AD DNS Zone" {</div><div><span class="" style="white-space:pre"> </span>database "dlopen /usr/local/samba/lib/bind9/dlz_bind9_9.so -d 3";</div><div>};</div><div>server ::/0 {</div>
<div><span class="" style="white-space:pre"> </span>bogus yes;</div><div>};</div></div><div><br></div><div>named -V</div><div>=======</div><div><div><div>BIND 9.9.5 (Extended Support Version) <id:f9b8a50e> built by make with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-libtool' '--localstatedir=/var' '--enable-ipv6' '--with-pic' '--disable-static' '--disable-openssl-version-check' '--with-dlz-ldap=yes' '--with-dlz-filesystem=yes' '--with-gssapi=/usr/include/gssapi' '--with-dlopen=yes' '--with-docbook-xsl=/usr/share/sgml/docbook/xsl-stylesheets' '--enable-fixed-rrset' '--enable-rrl' '--enable-newstats' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu'</div>
<div>compiled by GCC 4.4.7 20120313 (Red Hat 4.4.7-4)</div><div>using OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013</div><div>using libxml2 version: 2.7.6</div></div></div><div><br></div><div><div>pstack when named was unresponsive</div>
<div>============================</div><div>#0 0x00000039bb80e975 in fcntl () from /lib64/libpthread.so.0</div><div>#1 0x00007fc753c1b347 in fcntl_lock () from /usr/local/samba/lib/private/libtdb.so.1</div><div>#2 0x00007fc753c1b44f in tdb_brlock () from /usr/local/samba/lib/private/libtdb.so.1</div>
<div>#3 0x00007fc753c1b919 in tdb_nest_lock () from /usr/local/samba/lib/private/libtdb.so.1</div><div>#4 0x00007fc753c1bf2c in tdb_transaction_lock () from /usr/local/samba/lib/private/libtdb.so.1</div><div>#5 0x00007fc753c213be in _tdb_transaction_start () from /usr/local/samba/lib/private/libtdb.so.1</div>
<div>#6 0x00007fc753c216d3 in tdb_transaction_start () from /usr/local/samba/lib/private/libtdb.so.1</div><div>#7 0x00007fc74dafa321 in partition_metadata_start_trans () from /usr/local/samba/lib/ldb/partition.so</div><div>
#8 0x00007fc74daf5f1f in partition_start_trans () from /usr/local/samba/lib/ldb/partition.so</div><div>#9 0x00007fc7594586e1 in ldb_next_start_trans () from /usr/local/samba/lib/private/libldb.so.1</div><div>#10 0x00007fc74ed23925 in linked_attributes_start_transaction () from /usr/local/samba/lib/ldb/linked_attributes.so</div>
<div>#11 0x00007fc7594586e1 in ldb_next_start_trans () from /usr/local/samba/lib/private/libldb.so.1</div><div>#12 0x00007fc74d2d40c3 in replmd_start_transaction () from /usr/local/samba/lib/ldb/repl_meta_data.so</div><div>
#13 0x00007fc7594586e1 in ldb_next_start_trans () from /usr/local/samba/lib/private/libldb.so.1</div><div>#14 0x00007fc750b83b12 in descriptor_start_transaction () from /usr/local/samba/lib/ldb/descriptor.so</div><div>#15 0x00007fc7594586e1 in ldb_next_start_trans () from /usr/local/samba/lib/private/libldb.so.1</div>
<div>#16 0x00007fc74ba7f70f in schema_load_start_transaction () from /usr/local/samba/lib/ldb/schema_load.so</div><div>#17 0x00007fc759475eeb in ldb_transaction_start () from /usr/local/samba/lib/private/libldb.so.1</div>
<div>#18 0x00007fc759d12ea9 in dlz_newversion () from /usr/local/samba/lib/bind9/dlz_bind9_9.so</div><div>#19 0x00000000004720cd in dlopen_dlz_newversion ()</div><div>#20 0x00007fc75ac628ec in newversion () from /usr/lib64/libdns.so.100</div>
<div>#21 0x000000000045913f in update_action ()</div><div>#22 0x00007fc75a2d94eb in isc__taskmgr_dispatch () from /usr/lib64/libisc.so.95</div><div>#23 0x00007fc75a2deedb in evloop () from /usr/lib64/libisc.so.95</div><div>
#24 0x00007fc75a2df14a in isc__app_ctxrun () from /usr/lib64/libisc.so.95</div><div>#25 0x00000000004273cc in main ()</div></div><div><br></div><div>pstack when named was working</div><div>========================</div><div>
<div>#0 0x00000039bb4e9143 in __epoll_wait_nocancel () from /lib64/libc.so.6</div><div>#1 0x00007fc61b96bd0f in isc__socketmgr_waitevents () from /usr/lib64/libisc.so.95</div><div>#2 0x00007fc61b962eaf in evloop () from /usr/lib64/libisc.so.95</div>
<div>#3 0x00007fc61b96314a in isc__app_ctxrun () from /usr/lib64/libisc.so.95</div><div>#4 0x00000000004273cc in main ()</div></div><div><br></div><div>Thanks for your time</div><div><br></div><div>Nikos Mitas</div><div>
<br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-03-12 22:37 GMT+02:00 Evan Hunt <span dir="ltr"><<a href="mailto:each@isc.org" target="_blank">each@isc.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On Wed, Mar 12, 2014 at 10:09:44PM +0200, Nikos Mitas wrote:<br>
> at this point, nslookup and rndc are not working, the only option i have is<br>
> to kill samba and named and start over. But after a while the problem<br>
> repeats.<br>
><br>
> any help will be very welcome<br>
<br>
</div>Offhand I'd guess it's a deadlock. Does it happen if named is built<br>
without threads? Can you attach to the process with gdb and get a<br>
backtrace?<br>
<br>
You can submit a bug report against BIND at <a href="mailto:bind9-bugs@isc.org">bind9-bugs@isc.org</a> -- if<br>
you do, please include as much information as possible about the system<br>
you're running on, how you configured the built ("named -V" will tell<br>
you this), and your named.conf ("named-checkconf -px" dumps a copy of<br>
your configuration with key secrets omitted).<br>
<br>
If the problem's in the samba DLZ module, I probably won't be able to<br>
help you, but if it's in the dlopen driver, perhaps I can.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Evan Hunt -- <a href="mailto:each@isc.org">each@isc.org</a><br>
Internet Systems Consortium, Inc.<br>
</font></span></blockquote></div><br></div>