Named process suddenly down

Warren Kumari warren at kumari.net
Wed Jan 15 14:21:33 UTC 2014


BIND 9.9.3-P2  is vulnerable to CVE 2014-0591 ("A Crafted Query
Against an NSEC3-signed Zone Can Crash BIND") and CVE 2013-6320 ("A
Winsock API Bug can cause a side-effect affecting BIND ACLs")

While it doesn't look (after a very quick glance) like CVE 2014-0591 (
https://kb.isc.org/article/AA-01078 ) you should probably upgrade to
9.9.4-P2 (Extended Support Version) anyway and see if the issue goes
away. If it doesn't -- hey, at least you know...

W

On Wed, Jan 15, 2014 at 5:18 AM, AOYAGI Takashi
<svu00012 at partner.nri.co.jp> wrote:
> Hello
>
> I have very serious trouble about BIND.
> And we are in a hurry, so please help us.
>
> Enveroment:
> "BIND 9.9.3-P2 (Extended Support Version)".
>
>
> Here is situation:
>
> Named process suddenly down at 23 Dec 19:58, and the core was outputted.
> In addition to this, same kind of problem had occurred in several other servers.
> Therefore, I guess this problem is not coming from hardware problem which is like memory trouble
> or others. Because this incident happened on sevral servers at same time.
>
> I put some information(/var/log/message and core and some other logs) which may indicate the problem, below.
> So, please someone who could tell us about the following three questions from our given information.
>
>
> 1) The cause of downed process.
> 2) This is an discovered issue or undiscoverd?
> 3) Is there workaround corresponding to this?
>
>
> And any advise is welcome.
>
> Sincerely,
> AOYAGI
>
>
> messages
> ############################################################################
> Dec 23 19:58:13 hogedns021 kernel: named[1844]: segfault at 0 ip 000000000048fbb
> a sp 00007f6921778840 error 4 in named[400000+303000]
> Dec 23 19:58:15 hogedns021 abrtd: Directory 'ccpp-2013-12-23-19:58:13-1843' crea
> tion detected
> Dec 23 19:58:15 hogedns021 abrt[19323]: Saved core dump of pid 1843 (/usr/local/
> sbin/named) to /var/spool/abrt/ccpp-2013-12-23-19:58:13-1843 (166387712 bytes)
> Dec 23 19:58:15 hogedns021 abrtd: Executable '/usr/local/sbin/named' doesn't bel
> ong to any package
> Dec 23 19:58:15 hogedns021 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-12
> -23-19:58:13-1843' exited with 1
> Dec 23 19:58:15 hogedns021 abrtd: Corrupted or bad directory /var/spool/abrt/ccp
> p-2013-12-23-19:58:13-1843, deleting
> ############################################################################
>
> core info
> ############################################################################
> [root at hogedns021 log]# named -V
> BIND 9.9.3-P2 (Extended Support Version) <id:d8a6fe8b> built with '--prefix=/usr
> /local/' '--disable-openssl-version-check' '--enable-filter-aaaa' '--enable-thre
> ads' '--with-gssapi=no' 'CFLAGS=-DDIG_SIGCHASE'
> using OpenSSL version: OpenSSL 1.0.0 29 Mar 2010
> [root at hogedns021 log]#uname -a
> Linux hogedns021 2.6.32-279.19.1.el6.x86_64 #1 SMP Sat Nov 24 14:35:28 EST 2012
> x86_64 x86_64 x86_64 GNU/Linux
> [root at hogedns021 log]# ls -la /var/named/chroot/var/named/core.1843
> -rw------- 1 dns dns 166387712 Dec 23 19:58 2013 /var/named/chroot/var/named/co
> re.1843
> [root at hogedns021 log]# file /var/named/chroot/var/named/core.1843
> /var/named/chroot/var/named/core.1843: ELF 64-bit LSB core file x86-64, version
> 1 (SYSV), SVR4-style, from '/usr/local/sbin/named -u dns -c /etc/named.conf -t /
> var/named/chroot'
> ############################################################################
>
> gdb back trace("thread apply all bt")
> ############################################################################
> [root at hogedns021 tmp]# ldd /usr/local/sbin/named
>         linux-vdso.so.1 =>  (0x00007fffefdff000)
>         libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x0000003a0c800000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x0000003a0ac00000)
>         libcap.so.2 => /lib64/libcap.so.2 (0x0000003a11400000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003a0b000000)
>         libc.so.6 => /lib64/libc.so.6 (0x0000003a0a800000)
>         libz.so.1 => /lib64/libz.so.1 (0x0000003a0b400000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000003a0a400000)
>         libattr.so.1 => /lib64/libattr.so.1 (0x0000003a0f000000)
>
> [root at hogedns021 sue]# ls -la
> 合計 170176
> drwxr-xr-x 2 root root      4096 Dec 24 11:36 .
> drwxrwxrwt 5 root root      4096 Dec 24 11:36 ..
> -rw------- 1 dns  dns  166387712 Dec 23 19:58 core.1843
> -rwxr-xr-x 1 root root    156872 Oct 12  2012 ld-linux-x86-64.so.2
> -rwxr-xr-x 1 root root     21152 Aug  8  2011 libattr.so.1
> -rwxr-xr-x 1 root root   1922112 Oct 12  2012 libc.so.6
> -rwxr-xr-x 1 root root     19016 Aug 23  2011 libcap.so.2
> -rwxr-xr-x 1 root root   1665328 Aug 16  2012 libcrypto.so.10
> -rwxr-xr-x 1 root root     22536 Oct 12  2012 libdl.so.2
> -rw-r--r-- 1 root root     65928 Dec 24 11:35 libnss_files.so.2
> -rwxr-xr-x 1 root root    145720 Oct 12  2012 libpthread.so.0
> -rwxr-xr-x 1 root root     34008 Dec 24 10:55 libthread_db-1.0.so
> -rwxr-xr-x 1 root root     90952 Aug 10  2011 libz.so.1
> -rwxr-xr-x 1 root root   3491405 Aug  9 00:28 named
>
>
> [root at hogedns021 sue]# gdb
> GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> (gdb) set solib-absolute-prefix /tmp/sue
> (gdb) set solib-search-path /tmp/sue
> (gdb) file /tmp/sue/named
> Reading symbols from /tmp/sue/named...(no debugging symbols found)...done.
> (gdb) core-file /tmp/sue/core.1843
> [New Thread 1844]
> [New Thread 1843]
> [New Thread 1847]
> [New Thread 1846]
> [New Thread 1845]
> Reading symbols from /tmp/sue/libcrypto.so.10...(no debugging symbols found)...done.
> Loaded symbols for /tmp/sue/libcrypto.so.10
> Reading symbols from /tmp/sue/libdl.so.2...(no debugging symbols found)...done.
> Loaded symbols for /tmp/sue/libdl.so.2
> Reading symbols from /tmp/sue/libcap.so.2...(no debugging symbols found)...done.
> Loaded symbols for /tmp/sue/libcap.so.2
> Reading symbols from /tmp/sue/libpthread.so.0...(no debugging symbols found)...done.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Loaded symbols for /tmp/sue/libpthread.so.0
> Reading symbols from /tmp/sue/libc.so.6...(no debugging symbols found)...done.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Loaded symbols for /tmp/sue/libc.so.6
> Reading symbols from /tmp/sue/libz.so.1...(no debugging symbols found)...done.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Loaded symbols for /tmp/sue/libz.so.1
> Reading symbols from /tmp/sue/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Loaded symbols for /tmp/sue/ld-linux-x86-64.so.2
> Reading symbols from /tmp/sue/libattr.so.1...(no debugging symbols found)...done.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Loaded symbols for /tmp/sue/libattr.so.1
> Reading symbols from /tmp/sue/libnss_files.so.2...(no debugging symbols found)...done.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Loaded symbols for /tmp/sue/libnss_files.so.2
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> Core was generated by `/usr/local/sbin/named -u dns -c /etc/named.conf -t /var/named/chroot'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x000000000048fbba in socket_search ()
> (gdb) thread apply all bt
>
> Thread 5 (Thread 1845):
> #0  0x0000003a0b00e054 in __lll_lock_wait () from /tmp/sue/libpthread.so.0
> #1  0x0000003a0b009388 in _L_lock_854 () from /tmp/sue/libpthread.so.0
> #2  0x0000003a0b009257 in pthread_mutex_lock () from /tmp/sue/libpthread.so.0
> #3  0x0000000000490a9c in deactivate_dispsocket ()
> #4  0x0000000000491678 in udp_recv ()
> #5  0x0000000000491493 in udp_exrecv ()
> #6  0x0000000000637622 in dispatch ()
> #7  0x0000000000637916 in run ()
> #8  0x0000003a0b007851 in start_thread () from /tmp/sue/libpthread.so.0
> #9  0x0000003a0a8e811d in clone () from /tmp/sue/libc.so.6
>
> Thread 4 (Thread 1846):
> #0  0x0000003a0b00b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>    from /tmp/sue/libpthread.so.0
> #1  0x000000000064fba0 in isc_condition_waituntil ()
> #2  0x000000000063a70f in run ()
> #3  0x0000003a0b007851 in start_thread () from /tmp/sue/libpthread.so.0
> #4  0x0000003a0a8e811d in clone () from /tmp/sue/libc.so.6
> ---Type <return> to continue, or q <return> to quit---
>
> Thread 3 (Thread 1847):
> #0  0x0000003a0a8e8713 in epoll_wait () from /tmp/sue/libc.so.6
> #1  0x0000000000649bfa in watcher ()
> #2  0x0000003a0b007851 in start_thread () from /tmp/sue/libpthread.so.0
> #3  0x0000003a0a8e811d in clone () from /tmp/sue/libc.so.6
>
> Thread 2 (Thread 1843):
> #0  0x0000003a0a832c54 in sigsuspend () from /tmp/sue/libc.so.6
> #1  0x000000000063b6e2 in isc__app_ctxrun ()
> #2  0x000000000063b747 in isc__app_run ()
> #3  0x0000000000419fd7 in main ()
>
> Thread 1 (Thread 1844):
> #0  0x000000000048fbba in socket_search ()
> #1  0x0000000000490148 in get_dispsocket ()
> #2  0x0000000000497265 in dns_dispatch_addresponse2 ()
> #3  0x0000000000568581 in resquery_send ()
> #4  0x0000000000567d5b in fctx_query ()
> #5  0x0000000000576f3c in resquery_response ()
> ---Type <return> to continue, or q <return> to quit---
> #6  0x0000000000637622 in dispatch ()
> #7  0x0000000000637916 in run ()
> #8  0x0000003a0b007851 in start_thread () from /tmp/sue/libpthread.so.0
> #9  0x0000003a0a8e811d in clone () from /tmp/sue/libc.so.6
> (gdb)
> ############################################################################
>
>
> gdb back trace("thread apply all bt full")
> ############################################################################
> # gdb
> GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> (gdb) set solib-absolute-prefix /tmp/sue
> (gdb) set solib-search-path /tmp/sue
> (gdb) file /tmp/
> .ICE-unix/               libthread_db-1.0.so_org  sue-bak/
> .font-unix/              named_bak_20131224_sue   sue.tar.gz
> libnss_files.so.2        named_debug_cadns        sue2.tar.gz
> libpthread-2.12.so       sue/                     test/
> (gdb) file /tmp/named_debug_cadns
> Reading symbols from /tmp/named_debug_cadns...done.
> (gdb) core-file /tmp/sue/core.1843
> warning: core file may not match specified executable file.
> [New Thread 1844]
> [New Thread 1843]
> [New Thread 1847]
> [New Thread 1846]
> [New Thread 1845]
> Reading symbols from /tmp/sue/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
> Loaded symbols for /tmp/sue/ld-linux-x86-64.so.2
> Core was generated by `/usr/local/sbin/named -u dns -c /etc/named.conf -t /var/named/chroot'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x000000000048fbba in load_text (lctx=0xffffffffffffff90) at master.c:1611
> 1611                            GETTOKEN(lctx->lex, 0, &token, ISC_FALSE);
> (gdb) thread apply all bt full
>
> Thread 5 (Thread 1845):
> #0  0x0000003a0b00e054 in ?? ()
> No symbol table info available.
> #1  0x00000000000035c8 in ?? ()
> No symbol table info available.
> #2  0x0000000000000000 in ?? ()
> No symbol table info available.
>
> Thread 4 (Thread 1846):
> #0  0x0000003a0b00b7bb in ?? ()
> No symbol table info available.
> #1  0x0000000000000000 in ?? ()
> No symbol table info available.
>
> Thread 3 (Thread 1847):
> #0  0x0000003a0a8e8713 in ?? ()
> No symbol table info available.
> #1  0x0000000000000000 in ?? ()
> No symbol table info available.
>
> Thread 2 (Thread 1843):
> #0  0x0000003a0a832c54 in ?? ()
> ---Type <return> to continue, or q <return> to quit---
> No symbol table info available.
> #1  0x0000003a0a883a12 in ?? ()
> No symbol table info available.
> #2  0x00007ffffd02c180 in ?? ()
> No symbol table info available.
> #3  0x0000000000000057 in ?? ()
> No symbol table info available.
> #4  0x000000000063b6e2 in getservbyname ()
> No symbol table info available.
> #5  0x00007f6921782010 in ?? ()
> No symbol table info available.
> #6  0x0000000000915de0 in ?? ()
> No symbol table info available.
> #7  0x00000000004058f0 in ?? ()
> No symbol table info available.
> #8  0x0000000000610ad7 in ?? ()
> No symbol table info available.
> #9  0x00007ffffd02c170 in ?? ()
> No symbol table info available.
> #10 0x00007f692179b030 in ?? ()
> No symbol table info available.
> #11 0x0000000021796540 in ?? ()
> No symbol table info available.
> ---Type <return> to continue, or q <return> to quit---
> #12 0x0000000000000000 in ?? ()
> No symbol table info available.
>
> Thread 1 (Thread 1844):
> #0  0x000000000048fbba in load_text (lctx=0xffffffffffffff90) at master.c:1611
>         rdclass = 0
>         type = 20
>         covers = <value optimized out>
>         ttl_offset = 0
>         new_name = <value optimized out>
>         current_has_delegation = isc_boolean_false
>         done = isc_boolean_false
>         finish_origin = isc_boolean_false
>         finish_include = isc_boolean_false
>         read_till_eol = 561482224
>         initialws = isc_boolean_false
>         include_file = 0x0
>         token = {type = 561485888, value = {as_char = -25 '\347',
>             as_ulong = 4682727, as_region = {
>               base = 0x4773e7 "D$\034\205\300\017\205", <incomplete sequence \313>, length = 6863648}, as_textregion = {
>               base = 0x4773e7 "D$\034\205\300\017\205", <incomplete sequence \313>, length = 6863648}, as_pointer = 0x4773e7}}
> ---Type <return> to continue, or q <return> to quit---
>         result = 65552
>         glue_list = {head = 0x7f69193cd310, tail = 0x7f6921779760}
>         current_list = {head = 0x908e70, tail = 0xb85300}
>         this = <value optimized out>
>         rdatalist = 0x0
>         new_rdatalist = 0x7f6910fe7f80
>         rdlcount = 0
>         rdlcount_save = 0
>         rdatalist_size = 3
>         buffer = {magic = 0, base = 0x7f6900000000, length = 6863648,
>           used = 0, current = 561485664, active = 32617, link = {
>             prev = 0x7f6921779760, next = 0x68bb20}, mctx = 0x1421779770}
>         target = {magic = 0, base = 0x7f69217796d0, length = 561485744,
>           used = 32617, current = 4682727, active = 0, link = {
>             prev = 0x908e70, next = 0x7ffffd02c100}, mctx = 0x7f6921779740}
>         target_ft = <value optimized out>
>         target_save = Asked for position 0 of stack, stack only has 0 elements on it.
> (gdb)
> (gdb) quit
>
> ###messages
> Dec 23 19:58:13 hogedns021 kernel: named[1844]: segfault at 0 ip 000000000048fbb a sp 00007f6921778840 error 4 in named[400000+303000] Dec 23 19:58:15 hogedns021 abrtd: Directory 'ccpp-2013-12-23-19:58:13-1843' crea tion detected Dec 23 19:58:15 hogedns021 abrt[19323]: Saved core dump of pid 1843 (/usr/local/
> sbin/named) to /var/spool/abrt/ccpp-2013-12-23-19:58:13-1843 (166387712 bytes) Dec 23 19:58:15 hogedns021 abrtd: Executable '/usr/local/sbin/named' doesn't bel ong to any package Dec 23 19:58:15 hogedns021 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-12 -23-19:58:13-1843' exited with 1 Dec 23 19:58:15 hogedns021 abrtd: Corrupted or bad directory /var/spool/abrt/ccp p-2013-12-23-19:58:13-1843, deleting ###
>
> ###info
> [root at hogedns021 log]# named -V
> BIND 9.9.3-P2 (Extended Support Version) <id:d8a6fe8b> built with '--prefix=/usr /local/' '--disable-openssl-version-check' '--enable-filter-aaaa' '--enable-thre ads' '--with-gssapi=no' 'CFLAGS=-DDIG_SIGCHASE'
> using OpenSSL version: OpenSSL 1.0.0 29 Mar 2010
> [root at hogedns021 log]# uname -a
> Linux hogedns021 2.6.32-279.19.1.el6.x86_64 #1 SMP Sat Nov 24 14:35:28 EST 2012
> x86_64 x86_64 x86_64 GNU/Linux
> [root at hogedns021 log]# ls -la /var/named/chroot/var/named/core.1843
> -rw------- 1 dns dns 166387712 Dec 23 19:58 2013 /var/named/chroot/var/named/co
> re.1843
> [root at hogedns021 log]# file /var/named/chroot/var/named/core.1843
> /var/named/chroot/var/named/core.1843: ELF 64-bit LSB core file x86-64, version
> 1 (SYSV), SVR4-style, from '/usr/local/sbin/named -u dns -c /etc/named.conf -t / var/named/chroot'
> ############################################################################
>
>
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users


More information about the bind-users mailing list