[bind10-dev] msgq crash and auth died in trac327

Jeremy C. Reed jreed at isc.org
Thu Dec 9 15:25:11 UTC 2010


I am running trac327.
b10-recurse is enabled and bind10 continually restarting (I understand).
In bindctl I did:

config set listen_on [ { "address": "127.0.0.1", "port": 5301 } ]
config commit

[bind10] Resurrecting dead b10-recurse process...

(jreed: I understand this part because was trying to bind to default 
port which was already in use. I also know that there isn't a way to 
restart recurse to listen on new port.)

[bind10] Resurrected b10-recurse (PID 13851)
[b10-recurse] Command line: b10-recurse -v
[b10-recurse] Server created.
[b10-recurse] Query shutdown
[b10-recurse] Query setup
[b10-recurse] IOService created.
[b10-recurse] Configuration session channel created.
[b10-cmdctl] Finish send command 'set_config' to module 'ConfigManager'
/Recurse> 
/Recurse> [b10-recurse] Server failed: Timeout while reading data from 
cc session
[b10-recurse] Query shutdown
[b10-recurse] Deleting the Recursor
[bind10] Process b10-recurse (PID 13851) died.
Traceback (most recent call last):
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 448, in <module>
    msgq.run()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 364, in run
    self.run_poller()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 382, in run_poller
    self.process_socket(fd)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 206, in process_socket
    self.process_packet(fd, sock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 273, in process_packet
    self.process_command(fd, sock, routingmsg, data)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 283, in process_command
    self.process_command_send(sock, routing, data)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 344, in process_command_send
    self.send_prepared_msg(socket, msg)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-msgq", 
line 311, in send_prepared_msg
    sock.send(msg)
socket.error: [Errno 32] Broken pipe
Traceback (most recent call last):
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-cfgmgr", 
line 69, in <module>
[b10-auth] Server failed: asynchronous read failed
[bind10] msgq channel disappeared.
Traceback (most recent call last):
Traceback (most recent call last):
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-xfrout", 
line 545, in <module>
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-zonemgr", 
line 578, in <module>
[bind10] msgq channel disappeared.
[bind10] msgq channel disappeared.
Traceback (most recent call last):
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 416, in <module>
[bind10] msgq channel disappeared.
    sys.exit(main())
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-cfgmgr", 
line 54, in main
[bind10] msgq channel disappeared.
[bind10] msgq channel disappeared.
[bind10] Process b10-auth (PID 13815) died.
[bind10] Resurrecting dead b10-auth process...
    cm.run()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/config/cfgmgr.py", 
line 410, in run
    zonemgrd.run()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-zonemgr", 
line 552, in run
    xfrout_server.run()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-xfrout", 
line 518, in run
    main()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 404, in main
[b10-xfrin] Read of 0 bytes: connection closed
    msg, env = self.cc.group_recvmsg(False)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 247, in group_recvmsg
    self._module_cc.check_command(False)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/config/ccsession.py", 
line 192, in check_command
    subject.check()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 166, in check
    self._cc.check_command(False)
    env, msg  = self.recvmsg(nonblock, seq)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/config/ccsession.py", 
line 192, in check_command
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 108, in recvmsg
    msg, env = self._session.group_recvmsg(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 247, in group_recvmsg
    self.notify('check')
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 128, in notify
    data = self._receive_full_buffer(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 184, in _receive_full_buffer
    msg, env = self._session.group_recvmsg(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 247, in group_recvmsg
    env, msg  = self.recvmsg(nonblock, seq)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 108, in recvmsg
    listener.update(event)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 142, in update
    self._receive_len_data()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 144, in _receive_len_data
[bind10] Resurrected b10-auth (PID 13855)
    data = self._receive_full_buffer(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 184, in _receive_full_buffer
    env, msg  = self.recvmsg(nonblock, seq)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 108, in recvmsg
    return callback()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 92, in __call__
    new_data = self._receive_bytes(self._recv_len_size)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 132, in _receive_bytes
[bind10] msgq channel disappeared.
    self._receive_len_data()
    data = self._receive_full_buffer(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 144, in _receive_len_data
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 184, in _receive_full_buffer
    return self.callback(*args, **kwargs)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-stats", 
line 236, in check
[bind10] Process b10-msgq (PID 13813) died.
[bind10] The b10-msgq process died, shutting down.
    raise ProtocolError("Read of 0 bytes: connection closed")
isc.cc.session.ProtocolError: Read of 0 bytes: connection closed
    new_data = self._receive_bytes(self._recv_len_size)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 132, in _receive_bytes
    self._receive_len_data()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 144, in _receive_len_data
[bind10] msgq channel disappeared.
[bind10] Stopping the server.
    return self.cc_session.check_command(False)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/config/ccsession.py", 
line 192, in check_command
    raise ProtocolError("Read of 0 bytes: connection closed")
    new_data = self._receive_bytes(self._recv_len_size)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 132, in _receive_bytes
isc.cc.session.ProtocolError: Read of 0 bytes: connection closed
    msg, env = self._session.group_recvmsg(nonblock)
    raise ProtocolError("Read of 0 bytes: connection closed")
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 247, in group_recvmsg
isc.cc.session.ProtocolError: Read of 0 bytes: connection closed
    env, msg  = self.recvmsg(nonblock, seq)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 108, in recvmsg
    data = self._receive_full_buffer(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 184, in _receive_full_buffer
[b10-auth] Server created.
[b10-auth] IOService created.
[b10-auth] Configuration session channel created.
    self._receive_len_data()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 144, in _receive_len_data
[b10-auth] Server failed: Unable to connect to message queue: Connection 
refused
    new_data = self._receive_bytes(self._recv_len_size)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 132, in _receive_bytes
    raise ProtocolError("Read of 0 bytes: connection closed")
isc.cc.session.ProtocolError: Read of 0 bytes: connection closed
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/jreed/pkg/lib/python3.1/threading.py", line 509, in 
_bootstrap_inner
    self.run()
  File "/home/jreed/pkg/lib/python3.1/threading.py", line 462, in run
    self._target(*self._args, **self._kwargs)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/libexec/bind10-devel/b10-cmdctl", 
line 330, in _handle_msg_from_msgq
    self._module_cc.check_command(False)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/config/ccsession.py", 
line 192, in check_command
    msg, env = self._session.group_recvmsg(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 247, in group_recvmsg
    env, msg  = self.recvmsg(nonblock, seq)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 108, in recvmsg
    data = self._receive_full_buffer(nonblock)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 211, in _receive_full_buffer
    raise se
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 184, in _receive_full_buffer
    self._receive_len_data()
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 144, in _receive_len_data
    new_data = self._receive_bytes(self._recv_len_size)
  File 
"/home/jreed/dnsbenchsuite/work/branches/trac327-HEAD/20101209134603/install/lib/python3.1/site-packages/isc/cc/session.py", 
line 130, in _receive_bytes
    data = self._socket.recv(size)
socket.error: [Errno 104] Connection reset by peer

[bind10] Process b10-cfgmgr (PID 13814) died.
[bind10] Process b10-xfrin (PID 13818) died.
[bind10] Process b10-stats (PID 13821) died.
[bind10] Process b10-auth (PID 13855) died.
[bind10] Sending SIGTERM to b10-xfrout (PID 13817).
[bind10] Sending SIGTERM to b10-zonemgr (PID 13819).
[bind10] Sending SIGTERM to b10-cmdctl (PID 13822).
[bind10] Process b10-cmdctl (PID 13822) died.
[bind10] Sending SIGKILL to b10-xfrout (PID 13817).
[bind10] Sending SIGKILL to b10-zonemgr (PID 13819).
[bind10] Process b10-xfrout (PID 13817) died.
[bind10] Process b10-zonemgr (PID 13819) died.
[bind10] All processes ended, server done.
[bind10] BIND 10 exiting

[Errno 111] Connection refused 
Failed to connect with b10-cmdctl module, is it running?


(This is on CentOS Linux 5.3.)



More information about the bind10-dev mailing list