BIND 10 #2249: order of shutdowns should not allow msgq or cfgmgr to exit before other components

[BIND 10 Development]

#2249: order of shutdowns should not allow msgq or cfgmgr to exit before other
components
-------------------------------------+-------------------------------------
            Reporter:  jreed         |                        Owner:
                Type:  defect        |                       Status:  new
            Priority:  medium        |                    Milestone:  New
           Component:  Boss of BIND  |  Tasks
           Sensitive:  0             |                     Keywords:
         Sub-Project:  Core          |              Defect Severity:  N/A
Estimated Difficulty:  0             |  Feature Depending on Ticket:
         Total Hours:  0             |          Add Hours to Ticket:  0
                                     |                    Internal?:  0
-------------------------------------+-------------------------------------
 See https://lists.isc.org/pipermail/bind10-dev/2012-September/003793.html

 The order of the logging and timestamps may be misleading, but it seemed
 to show that the cfgmgr shutdown cleanly and msgq received a SIGTERM
 before b10-xfrin tried to use them.

 {{{
 2012-09-03 16:03:18.877 INFO  [b10-boss.boss] BIND10_STOP_PROCESS asking
 cfgmgr to shut down
 ...
 2012-09-03 16:03:18.877 INFO  [b10-boss.boss] BIND10_STOP_PROCESS asking
 b10-xfrin to shut down
 ...
 2012-09-03 16:03:19.878 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED
 process 1953 of cfgmgr ended with status 0
 2012-09-03 16:03:19.879 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED
 process 1957 of b10-auth-1 ended with status 256
 2012-09-03 16:03:19.879 INFO  [b10-boss.boss] BIND10_PROCESS_ENDED
 process 1959 of b10-stats ended with status 0
 2012-09-03 16:03:19.879 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM
 sending SIGTERM to msgq (PID 1952)
 2012-09-03 16:03:19.879 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM
 sending SIGTERM to b10-xfrin (PID 1954)
 2012-09-03 16:03:19.879 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM
 sending SIGTERM to b10-zonemgr (PID 1955)
 2012-09-03 16:03:19.879 INFO  [b10-boss.boss] BIND10_SEND_SIGTERM
 sending SIGTERM to b10-cmdctl (PID 1958)
 ...
 2012-09-03 16:03:19.884 ERROR [b10-xfrin.config]
 CONFIG_SESSION_STOPPING_FAILED error
 sending stopping message: [Errno 32] Broken pipe
 2012-09-03 16:03:19.887 ERROR [b10-zonemgr.config]
 CONFIG_SESSION_STOPPING_FAILED
 error sending stopping message: [Errno 32] Broken pipe
 }}}

 Then zonemgr crashes (as seen at
 https://lists.isc.org/pipermail/bind10-users/2012-September/000391.html):

 {{{
 Traceback (most recent call last):
    File "/opt/bind10/libexec/bind10-devel/b10-zonemgr", line 699, in
 <module>
      zonemgrd = Zonemgr()
    File "/opt/bind10/libexec/bind10-devel/b10-zonemgr", line 523, in
 __init__
      self._setup_session()
    File "/opt/bind10/libexec/bind10-devel/b10-zonemgr", line 542, in
 _setup_session
      self._module_cc.add_remote_config(AUTH_SPECFILE_LOCATION)
    File
 "/opt/bind10/lib/python3.3/site-packages/isc/config/ccsession.py", line
 419, in add_remote_config
      self._add_remote_config_internal(module_spec, config_update_callback)
    File
 "/opt/bind10/lib/python3.3/site-packages/isc/config/ccsession.py", line
 343, in _add_remote_config_internal
      answer, _ = self._session.group_recvmsg(False, seq)
    File "/opt/bind10/lib/python3.3/site-packages/isc/cc/session.py",
 line 275, in group_recvmsg
      env, msg  = self.recvmsg(nonblock, seq)
    File "/opt/bind10/lib/python3.3/site-packages/isc/cc/session.py",
 line 130, in recvmsg
      data = self._receive_full_buffer(nonblock)
    File "/opt/bind10/lib/python3.3/site-packages/isc/cc/session.py",
 line 239, in _receive_full_buffer
      raise se
    File "/opt/bind10/lib/python3.3/site-packages/isc/cc/session.py",
 line 212, in _receive_full_buffer
      self._receive_len_data()
    File "/opt/bind10/lib/python3.3/site-packages/isc/cc/session.py",
 line 172, in _receive_len_data
      new_data = self._receive_bytes(self._recv_len_size)
    File "/opt/bind10/lib/python3.3/site-packages/isc/cc/session.py",
 line 158, in _receive_bytes
      data = self._socket.recv(size)
 ConnectionResetError: [Errno 104] Connection reset by peer
 Exception BrokenPipeError: BrokenPipeError(32, 'Broken pipe') in <bound
 method ModuleCCSession.__del__ of <isc.config.ccsession.ModuleCCSession
 object at 0x7f4106438390>> ignored
 }}}


 From the logging above it appears that stopping the msqg and cfgmgr didn't
 get postponed.

 Also see ticket #2245 which has similar example.

-- 
Ticket URL: <http://bind10.isc.org/ticket/2249>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development