BIND 10 #2856: memory manager initialization
BIND 10 Development
do-not-reply at isc.org
Tue Jun 18 04:33:39 UTC 2013
#2856: memory manager initialization
-------------------------------------+-------------------------------------
Reporter: jinmei | Owner:
Type: task | Status: new
Priority: medium | Milestone:
Component: shmem | Sprint-20130625
manager | Resolution:
Keywords: | CVSS Scoring:
Sensitive: 0 | Defect Severity: N/A
Sub-Project: DNS | Feature Depending on Ticket:
Estimated Difficulty: 4 | shared memory data source
Total Hours: 0 | Add Hours to Ticket: 0
| Internal?: 0
-------------------------------------+-------------------------------------
Description changed by jinmei:
Old description:
> Subtask of #2830. Depend on #2854, #2855, #2993.
>
> (Revised, based on more details of resulting #2854, although the
> details may even change more after review).
>
> First, this is how the entire memmgr will look like after the series
> of related tasks, not only this particular ticket:
>
> - (the base class of) `SegmentInfo` will have the `__state` attribute.
> This is independent from the underlying segment type ("mapped", etc),
> and can take the following 4 values:
> - UPDATING: the segment is being updated (by the builder thread,
> although `SegmentInfo` won't care about this level of details).
> - SYNCHRONIZING: one pair of underlying segments has been updated,
> and readers are now migrating to the updated version of the segment.
> - COPYING: all readers that used the old version of segment have been
> migrated to the updated version, and the old segment is now being
> updated.
> - READY: both segments of the pair have been updated. it can now
> handle further updates (e.g., from xfrin).
> - `__state` is private to `SegmentInfo`; but maybe we provide
> read-only access for tests
> - To manage state transition, `SegmentInfo` will have some more
> attributes and methods:
> - `__readers` (set of 'reader_session_id'): private to `SegmentInfo`.
> It
> consists of the (ID of) reader modules that are using the "current"
> reader
> version of the segment.
> - `__old_readers` (set of 'reader_session_id'): private to
> `SegmentInfo`
> for write (update), but publicly readable. It can be non empty
> only in the SYNCHRONIZING state, and consists of (ID of) reader
> modules that are using the old version of the segments (and have
> to migrate to the updated version).
> - `__events` (FIFO queue of opaque data): queue for pending update
> events. update events can come at any timing (e.g., after
> xfr-in), but can be only handled if `SegmentInfo` is in the READY
> state. This maintain such pending events in the order of coming.
> `SegmentInfo` doesn't have to know the details of the stored data;
> it only matters for the memmgr.
> - `complete_update()`: called when memmgr is notified from the
> builder of the completion of segment update. change the state
> from UPDATING to SYNCHRONIZING, and COPYING to READY. In the
> former case, elements of `__readers` are moved to `__old_readers`.
> and, if `__old_readers` is empty, it pops the head (oldest) event
> from `__events` and return it. error if it's called in other
> states than UPDATING and COPYING.
> - `sync_reader(reader_session_id)`: can only be called in the
> SYNCHRONIZING state. memmgr calls it when it receives
> segment_update_ack message from a reader module. It moves the
> given ID from `__old_readers` to `__readers`, and if
> `__old_readers` becomes empty, change the state to COPYING.
> If the state has changed to COPYING, it pops the head (oldest) event
> from `__events` and return it; otherwise it returns None.
> - `add_event(event_data)`: called by memmgr when it receives a
> request for reloading a zone, and simply append the given data to
> `__events`.
> - `start_update()`: if the state is READY and `__events` is non
> empty, changes the state to UPDATING and returns the head (oldest)
> event (not popping it). This tells the caller (memmgr) should
> initiate the update process with the builder. In all other cases
> it returns None.
> - `add_reader(reader_session_id)`: called by memmgr when it first
> gets the pre-existing readers or when it's notified of a new
> reader. It simply adds the given ID to `__readers`. no state
> transition happens.
> - `remove_reader(reader_session_id)`: called by memmgr when it's
> notified that an existing reader has unsubscribed. It removes
> the given reader from either `__readers` or `__old_readers`
> (wherever the reader belonged), and in the latter case, if
> `__old_readers` becomes empty, change the state to COPYING
> (the state should have been SYNCHRONIZING), pops the head (oldest)
> event from `__events` and return it.
>
> A summarized (and simplified) state transition diagram would be as
> follows:
> {{{
> +--sync_reader()/remove_reader()
> | still have old readers
> | |
> UPDATING-----complete_--->SYNCHRONIZING<---+
> ^ update() |
> start_update()| |
> sync_reader()/remove_reader()
> events | V no more old reader
> exist READY<------complete_----------COPYING
> update()
> }}}
>
> On construction, `SegmentInfo` object is in the READY state, with
> `__readers`, `__old_readers`, `__events` being all empty.
>
> Commands (elements of `__events`) passed from the main memmgr thread
> to the builder would be a tuple (or whatever, this is just a
> conceptual example):
> {{{#!python
> ('load', zone_name, dsrc_info, rrclass, dsrc_name, client_list,
> sgmt_info)
> }}}
> where
> - zone_name is None or isc.dns.Name, specifying the zone name to load.
> If it's None, it means all zones to be cached in the specified data
> source (used for initialization)
> - dsrc_info is a DataSrcInfo object corresponding to the generation ID
> of the set of data sources for this loading.
> - rrclass is isc.dns.RRClass object, the RR class of the data source.
> - dsrc_name is a string, specifying a data source name.
>
> The builder thread tells the main memmgr thread when it completes a
> given load command with the following parameter (again, this is just a
> conceptual example):
> {{{
> ('load-completed', dsrc_info, rrclass, dsrc_name)
> }}}
> where dsrc_info, rrclass and dsrc_name are the same values of those in
> the
> corresponding command.
>
> In this particular ticket, we do the following:
>
> - Extend `SegmentInfo`, adding the new attributes and methods (even
> though not all of them are needed right now).
> - Then extend the `MemorySegmentBuilder` class (introduced in #2855)
> for the 'load' command. It will look like as follows:
> {{{#!python
> def __handle_load(self, zname, dsrc_info, rrclass, dsrc_name):
> clist = dsrc_info.clients_map[rrclass]
> sgmt_info = dsrc_info.segmet_info_map[(rrclass, dsrc_name)]
> clist.reset_memory_segment(dsrc_name, READ_WRITE,
> sgmt_info.get_reset_param(WRITER))
>
> if zname is not None:
> zones = [(None, zname)]
> else:
> zones = clist.get_zone_table_accessor(dsrc_name, True)
>
> for _, zname in zones:
> # Note: until #2993 is completed we cannot use this extension
> cache_load_error = (zname is None) # install empty zone
> initially
> writer = clist.get_cached_zone_writer(zname, dsrc_name,
> cache_load_error)
> try:
> error = writer.load()
> if error is not None:
> # log the error
> continue
> except Exception:
> # log it
> continue
> writer.install()
> writer.cleanup()
>
> with self.__lock_update_queue:
> self.__update_queue.append(('load-completed', dsrc_info,
> rrclass,
> dsrc_name))
> self.__sock_to_main.send(<some_data>)
> }}}
New description:
Subtask of #2830. Depend on #2854, #2855, #2993.
(Revised, based on more details of resulting #2854, although the
details may even change more after review).
First, this is how the entire memmgr will look like after the series
of related tasks, not only this particular ticket:
- (the base class of) `SegmentInfo` will have the `__state` attribute.
This is independent from the underlying segment type ("mapped", etc),
and can take the following 4 values:
- UPDATING: the segment is being updated (by the builder thread,
although `SegmentInfo` won't care about this level of details).
- SYNCHRONIZING: one pair of underlying segments has been updated,
and readers are now migrating to the updated version of the segment.
- COPYING: all readers that used the old version of segment have been
migrated to the updated version, and the old segment is now being
updated.
- READY: both segments of the pair have been updated. it can now
handle further updates (e.g., from xfrin).
- `__state` is private to `SegmentInfo`; but maybe we provide
read-only access for tests
- To manage state transition, `SegmentInfo` will have some more
attributes and methods:
- `__readers` (set of 'reader_session_id'): private to `SegmentInfo`.
It
consists of the (ID of) reader modules that are using the "current"
reader
version of the segment.
- `__old_readers` (set of 'reader_session_id'): private to `SegmentInfo`
for write (update), but publicly readable. It can be non empty
only in the SYNCHRONIZING state, and consists of (ID of) reader
modules that are using the old version of the segments (and have
to migrate to the updated version).
- `__events` (FIFO queue of opaque data): queue for pending update
events. update events can come at any timing (e.g., after
xfr-in), but can be only handled if `SegmentInfo` is in the READY
state. This maintain such pending events in the order of coming.
`SegmentInfo` doesn't have to know the details of the stored data;
it only matters for the memmgr.
- `complete_update()`: called when memmgr is notified from the
builder of the completion of segment update. change the state
from UPDATING to SYNCHRONIZING, and COPYING to READY. In the
former case, elements of `__readers` are moved to `__old_readers`.
and, if `__old_readers` is empty, it pops the head (oldest) event
from `__events` and return it. error if it's called in other
states than UPDATING and COPYING.
- `sync_reader(reader_session_id)`: can only be called in the
SYNCHRONIZING state. memmgr calls it when it receives
segment_update_ack message from a reader module. It moves the
given ID from `__old_readers` to `__readers`, and if
`__old_readers` becomes empty, change the state to COPYING.
If the state has changed to COPYING, it pops the head (oldest) event
from `__events` and return it; otherwise it returns None.
- `add_event(event_data)`: called by memmgr when it receives a
request for reloading a zone, and simply append the given data to
`__events`.
- `start_update()`: if the state is READY and `__events` is non
empty, changes the state to UPDATING and returns the head (oldest)
event (not popping it). This tells the caller (memmgr) should
initiate the update process with the builder. In all other cases
it returns None.
- `add_reader(reader_session_id)`: called by memmgr when it first
gets the pre-existing readers or when it's notified of a new
reader. It simply adds the given ID to `__readers`. no state
transition happens.
- `remove_reader(reader_session_id)`: called by memmgr when it's
notified that an existing reader has unsubscribed. It removes
the given reader from either `__readers` or `__old_readers`
(wherever the reader belonged), and in the latter case, if
`__old_readers` becomes empty, change the state to COPYING
(the state should have been SYNCHRONIZING), pops the head (oldest)
event from `__events` and return it.
A summarized (and simplified) state transition diagram would be as
follows:
{{{
+--sync_reader()/remove_reader()
| still have old readers
| |
UPDATING-----complete_--->SYNCHRONIZING<---+
^ update() |
start_update()| |
sync_reader()/remove_reader()
events | V no more old reader
exist READY<------complete_----------COPYING
update()
}}}
On construction, `SegmentInfo` object is in the READY state, with
`__readers`, `__old_readers`, `__events` being all empty.
Commands (elements of `__events`) passed from the main memmgr thread
to the builder would be a tuple (or whatever, this is just a
conceptual example):
{{{#!python
('load', zone_name, dsrc_info, rrclass, dsrc_name)
}}}
where
- zone_name is None or isc.dns.Name, specifying the zone name to load.
If it's None, it means all zones to be cached in the specified data
source (used for initialization)
- dsrc_info is a DataSrcInfo object corresponding to the generation ID
of the set of data sources for this loading.
- rrclass is isc.dns.RRClass object, the RR class of the data source.
- dsrc_name is a string, specifying a data source name.
The builder thread tells the main memmgr thread when it completes a
given load command with the following parameter (again, this is just a
conceptual example):
{{{
('load-completed', dsrc_info, rrclass, dsrc_name)
}}}
where dsrc_info, rrclass and dsrc_name are the same values of those in the
corresponding command.
In this particular ticket, we do the following:
- Extend `SegmentInfo`, adding the new attributes and methods (even
though not all of them are needed right now).
- Then extend the `MemorySegmentBuilder` class (introduced in #2855)
for the 'load' command. It will look like as follows:
{{{#!python
def __handle_load(self, zname, dsrc_info, rrclass, dsrc_name):
clist = dsrc_info.clients_map[rrclass]
sgmt_info = dsrc_info.segmet_info_map[(rrclass, dsrc_name)]
clist.reset_memory_segment(dsrc_name, READ_WRITE,
sgmt_info.get_reset_param(WRITER))
if zname is not None:
zones = [(None, zname)]
else:
zones = clist.get_zone_table_accessor(dsrc_name, True)
for _, zname in zones:
# Note: until #2993 is completed we cannot use this extension
cache_load_error = (zname is None) # install empty zone
initially
writer = clist.get_cached_zone_writer(zname, dsrc_name,
cache_load_error)
try:
error = writer.load()
if error is not None:
# log the error
continue
except Exception:
# log it
continue
writer.install()
writer.cleanup()
with self.__lock_update_queue:
self.__update_queue.append(('load-completed', dsrc_info,
rrclass,
dsrc_name))
self.__sock_to_main.send(<some_data>)
}}}
--
--
Ticket URL: <http://bind10.isc.org/ticket/2856#comment:6>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development
More information about the bind10-tickets
mailing list