BIND 10 #1634: Abstract pool/lease store: initial requirements list

[BIND 10 Development]

#1634: Abstract pool/lease store: initial requirements list
-------------------------------------+-------------------------------------
                   Reporter:  tomek  |                 Owner:  tomek
                       Type:  task   |                Status:  reviewing
                   Priority:  major  |             Milestone:  Sprint-
                  Component:  dhcp   |  DHCP-20120220
                   Keywords:         |            Resolution:
            Defect Severity:  N/A    |             Sensitive:  0
Feature Depending on Ticket:         |           Sub-Project:  DHCP
        Add Hours to Ticket:  0      |  Estimated Difficulty:  0
                  Internal?:  0      |           Total Hours:  0
-------------------------------------+-------------------------------------

Comment (by tomek):

 Replying to [comment:6 sar]:
 > ''MUST allow lease manipulation on-line (during server operation)''
 > Do we need to allow this while the server is running?  Or is it possible
 to do something similar to Bind9 where the database is frozen for a bit
 while the changes are made?
 Since we already have requirement for concurrent DB writes, I think we
 should go with real on-line change. I imagine that there will be some
 short time freeze, e.g. checking if existing leases are still within new
 pool definitions.

 > Re: Storage of additional information
 > Stephen has this in his notes:
 > * It MUST be possible to store multiple options of the same type with a
 >   particular lease.
 >
 > Can we elaborate on this?  Does this mean multiple options from multiple
 sources or multiple options within the same packet?  In v4 multiple
 options tend to get concatenated and we should be clear if it applies to
 v4 and if so how it applies.
 This was about v6 and relay included options. Added clarification that v4
 options will be concatenated.

 > I read the statement ''MAY allow view/update by existing tools''
 > as saying that we may allow Omapi to access the database in some
 fashion.  As there seems to be confusion as to what the document means it
 should be clarified.  I assume we would only worry about Omapi and we
 wouldn't worry about tools that might read the current lease database file
 as there isn't much of a way for them to connect to the new database.  And
 I would agree that this is a "may".  We've told folks that we will see
 what we can do but haven't promised anything.
 I was too vague here. I meant that existing SQL clients or existing BIND10
 tools like cfgmgr. I definitely didn't want to suggest OMAPI. Clarified
 that.

 >
 > ''string or char[128] client-id - DUID is up to 128 bytes long, DHCPv4
 uses HW address;''
 > DHCPv4 client id doesn't have to be the HW address and is mostly treated
 as an opaque blob,
 > though I don't think that really affects the DB except that we need
 fields for both HW address
 > and client id.
 Agree that they should be separated. However, for v4 we still should keep
 HW address. Users may want to assign fixed leases based on MAC addresses.
 Even in v6, folks are not happy with DUID not being equal to MAC.
 Therefore HW address option for DHCPv6 was proposed recently and has good
 chance of being approved.

 > ''timestamp cltt - client last communication time - required for
 failover;''
 > I think there are several other fields we may need or want for failover
 if we choose to do the current failover draft again.  However we may do
 failover in a different fashion.  If we allow multiple DHCP servers to
 access the same backend database then we don't need to have the DHCP
 servers talk to each other to do failover.  The failover is handled by the
 backend DB.  Using this model does have implications - in particular with
 what happens if the two servers are partitioned.
 CLTT is also useful for other things, like timers calculation or
 leasequery. Do you think it would be useful to review failover-12 draft
 and see which fields should be kept in DB?

 On a related note, since v6 failover is currently not defined and we can't
 defer DB design until it is, we will need to realize that database format
 will get updated over time. Added requirement about it.

 > ''bool fixed - Is this lease fixed? If yes, never remove it from DB''
 > In the current v4 code fixed leases don't appear in the DB, they are in
 the configuration file but don't get written to the DB.  Reserved leases
 do get written to the DB as well as going through the other steps of a
 lease.
 I think we should use it differently. In DHCP4 support for v4 appeared
 first and v6 was added much later. This is clearly visible in how the code
 operates, e.g. v6 requires server to have global addresses defined on
 intefaces (that is clearly an artificial requirement introduced by the
 software itself, not the protocol).

 > This brings up the question of configuration vs DB.  In the current
 split information such as pool start and end is in the configuration file
 but not in the DB.  Are we planning to have a separate configuration
 store?  (I thought the answer was yes).  If so how are we deciding which
 information goes into which area?
 That was my assumption as well. We are now trying to design data
 structures. Where specific data will be kept is an implementation details.
 Added question about making that decision for each table. We should get
 back to that question.

 > ''bool fqdn_fwd - was forward update performed? Required during lease
 release or expiration.''
 > What information we choose to store for DDNS will depend quite strongly
 on how we choose to implement it.  The original suggestion was that there
 would be a co-located DNS server.  The dhcp server would update the DNS
 server via messages.  The DNS server would update the master DNS server
 using DNS magic.  The more recent suggestion from Bluecat was that there
 would be a DDNS update module.  The DHCP server would simply send the
 appropriate information to the DDNS module and it would be responsible for
 doing the DDNS transactions with an actual DNS server.  It would be
 possible for the DDNS module to update the lease DB but it would probably
 be neater to have a separate DB that contained only the DDNS information.
 The DHCP server would pass messages to the DDNS module whenever there was
 a transition and let the DDNS module sort out if it needed to do an
 update.
 Stephen suggested that we should keep a separate table for that. Although
 the co-located mode will probably be used in first implementation, regular
 (DDNS over wire) updates will later be supported as well. Personally, I
 think that having a separate module for that is very good approach.
 Nevertheless, I think this module will use both DB back-end and
 configuration store for its operation. Therefore we should pay attention
 to DDNS now. There are customers, who complained that after they
 reconfigure DHCP4, it does not clean up DNS properly. We should make this
 design flexible to keep DDNS info even after it no longer matches current
 configuration. Exact details still need to be worked out.

 > In short we may not need to store any DDNS information (though we may
 wish to track anything we need to generate such as FQDN names.)
 >
 > As a minor note currently we don't actually know which DNS server we
 updated.  We can track up to 3 servers for a zone and simply hand them all
 to the DNS client code.  Mostly we get back success of failure - but no
 indication of which server it contacted.
 That was exactly my point. Customers complained about that. Once they
 change their DNS configuration, it is no longer possible to clean up older
 entries. I think it would be good to use following algorithm:

 1. for new updates, use whatever is stored in configuration
 2. for existing leases, keep info on which server/key/protocol etc. was
 used to remove the entry in the same way as it was defined. Keep that even
 when current configuration changes.
 3. when removing RR, try to use info stored with lease. If it does not
 work, try to use current configuration.

 That approach would increase our chances to remove RRs successfully.

 > ''t1, t2 timers (v6 only)''
 > I believe we currently keep similar things for v4 and use them to set
 the suggested renew and rebind times.
 I have missed that. Thanks. Renamed both (v4 and v6) fields to renew timer
 and rebind timer.

 > ''Do we want to keep separate list of fixed-leases?''
 > I think there are two features in our current separation of fixed-
 leases.  1) We don't have to put them into the DB, and indeed don't do
 several things with them that we do with normal leases.  2) The provide a
 cheap way of doing failover.  There are some places that only do fixed
 leases.  When a new machine appears an address is selected for it and a
 new conf file is generated (or they use omapi to generate the new entry).
 In this case if the same set of fixed addresses are shared between a set
 of servers they can all the same clients without conflict.  (But woe unto
 you if your files become inconsistent.)
 >
 > I don't think (1) is that interesting and we have had to add the
 reserved option to allow for fixed addresses that do go through the normal
 cycle.  (2) has some interest but we can probably get the same features
 out of using the same DB for multiple DHCP servers.
 Since this is a clean sheet design, do you think that we need to keep both
 fixed and reserved leases? Wouldn't it be simpler if we decided to use
 just one type?

 > ''Do we want to have subnets and pools split?''
 > I believe we should try and simplify the current arrangement, but the
 arrangement exists for some reasons so we should 1) try to figure out what
 they are 2) provide similar features in the new arrangement 3) be prepared
 to add some of the hierarchy as needed.  (It may be that much of the
 complexity is simply due to adding new features on top of old and we don't
 actually need most of it, but I'm not sure if that is wishful thinking or
 not.)
 I came up with 2 reasons why subnet and pools are split:
 - v6 relays don't have to include interface-id. In that case relay's
 global address is needed to find out which link client is attached to. By
 definition, this address is statically configured, so it is not in pool. I
 believe the same holds true for v4.
 - When server responds to CONFIRM in v6, it really answers if address is
 valid on this link, not if server has a lease for this address. Any server
 on link can answer CONFIRM. That is somewhat counter-intuitive, but please
 see recent discussion on DHC WG about it.

 On the other hand, using only subnets is simpler. In some cases users may
 not care about confirm (e.g. fixed networks like broadband), and can use
 different mechanism (like interface-id or remote-id) to match clients with
 their pools.

 > I believe there is one place where the current arrangement isn't
 strictly hierarchical but I can't remember it offhand.  In any case we
 should try and make any hierarchy clean and clear.  Currently it is the
 case that even if you define some things within a restricted range (pool,
 subnet etc) the affects are global.  People regularly run into this as a
 problem.
 There is another possible hierarchy that is not present in DHCP4:
 interface. Dibbler users like the ability to simply say "serve pool X over
 interface Y". Such configuration is very simple. We need to embrace that
 idea anyway, as DHCPv6 server do not need to have global addresses, so it
 is not possible to detect server's attachment based on subnet analysis.

 > ''Do we want to store information in human readable form''
 > Stephen's comment seems reasonable.  I would add that in case of any
 inconsistency the binary data wins and the human readable is replaced.
 My understanding is that Stephen suggested to use binary form and convert
 it to human form only if needed (e.g. to log if logging is enabled).

 > ''Network topology''
 > I don't think we currently need a complete topology.  If I'm running the
 server on a machine with 3 interfaces but only provide it one via the
 command line it seems to be happy with a configuration that only describes
 that interface.

 > We do need to supply information about an interface even if there won't
 be any clients using it directly though.
 That is what I was referring to by saying complete topology. That does not
 work for v6. Server does not have to have global addresses at all.

 > As to the complexity I think we may need to write up what we need for
 (b) before making a final decision.  As a user I'd be unhappy if I needed
 to include specific interface to pool mappings.  That would seem to make
 it difficult to modify the server hardware without having to reconfigure
 the DHCP server.
 How are you going to replace failing NIC without restarting server?

 I see your point, though. Perhaps we can make interface definition
 optional? There are many simple cases, when it is useful and there are
 complicated cases when it is not.

 > ''DNS Updates''
 > See previous comments on DDNS.  If we go with a separate module handling
 all of this then all these issues move to there.
 See my comments above. This separate module will still use DB and
 configuration store. Thus we need to design data layout.

 > From the scratchpad section
 > ''Are there requirements on detecting overlapping pools or other
 inconsistencies''
 > My first though is that it would be part of the configuration code to
 figure out if something is inconsistent before it can be put into the
 lease database - assuming that some configuration information is going
 into the DB.  Or were you thinking about bugs in the server causing it to
 try and write inconsistent information?
 I really haven't given much thought on this. This question was also asked
 in hook context. What if hook overrides our proposed lease with something
 that does not match current configuration. Do we implement "user knows
 best" strategy?

 It is tempting to say that any changes to lease and host data must be
 consistent with current configuration of pools. But do we want the
 opposite as well? What happens when user removes pool that has active
 leases in it? Do we keep the leases or toss them immediately? It would be
 better to keep them and remove them the next time client renews. (Removing
 a lease will trigger DDNS removal, while client is convinced that his
 lease is still valid and would be surprised that its FQDN stops
 resolving).

-- 
Ticket URL: <http://bind10.isc.org/ticket/1634#comment:10>
BIND 10 Development <http://bind10.isc.org>
BIND 10 Development