DHCP Failover Complexity?

David W. Hankins David_Hankins at isc.org
Thu May 21 22:36:58 UTC 2009


On Thu, May 21, 2009 at 07:29:26PM +0100, Matt Causey wrote:
> What's the recommendation, then then for pool allocation?  We have at
> largest, /22 network allocations - with very close to 1000 devices
> attached.  We expect that a lease pool of most of that /22 should be
> available and shared between the dhcp servers.  For broadcast domains
> with this many hosts - should we be allocating more IP address space
> to give dhcp failover some breathing room?

At any given time, you basically want the number of free or backup
leases to be sufficient to ride out your average time to repair a
failover server (to move the survivor to partner-down or to get the
peer back online).  The variables are how rapidly you receive new
clients, how rapidly you can repair a dhcp server, and what threshold
for address starvation your local network has ("it's OK if the last
(n) people don't get online, they can wait and retry" vs "never").

I would guess that ~12 (~24/2) is probably not where I'd be
comfortable.  I'd suggest at least building a shared network with a
/25 or so on to that /22 (carrying about ~10% free at any given
time gives you ~1/20th of one day to repair-or-partner-down).

> > The situation is exhasperated because in failover, the server cannot
> > reallocate an expired lease until it has negotiated that expiration
> > with its peer.  This is because it cannot know what actions its peer
> > has taken while it was out of communication.
> 
> So, what does it mean if a single server in the pair crashes?

The preceeding paragraph means, to a server entering communications-
interrupted, that its current free-list is all it's got for new
clients until an operator attends to it, or until its peer is back.

The active leases will continue to get renewed after the clients
reach rebinding time, but will be unusable if they expire (as it is
unknown if the peer is extending them, they cannot be expired).

The surviving server's pool of free-state leases will become active,
and then become afflicted by the same rule as they too expire.

The optimization we don't implement allows a server essentially to
allocate an expired lease to the client it was last active with, and
to re-allocate to any client an expired lease that was last in the
local free-state.

In either case, that the server will run out of leases to allocate is
a mathematical certainty, and one cannot operate in communications-
interrupted indefinitely.  The only question is time.

>  Is it
> expected that SysAdmin intervene in order to make sure that the
> remaining server can serve leases for all clients at the site?

Essentially, yes, that is one of the assumptions of the failover
protocol.  This ties back to the protocol's insistence that it cannot
determine its partner is down merely because its connection is down.

> Well, perhaps we're doing something wrong.  But on our sites, if I
> cleanly (kill -15) take down a dhcp server in the pair, the other
> server goes to 'communications-interrupted'.  Are you saying that the
> expected behavior is that the remaining node should move to
> partner-down?

No, that is the correct behaviour.  Kill -15 isn't "clean," we have no
signal handlers.  I was mostly mentioning this to be complete; the
only clean shutdown is through omapi, but most people don't do it
that way, and I wouldn't recommend starting without some other
motivation.

> In any event, if a server in the pair goes down unexpectedly (i.e. the
> remaining server has no hope of gracefully moving to partner-down), we
> want the remaining server to service the leases owned by the other
> server without the need for an engineer to intervene.

'Ownership' is a funny word.  If you mean the active leases, yes, it
does that, but neither server can be said to own an active lease.  As
it turns out, the DHCPv4 clients will continue to query the original
server that gave them the lease for renewals, but this has nothing to
do with failover.

If you mean the free-state (free or backup) leases, failover can't
really help you here without creating situations where overlapping
allocations are made.

It is, essentially, the very point of failover's existence to avoid
these.

> > It's impossible to say.  If your servers are connected e.g. by a
> > heartbeat cable, then there's no reason why you shouldn't enter
> > partner-down immediately (and in fact, a feature to do this is in
> > review for 4.2.0).  A server in partner-down won't allocate leases
> > in the partner's free state until STOS+MCLT expires anyway.  The
> > 'risk' that there will be an addressing collision exists, but it
> > is next to zero (heartbeat cable fail), and you have MCLT seconds
> > to discover the condition and repair it anyway.
> 
> Can you elaborate on 'detect and repair' an address conflict?  Do you
> mean within dhcpd?

In this paragraph I was referring to discovering the failure of the
hypothetical heartbeat cable, or other "high degree of confidence"
interconnect between the failover servers.

You'd have MCLT seconds to conduct a repair before either server had
actually been in a position to give a conflicting allocation (where
one address is given to two different clients), as both servers will
refuse to allocate from its partner's pool until their own STOS+MCLT
timers expire.

As for resolving addressing conflicts, the method relies greatly upon
the clients involved.  Most often, the clients sort this out for
themselves as their users reboot or re-connect their NICs when they
are having problems and people in their immediate vicinity are not.

-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		     you'll just have to do it again."
Internet Systems Consortium, Inc.		-- Jack T. Hankins
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20090521/aad91573/attachment.bin>


More information about the dhcp-users mailing list