Communications Interrupted, Leases Stuck in Released / Expired

[Oliver Garraux]

Hey all,
I recently had an issue on a pair of ISC DHCP servers were not giving
out leases.  I think I understand what happened now, but I'd like to
run it by someone with a little more knowledge on this to make sure my
understanding is correct.
For some reason, a couple weeks ago the DHCPD daemon was shut down on
the secondary server.  I'm still not sure exactly why this happened.
Puppet also stopped running at some point.  So the primary server was
in the communications-interrupted state.
After that point, DHCP appeared to work as desired.  Earlier today
though, leases stopped being given out.  In syslog on the primary
server (the one that was still running), I see messages like this:
	DHCPDISCOVER from 40:61:86:e3:f7:7f via 192.168.25.1: peer holds all
free leases

After restarting dhcpd on the secondary server, DHCP leases are now
being given out.  I see that the primary server indeed had 0 free
addresses, and that after restarting the secondary server, it regains
it's share of the free IP's.
	balancing pool 7f7fd61d8a80 192.168.25.0/24  total 205  free 0
backup 100  lts -50  max-own (+/-)10	...		balancing pool 7f7fd61d8a80
192.168.25.0/24  total 205  free 102  backup 99  lts 1  max-own
(+/-)20
In the dhcpd.leases file (prior to bringing up the secondary server),
I see a bunch of leases that are released / expired, but have not
moved to "free".	  binding state released;	  next binding state
free;		  binding state expired;	  next binding state free;
These entries appear to start around the time that the secondary
server went offline a couple weeks ago.  Now (after bringing the
secondary back up), those leases are back in the "free" state.
I haven't been able to find a whole lot of documentation on what the
other lease states beyond "free" and "active" mean.  What I'm thinking
was happening, was the primary server continued to serve out leases
when the secondary server was down.  But, once those leases expired or
were released by the client, it could not return them to the pool of
"free" addresses, because it could not communicate this with it's
peer.  This is the part I'm curious about.  What is required for dhcpd
to move an address from released / expired and free?  Would not being
able to communicate with the partner inhibit this?
Ultimately, the issue here was that dhcpd was unintentionally shut
down on one of the servers, and I did not manually put the primary
server in a partner-down state. But I'd like to learn more about what
was going on.
Thanks,
Oliver
-------------------------------------

Oliver Garraux
Check out my blog:  www.GetSimpliciti.com/blog
Follow me on Twitter:  twitter.com/olivergarraux