Add scope without service restart, possible? (Follow-up)
Steve van der Burg
Steve.vanderBurg at lhsc.on.ca
Fri May 29 20:13:15 UTC 2020
(Warning, long message ahead)
We have a large network here, connected together with Cisco gear that I don’t run or have access to. The network folks have configured it to forward DHCP broadcast messages from clients to our two DHCP servers (the failover pair). It’s my understanding that anything that the clients do by unicast (ie. where they send a request (an INFORM, renew, etc) to the server that granted them the lease) they will fall back to broadcast for. So that covers anything urgent that might happen while one server is down. And most things aren’t urgent – clients start trying to renew halfway through their leases, so they’ll just try again a few times if a server doesn’t answer.
Anyway, I reviewed my code (that I developed about 15 years ago and have treaked a few times since) and I’m not actually pushing. Each DHCP server polls my central admin server (another linux server) once a minute. I could certainly push the configs, though. It’s a pretty minor detail, really.
For the polling I even pull the configs off of a web server running on the admin server. If I were starting over today, I would just use sftp or scp with key auth and pull (or push from the admin server).
Anyway, my ‘download config and maybe restart’ code is all perl. The config gets tarred up on the admin server and includes a small ‘main’ config, two peer configs (things specific to each peer) and one large, included config (generated by code out of another system that I set up).
On each DHCP server, this runs once a minute by cron. The test for which minute to run on is simple:
# To ensure that both servers aren't down at the same time, we run on odd minutes on the secondary
# and even minutes on the primary.
#
if ( -e "$spath/dhcpd.i.am.secondary" ) {
exit if (localtime)[1] % 2 == 0;
$me = "dhcp2";
$peer = "dhcp1";
}
else {
exit if (localtime)[1] % 2 == 1;
$me = "dhcp1";
$peer = "dhcp2";
}
The secondary server just has one file on it (“dhcpd.i.am.seconday”) that the primary doesn’t. I could also test for the hostname or something just as easily. The $me/$peer stuff is there to let the script manipulate the config later.
I grab the config from the web server (not shown here – a 1-liner using perl’s LWP package) and then do a check to ensure that something weird hasn’t happened that shrank the config a lot, and also see if the new tar file is different than the last one that I saved a copy of (the one that became the current running config):
exit unless -e $cf; # get out if there's no new file
unless ( -e $of ) {
`/bin/cp $cf $of`; # create the older version if it never existed
}
exit if (-s $cf) / (-s $of) < 0.5; # get out if the file has gotten a lot smaller
# Do MD5 hashes on both tar files (new one, and the one that holds the current production files):
#
my ($od,$nd);
if ( open(FO,$of) && open(FN,$cf) ) {
$od = Digest::MD5->new->addfile(*FO)->hexdigest;
$nd = Digest::MD5->new->addfile(*FN)->hexdigest;
close FO;
close FN;
}
exit if $nd eq $od && (-s $of) == (-s $cf); # if the MD5 hashes and sizes are the same, we can quit now.
If what was sent is different, I untar it (and for some reason run MD5 hashes on the pieces against the current pieces – no idea why, really) – not shown here.
After untarring, the $me/$peer stuff gets used to select the correct small config piece specific to this server and trash the other one:
rename "$me.peer.conf","dhcpd.peer.conf";
unlink "$peer.peer.conf";
The small ‘main’ config contains the include statements for the other pieces, but I have untarred the new one, for testing, into /some/test/path, so I need to rewrite those includes:
# Create a copy of the root config that we can test with:
#
if ( open(MCF,"dhcpd.conf") && open(NCF,">dhcpd.test.conf") ) {
while (<MCF>) {
s#include\s+"/etc/dhcp#include "$extpath#;
print NCF $_;
}
close MCF;
close NCF;
}
Now I can test the new config:
my $testres = `/usr/sbin/dhcpd -t -cf $extpath/dhcpd.test.conf 2>&1`;
my $goodconf = 1;
$goodconf = 0 if $testres =~ /Configuration file errors encountered/;
If it passes, I copy it into the right place (not shown), wait until near the end of the current minute and then stop the server. Another check, run by cron once a minute, will start it again a few seconds later. And even with thousands of subnets and hundred of thousands of leasable addresses, ISC’s dhcpd still starts in under 5 seconds (probably way under) on my servers. Again, a lot of this is from a long time ago. I should just let systemd instantly restart dhcpd once I stop it.
while ( 1 ) {
last if (localtime)[0] > 57;
sleep 1;
}
`/usr/sbin/service isc-dhcp-server stop`;
And that’s it. Over on the admin server, something else creates the huge set of subnet stanzas (99% of the overall config), tars it up with the static pieces and places it where this script can get it. So a changed config written there is loaded and live on both servers within a couple of minutes. And each one emails me (also in the script and not shown here) telling me about a successful restart or a failure of the config test shown above.
…Steve
From: dhcp-users <dhcp-users-bounces at lists.isc.org> On Behalf Of Cayab, Jefrey E.
Sent: Friday,May 29,2020 12:28 PM
To: dhcp-users at lists.isc.org
Subject: Re: Add scope without service restart, possible? (Follow-up)
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Thank you all for the very informative replies.
I have to admit that I don't have experience with dual servers with failover setup so if you could share more info, that'll be really great.
Maybe not only for me but for others also.
But just wanted to get this off my mind - so if I have 2 servers in failover mode, I could update configs in the Primary server and restart dhcpd service on it but will not have service downtime to DHCP clients (existing and new)?
@Steve van der Burg, really interested to know more how you push new configs to both servers (manual or automated?) and then the servers test them and, if they pass, restart on alternate minutes.
@Patrick Trapp, I really think a lot of us here could benefit from your workflow.
Thank you
Jef
On Sat, May 23, 2020 at 5:44 AM <dhcp-users-request at lists.isc.org<mailto:dhcp-users-request at lists.isc.org>> wrote:
Send dhcp-users mailing list submissions to
dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/dhcp-users
or, via email, send a message with subject or body 'help' to
dhcp-users-request at lists.isc.org<mailto:dhcp-users-request at lists.isc.org>
You can reach the person managing the list at
dhcp-users-owner at lists.isc.org<mailto:dhcp-users-owner at lists.isc.org>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of dhcp-users digest..."
Today's Topics:
1. Re: Add scope without service restart, possible? (Simon Hobson)
2. RE: Add scope without service restart, possible?
(Steve van der Burg)
3. Re: Add scope without service restart, possible? (Patrick Trapp)
----------------------------------------------------------------------
Message: 1
Date: Fri, 22 May 2020 20:22:14 +0100
From: Simon Hobson <dhcp1 at thehobsons.co.uk<mailto:dhcp1 at thehobsons.co.uk>>
To: Users of ISC DHCP <dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>>
Subject: Re: Add scope without service restart, possible?
Message-ID: <B329B452-5475-4929-B042-85BAAFCE3CAB at thehobsons.co.uk<mailto:B329B452-5475-4929-B042-85BAAFCE3CAB at thehobsons.co.uk>>
Content-Type: text/plain; charset=us-ascii
Cayab, Jefrey E. <jcayab at gmail.com<mailto:jcayab at gmail.com>> wrote:
> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?
Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.
Simon
------------------------------
Message: 2
Date: Fri, 22 May 2020 19:30:10 +0000
From: Steve van der Burg <Steve.vanderBurg at lhsc.on.ca<mailto:Steve.vanderBurg at lhsc.on.ca>>
To: Users of ISC DHCP <dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>>
Subject: RE: Add scope without service restart, possible?
Message-ID:
<YQXPR01MB2326AD7B94E1161FC4EE51BFBAB40 at YQXPR01MB2326.CANPRD01.PROD.OUTLOOK.COM<mailto:YQXPR01MB2326AD7B94E1161FC4EE51BFBAB40 at YQXPR01MB2326.CANPRD01.PROD.OUTLOOK.COM>>
Content-Type: text/plain; charset="us-ascii"
I run a pair of servers with all leases (~40k active right now, out of ~370k defined) in failover pools and routinely change the configs and restart with no interruption to service, as Simon just indicated.
I push new configs to both servers and then the servers test them and, if they pass, restart on alternate minutes (primary on even minutes, for example, and secondary on odd minutes).
...Steve
-----Original Message-----
From: dhcp-users <dhcp-users-bounces at lists.isc.org<mailto:dhcp-users-bounces at lists.isc.org>> On Behalf Of Simon Hobson
Sent: Friday,May 22,2020 3:22 PM
To: Users of ISC DHCP <dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>>
Subject: Re: Add scope without service restart, possible?
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Cayab, Jefrey E. <jcayab at gmail.com<mailto:jcayab at gmail.com>> wrote:
> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?
Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.
Simon
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
dhcp-users mailing list
dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>
https://lists.isc.org/mailman/listinfo/dhcp-users
This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.
------------------------------
Message: 3
Date: Fri, 22 May 2020 21:44:15 +0000
From: Patrick Trapp <ptrapp at nex-tech.com<mailto:ptrapp at nex-tech.com>>
To: Users of ISC DHCP <dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>>
Subject: Re: Add scope without service restart, possible?
Message-ID:
<DM6PR12MB3577A9078E41E04A79D0A2D9E6B40 at DM6PR12MB3577.namprd12.prod.outlook.com<mailto:DM6PR12MB3577A9078E41E04A79D0A2D9E6B40 at DM6PR12MB3577.namprd12.prod.outlook.com>>
Content-Type: text/plain; charset="us-ascii"
I can second that. We run a pair of servers plus a staging server. We make changes and test the code on the staging server. If it passes, we sync to a git server. We run a script that causes the production servers to pull the new config from git and restart, one at a time.
________________________________
From: dhcp-users <dhcp-users-bounces at lists.isc.org<mailto:dhcp-users-bounces at lists.isc.org>> on behalf of Steve van der Burg <Steve.vanderBurg at lhsc.on.ca<mailto:Steve.vanderBurg at lhsc.on.ca>>
Sent: Friday, May 22, 2020 2:30 PM
To: Users of ISC DHCP <dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>>
Subject: RE: Add scope without service restart, possible?
CAUTION: This email originated from outside of the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I run a pair of servers with all leases (~40k active right now, out of ~370k defined) in failover pools and routinely change the configs and restart with no interruption to service, as Simon just indicated.
I push new configs to both servers and then the servers test them and, if they pass, restart on alternate minutes (primary on even minutes, for example, and secondary on odd minutes).
...Steve
-----Original Message-----
From: dhcp-users <dhcp-users-bounces at lists.isc.org<mailto:dhcp-users-bounces at lists.isc.org>> On Behalf Of Simon Hobson
Sent: Friday,May 22,2020 3:22 PM
To: Users of ISC DHCP <dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>>
Subject: Re: Add scope without service restart, possible?
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Cayab, Jefrey E. <jcayab at gmail.com<mailto:jcayab at gmail.com>> wrote:
> Wanted to check if it's possible to add new DHCP scope to a running DHCPd service and to take effect without restart the service? Is there a reload equivalent that can take in the updated config?
Not with the ISC server. However, a restart should be very quick for any reasonable configuration - and if you use dual servers with failover, then no interruption in service to end devices.
Simon
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
dhcp-users mailing list
dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>
https://lists.isc.org/mailman/listinfo/dhcp-users
This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
dhcp-users mailing list
dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>
https://lists.isc.org/mailman/listinfo/dhcp-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20200522/56f5cb30/attachment.htm>
------------------------------
Subject: Digest Footer
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
dhcp-users mailing list
dhcp-users at lists.isc.org<mailto:dhcp-users at lists.isc.org>
https://lists.isc.org/mailman/listinfo/dhcp-users
------------------------------
End of dhcp-users Digest, Vol 139, Issue 8
******************************************
This email is directed in confidence solely to the person named above and may contain confidential, privileged or personal health information. Please be aware that this email may also be released to members of the public under Ontario's Freedom of Information and Protection of Privacy Act if required. Review, distribution, or disclosure of this email by anyone other than the person(s) for whom it was originally intended is strictly prohibited. If you are not an intended recipient, please notify the sender immediately via a return email and destroy all copies of the original message. Thank you for your cooperation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20200529/5ec16bc1/attachment-0001.htm>
More information about the dhcp-users
mailing list