config help - scaling problem

[David W. Hankins]

On Mon, Mar 01, 2010 at 09:56:14AM -0600, Marc Perea wrote:
> I know there are quite a few helpful and ingenuitive folk who follow this list, and I'm hoping anyone can provide some insight on how I might better optimize my dhcpd.conf and/or system performance. My problem is that today it takes over a minute from when I type dhcpd until the daemon finishes loading into memory and is ready to service DHCP requests.

What version are you running?  Most of the folks reporting long
startup delays reported a major drop in upgrading past 3.0.x.

Most of the startup delay was in reading and reshuffling the
dhcpd.leases file, not the dhcpd.conf file.

> Additionally, when we perform periodic maintenance or upgrades, there are times when we require all clients to DHCP again, since our BRAS builds sessions and maintains state based on the DHCP process. Our Solaris Sun-fire V240 struggles to service anywhere from 1 to 5 DORA / second - with 10 thousand client requests to fulfill, you can see that it takes well over an hour before all clients get serviced.

You're using dynamic leases, so this isn't surprising.

ISC DHCP performance right now is coupled directly to fsync() rate.  It
is the Requests/s that are the major factor; in our interpretation of
non-normative language in RFC 2131, there must be an fsync() on the
database's file descriptor before any reply is sent on the wire.

Christof Chen submitted a patch that implements a "delayed fsync()", I
think that was in 4.1.0.

It changes ISC DHCP behaviour so that instead of one DHCPREQUEST per
fsync(), it queues multiple DHCPREQUEST-replies (and may make other
message replies without waiting), fsync()'s once for all of them, and
then transmits the group of replies all at once (within obvious limits
and timeouts).  I think we may have broken this feature in 4.2.0 (and
haven't fixed it yet) due to our changes to use the BIND export
library's event dispatcher system, but you can try 4.1.1.  We left it
compiled-out by default (there is a configure flag for it) because I
couldn't get it to test clean in our test failover system harness;
since then, I've had reports from folks that it works fine when using
failover, and that they've been doing so for months, so YMMV.

> Requirements: The business rules dictate that we provide a static IP to each client, and each client is identified by their unique, controlled and trusted agent.circuid-id inserted by company equipment. MAC is unusable for us due to spoofing concerns.

If you don't require to maintain state on the assigned leases, the
best thing, as was suggested, is the new host record "option matching"
semantic in 4.2.0.

Because host records are fixed, there's no reason to store changes to
them to keep track of expiry and the like, so there is zero
statekeeping and thus no disk hit.  Because there's no statekeeping,
there's no reason to share or negotiate them over something like a
failover channel; all you have to do is configure your (N) servers
consistently.

If you need to maintain that free/active/expired/free/...state, then
you become disk-bound.

> My first question - should I, and maybe could I, be using sub-classes instead of classes? Would that help me any?

Unfortunately, sub-classes can't be matched uniquely on an ACL; you
can only use the sub-class to match a class that can be matched in an
ACL...so it solves a reverse problem where you want to be able to
match many clients in one ACL statement.

Further unfortunately, because classes match on an executable boolean
statement (and not a data hash like subclasses use), searching them is
a /linear search/, and the software will search the entire list every
time, executing their boolean expressions individually.

Whatever you do, don't try to make your classes so that they have one
subclass member.  This still searches all the classes linearly, and
worse now allocates a hash table per class entry.

> Next, I modified syslog and added the hyphen so that (I hope) I am doing asynchronous logging to improve performance. Is there anything else I could do to enhance my server performance?
>  
> I also just implemented a ramdisk to store the dhcpd.conf and dhcpd.leases files, and invoking them with -cf and -lf hoping to improve performance as much as I can.

If you stick with dynamic (stateful) leases,

Because ISC DHCP is essentially disk-bound on DHCPREQUEST-replies,
it's my suggestion you not focus on CPU performance, but rather on
storage system performance.

Any storage system can probably keep up with ISC DHCP if it had e.g. a
battery-backed write-back cache.  Barring that, higher rates of
revolution statistically reduce the wait time for an fsync() to
complete.

I wouldn't really recommend using a ramdisk long term, as your server
will lose its database on every boot.  Old copies of the database are
usually worse than losing the database entirely.

But if you solve that performance bottleneck or move away from dynamic
leases to fixed, stateless assignments, then basically you become CPU
and network I/O bound (hopefully mostly CPU bound).

In this case avoiding long linear searches is desirable, so something
would have to be done about those class statements.

-- 
David W. Hankins	BIND 10 needs more DHCP voices.
Software Engineer		There just aren't enough in our heads.
Internet Systems Consortium, Inc.		http://bind10.isc.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20100301/eed6f603/attachment.bin>