<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<div class="default-style" style="">Hi, I'm trying to set up a hot-standby redundant dhcp server for my home network and I'm having a real struggle trying to get the kea-dhcp4 daemon to find and load the libdhcp_ha.so library hook so that it can run in HA mode.</div>
<div class="default-style" style="">
It's working great when I don't ask it to load the HA library hook, but I'm not having any luck with loading the .so. When I try to start the official ubuntu-package-provided kea-dhcp4-server systemd service, it reports the following:
<br>
ERROR HOOKS_OPEN_ERROR failed to open hook library /usr/local/lib/aarch64-linux-gnu/kea/hooks/libdhcp_nosuchfile.so: /usr/local/lib/aarch64-linux-gnu/kea/hooks/libdhcp_nosuchfile.so: cannot open shared object file: No such file or directory
<br>
<br>
</div>
<div class="default-style" style="">I believe the issue being reported by the service in my system logs is actually something other than a failed read operation on the listed path, and I can explain my reasoning for drawing this conclusion.</div>
<div class="default-style" style="">I am hoping someone with the patience to read all this can suggest something I should try that I haven't thought of yet, or point out an error that I've made, that will help me get my HA pair online! thanks in advance for your attention.</div>
<div class="default-style" style="">
<br>
Here's a list of the things I have tried so far to fix this issue in approximate chronological order:
</div>
<div class="default-style" style="">
<ul>
<li>I manually confirmed that the file exists, by copying the full file name and path directly from the error message and pasting it into the terminal. the file exists.
<br>
<br></li>
<li>checked the unix file permissions on the .so itself (at /usr/local/lib/aarch64-linux-gnu/kea/hooks/libdhcp_sh.so), and also of each directory in that file path all the way down to /usr.
<br>
all have mode o+r set, and I have confirmed that I can read the contents of the file from a shell running as the kea user (_kea on my system). it is not a file permissions error.
<br>
<br></li>
<li>checked for enhanced access control subsystems. selinux is not installed on this server. apparmor is installed, and when I checked my kernel log I found that apparmor was reporting some blocked access to the library hook file in question.
<br>
I addressed these by updating the apparmor profile for /usr/sbin/kea-dhcp4, which required adding explicit permits for the r and m masks.
<br>
I made these changes by hand at first, then reset and used the aa-genprof tool from the official apparmor-utils package which generated the same set of changes I had applied by hand. These changes fixed the apparmor permissions conflict that was reported in my kernel log.
<br>
I confirmed that apparmor was no longer responsible for blocking access to the file by shutting off apparmor protections system-wide, which did not fix the issue.
<br>
<br></li>
<li>I copied the target .so file to a new directory under /etc/kea/lib/, and updated the service configs and the apparmor profile to point at this new file location.
<br>
The service continued to report that it cannot open the file because it does not exist, this time naming the new path under /etc/kea/lib/. I again copy-pasted the path from the error message to ensure I hadn't made any errors in my typing.
<br>
This didn't produce any positive results, so I reverted the configs to point to the default path to avoid unnecessary complication in further debugging.
<br>
<br></li>
<li>I found the presented troubleshooting advice in the official kea-dhcp4-server install guide from ISC, which indicates that in cases exactly like the one I was dealing with, one should run ldconfig with root permissions to update the linker cache. I did this.
<br>
I checked that the directory path containing the library was listed in a file under /etc/ld.so.conf.d/. I checked that /etc/ld.so.conf.d/ was correctly included in /etc/ld.so.conf. I ran ldconfig again as root while explicitly passing the directory immediately containing the .do file.
<br>
I ran ldconfig -p | grep to confirm that the desired shared object file was present in the cache, and that it was indexed at the expected file path, which it was.
<br>
Out of pure superstition, I even rebooted the server and checked again -- still present in the cache at the correct path.
<br>
<br></li>
<li>I opened the systemd service unit file to look for clues about any restricted filesystem access, system call capabilities, or anything else of this nature. I didn't find any in the unit file for this service.
<br>
<br></li>
<li>I explicitly added LD_LIBRARY_PATH=/usr/local/lib/aarch64-linux-gnu/kea/hooks to the service's environment variables exposed by systemd, then reloaded the systemd daemon to read the unit file changes. no help.
<br>
<br></li>
<li>I inserted strace into the unit file exec line to report any system calls related to the target file, reloaded the systemd daemon, and attempted to start the service again.
<br>
For comparison, I edited the service config at /etc/kea/kea-dhcp4.conf and the strace invocation in the service unit file to look for a known-bogus .so file in the same directory, reloaded the daemon again, and started the service again.
<br>
Comparing the strace output from the two calls, there is a clear difference between the system calls and results reported by strace. In the output from the service start with a known-bad file path, strace reports a single openat() system call to the expected file, which as expected returns -1 with errno ENOENT.
<br>
By comparison, the strace output from the service start with configs pointing to the real .so file which should be accessible to the service shows 6 system calls related to the .so file in question: openat(), read(), fstat(), mmap(), mmap(), close(). none of these system calls are shown to have errored with an associated errno code, or to have returned with a negative value.</li>
</ul>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;">At this point I'm not really sure where to go, I am intimidated by the prospect of trying to dive into the kea-dhcp codebase and figure out what is reporting an ENOENT-related error message when there (seemingly, to my eyes) doesn't seem to be any such error occurring.</div>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;"> </div>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;">
I am using binaries and config files as provided from the Ubuntu repos (24.04, arm64), running on a raspberry pi 5, unmodified except as mentioned above and for miscellaneous config changes to /etc/kea/kea-dhcp4.conf to suit the needs of my network.
<br>
<br>
The debian packages I'm using are as follows:
<br>
ii kea-admin 2.4.1-3ubuntu0.1 arm64 Administration utilities for Kea DHCP server
<br>
ii kea-common 2.4.1-3ubuntu0.1 arm64 Common libraries for the Kea DHCP server
<br>
ii kea-dev 2.4.1-3ubuntu0.1 arm64 Development headers for Kea DHCP server
<br>
ii kea-dhcp4-server 2.4.1-3ubuntu0.1 arm64 IPv4 DHCP server
<br>
<br>
The kea version number reported by the binary that shipped with the above packages is 3.1.5
<br>
<br>
I am putting a copy of some relevant terminal buffers and config files up on my public webserver for reference, links to follow:
<br>
<br>
(config files doctored to target a nonexistent libdhcp_nosuchfile.so)
<br>
<a href="https://pdxlents.online/kea-dhcp4.conf.bad_configs.txt">https://pdxlents.online/kea-dhcp4.conf.bad_configs.txt</a>
<br>
<a href="https://pdxlents.online/systemd_unit_file.bad_configs.txt">https://pdxlents.online/systemd_unit_file.bad_configs.txt</a>
</div>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;">
<a href="https://pdxlents.online/terminal_log.bad_configs.txt">https://pdxlents.online/terminal_log.bad_configs.txt</a>
<br>
<br>
(config files targeting an actually existing libdhcp_ha.so)
</div>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;">
<a href="https://pdxlents.online/kea-dhcp4.conf.good_configs.txt">https://pdxlents.online/kea-dhcp4.conf.good_configs.txt</a>
<br>
<a href="https://pdxlents.online/terminal_log.good_configs.txt">https://pdxlents.online/terminal_log.good_configs.txt</a>
</div>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;">
<a href="https://pdxlents.online/systemd_unit_file.good_configs.txt">https://pdxlents.online/systemd_unit_file.good_configs.txt</a>
<br>
<br>
(version output from my server binary)
</div>
<div class="default-style" style="font-family: -apple-system, BlinkMacSystemFont, helvetica, sans-serif;">
<a href="https://pdxlents.online/kea-dhcp4_-V.txt">https://pdxlents.online/kea-dhcp4_-V.txt</a>
<br>
<br>
Thanks again for reading, and for any advice or suggestions!
</div>
</div>
</body>
</html>