[Kea-users] Intermittent "Address already in use" Error When Creating DHCP Reservations (Micetro 25.2.1, Kea 2.6.4)

Zenz, Thomas Thomas.Zenz at oenb.at
Thu Jul 16 12:27:33 UTC 2026 

Hi,

you are right, I do have HA and KeaCtrl running. But because of your Documentation, I carefuly checked to use different Port. See my config snips below. Furter down I copied the log during the "Failed State" and the output of the HA Status.
I had to use the same user and password for both services, and could not user ssl because micetro did not support ssl.

        "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
        "parameters": {
          "high-availability": [
            {
              "delayed-updates-limit": 0,
              "heartbeat-delay": 10000,
              "max-ack-delay": 5000,
              "max-rejected-lease-updates": 10,
              "max-response-delay": 60000,
              "max-response-time": 20000,
              "max-unacked-clients": 0,
              "mode": "hot-standby",
              "multi-threading": {
                "enable-multi-threading": true,
                "http-client-threads": 4,
                "http-dedicated-listener": true,
                "http-listener-threads": 4
              },
              "peers": [
                {
                  "auto-failover": true,
                  "basic-auth-password": "****",
                  "basic-auth-user": "****",
                  "name": "server35",
                  "role": "primary",
                  "url": "http://169.254.67.35:8001/"
                },
                {
                  "auto-failover": true,
                  "basic-auth-password": "****",
                  "basic-auth-user": "****",
                  "name": "server36",
                  "role": "standby",
                  "url": "http://10.15.21.36:8001/"
                }
              ],
              "require-client-certs": true,
              "restrict-commands": false,
              "send-lease-updates": true,
              "sync-leases": true,
              "sync-page-limit": 10000,
              "sync-timeout": 60000,
              "this-server-name": "anlpn35",
              "wait-backup-ack": false
            }
          ]
        }
-----------------------------

"Control-agent": {
    "http-host": "0.0.0.0",
    "http-port": 8000,
#    "trust-anchor": "/etc/kea/secure/cert.pem",
#    "cert-file": "/etc/kea/secure/cert.pem",
#    "key-file": "/etc/kea/secure/key.pem",
    "cert-required": false ,
    "authentication": {
      "type": "basic",
      "realm": "kea-control-agent",
      "clients": [
        {
          "user": "****",
          "password": "****"
        }
      ]
    },

    "control-sockets": {
        "dhcp4": {
            "socket-type": "unix",
            "socket-name": "/var/run/kea/kea4-ctrl-socket"
        },
        "dhcp6": {
            "socket-type": "unix",
            "socket-name": "/var/run/kea/kea6-ctrl-socket"
        },
        "d2": {
            "socket-type": "unix",
            "socket-name": "/var/run/kea/kea-ddns-ctrl-socket"
        }
    },

    "hooks-libraries": [
    ],

    "loggers": [
    {
        "name": "kea-ctrl-agent",
        "output_options": [
            {
                "output": "/var/log/kea/kea-ctrl-agent.log"
            }
        ],
        "severity": "INFO",
        "debuglevel": 0
    }
  ]
}
}


2026-07-16 14:00:11.164 INFO [ddns-script/e596b862] leases4_committed for e2:dd:b8:c5:ec:80 dhcp-e2-dd-b8-c5-ec-80 172.20.70.109
2026-07-16 14:00:11.196 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'config-hash-get'
2026-07-16 14:00:11.393 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'config-test'
2026-07-16 14:00:11.393 INFO  [kea-dhcp4.hosts/35.139655246579840] HOSTS_BACKENDS_REGISTERED the following host backend types are available: mysql postgresql
2026-07-16 14:00:11.394 WARN  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
2026-07-16 14:00:11.394 WARN  [kea-dhcp4.dhcp4/35.139655246579840] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
2026-07-16 14:00:11.394 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_lease_cmds.so successfully closed
2026-07-16 14:00:11.394 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_run_script.so successfully closed
2026-07-16 14:00:11.394 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_ha.so successfully closed
2026-07-16 14:00:11.395 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 169.254.67.0/24 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.395 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.0.0/16 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.395 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.11.0.0/16 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.395 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.12.0.0/16 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.395 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 192.168.1.0/24 with params: t1=900, t2=1800, valid-lifetime=1209600
...
2026-07-16 14:00:11.430 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.16.196.0/24 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.430 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
2026-07-16 14:00:11.430 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type udp
2026-07-16 14:00:11.436 INFO  [kea-dhcp4.dhcp4/35.139655144646208] DHCP4_QUERY_LABEL received query: [hwtype=1 58:38:79:a7:d5:ed], cid=[01:58:38:79:a7:d5:ed], tid=0x166b5d84
2026-07-16 14:00:11.609 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'config-set'
2026-07-16 14:00:11.614 INFO  [kea-dhcp4.hosts/35.139655246579840] HOSTS_BACKENDS_REGISTERED the following host backend types are available: mysql postgresql
2026-07-16 14:00:11.616 WARN  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
2026-07-16 14:00:11.616 WARN  [kea-dhcp4.dhcp4/35.139655246579840] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
2026-07-16 14:00:11.616 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_lease_cmds.so successfully closed
2026-07-16 14:00:11.616 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_run_script.so successfully closed
2026-07-16 14:00:11.616 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_ha.so successfully closed
2026-07-16 14:00:11.616 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 169.254.67.0/24 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.616 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.14.0.0/16 with params: t1=900, t2=1800, valid-lifetime=1209600
...
2026-07-16 14:00:11.652 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.16.196.0/24 with params: t1=900, t2=1800, valid-lifetime=1209600
2026-07-16 14:00:11.653 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
2026-07-16 14:00:11.653 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type udp
2026-07-16 14:00:11.654 INFO  [kea-dhcp4.ha-hooks/35.139655246579840] HA_DEINIT_OK unloading High Availability hooks library successful
2026-07-16 14:00:11.654 INFO  [kea-dhcp4.run-script-hooks/35.139655246579840] RUN_SCRIPT_UNLOAD Run Script hooks library has been unloaded
2026-07-16 14:00:11.654 INFO  [kea-dhcp4.lease-cmds-hooks/35.139655246579840] LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful
2026-07-16 14:00:11.655 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_ha.so successfully closed
2026-07-16 14:00:11.655 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_run_script.so successfully closed
2026-07-16 14:00:11.655 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_CLOSED hooks library /usr/lib64/kea/hooks/libdhcp_lease_cmds.so successfully closed
2026-07-16 14:00:11.656 INFO  [kea-dhcp4.lease-cmds-hooks/35.139655246579840] LEASE_CMDS_INIT_OK loading Lease Commands hooks library successful
2026-07-16 14:00:11.656 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_LOADED hooks library /usr/lib64/kea/hooks/libdhcp_lease_cmds.so successfully loaded
2026-07-16 14:00:11.656 INFO  [kea-dhcp4.run-script-hooks/35.139655246579840] RUN_SCRIPT_LOAD Run Script hooks library has been loaded
2026-07-16 14:00:11.656 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_LOADED hooks library /usr/lib64/kea/hooks/libdhcp_run_script.so successfully loaded
2026-07-16 14:00:11.657 INFO  [kea-dhcp4.ha-hooks/35.139655246579840] HA_CONFIGURATION_SUCCESSFUL HA hook library has been successfully configured
2026-07-16 14:00:11.657 INFO  [kea-dhcp4.ha-hooks/35.139655246579840] HA_INIT_OK loading High Availability hooks library successful
2026-07-16 14:00:11.657 INFO  [kea-dhcp4.hooks/35.139655246579840] HOOKS_LIBRARY_LOADED hooks library /usr/lib64/kea/hooks/libdhcp_ha.so successfully loaded
2026-07-16 14:00:11.657 INFO  [kea-dhcp4.dhcp4/35.139655246579840] DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: added IPv4 subnets: 254; DDNS: enabled
2026-07-16 14:00:11.681 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MEMFILE_DB opening memory file lease database: lfc-interval=3600 name=/var/lib/kea/dhcp4.leases persist=true type=memfile universe=4
2026-07-16 14:00:11.682 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file /var/lib/kea/dhcp4.leases.2
2026-07-16 14:00:11.757 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file /var/lib/kea/dhcp4.leases
2026-07-16 14:00:11.835 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 4409 leases, extended info sanity checks modified 0 / updated 0 leases and 0 leases have relay or remote id
2026-07-16 14:00:11.835 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to 3600 sec
2026-07-16 14:00:11.835 INFO  [kea-dhcp4.ha-hooks/35.139655246579840] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the anlpn35 is in the WAITING state
2026-07-16 14:00:11.836 INFO  [kea-dhcp4.ha-hooks/35.139655246579840] HA_SERVICE_STARTED anlpn35: started high availability service in hot-standby mode as primary server
2026-07-16 14:00:11.836 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 169.254.67.0/24
2026-07-16 14:00:11.836 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.14.0.0/16
...
2026-07-16 14:00:11.846 INFO  [kea-dhcp4.dhcpsrv/35.139655246579840] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.16.196.0/24
2026-07-16 14:00:11.874 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'config-get'
2026-07-16 14:00:12.037 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'ha-heartbeat'
2026-07-16 14:00:18.469 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'status-get'
2026-07-16 14:00:21.865 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_STATE_TRANSITION anlpn35: server transitions from WAITING to SYNCING state, partner state is WAITING
2026-07-16 14:00:21.865 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_LEASE_UPDATES_DISABLED anlpn35: lease updates will not be sent to the partner while in SYNCING state
2026-07-16 14:00:21.866 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_SYNC_START server35: starting lease database synchronization with server36 (http://10.15.21.36:8001/)
2026-07-16 14:00:22.074 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED server35: received 4409 leases from server36 (http://10.15.21.36:8001/)
2026-07-16 14:00:22.130 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_LEASES_SYNC_APPLIED_LEASES server35: applied 0 leases received from the partner in the local lease database
2026-07-16 14:00:22.140 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_SYNC_SUCCESSFUL server35: lease database synchronization with server36 (http://10.15.21.36:8001/) completed successfully in 274.218 ms
2026-07-16 14:00:22.140 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_STATE_TRANSITION server35: server transitions from SYNCING to READY state, partner state is WAITING
2026-07-16 14:00:22.140 INFO  [kea-dhcp4.ha-hooks/35.139655169824320] HA_LEASE_UPDATES_DISABLED server35: lease updates will not be sent to the partner while in READY state
2026-07-16 14:01:18.499 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'status-get'
2026-07-16 14:01:29.749 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'status-get'
2026-07-16 14:02:18.525 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'status-get'
2026-07-16 14:02:25.643 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'ha-heartbeat'
2026-07-16 14:03:18.551 INFO  [kea-dhcp4.commands/35.139655246579840] COMMAND_RECEIVED Received command 'status-get'
:
[ { "arguments": { "date-time": "Thu
 16 Jul 2026 12:05:21 GMT"
 "scopes": [  ]
 "state": "ready"
 "unsent-update-count": 0 }
 "result": 0
 "text": "HA peer status returned." } ]



-----Ursprüngliche Nachricht-----
Von: Kea-users <kea-users-bounces at lists.isc.org> Im Auftrag von Darren Ankney
Gesendet: Samstag, 11. Juli 2026 17:11
An: Kea user's list <kea-users at lists.isc.org>
Betreff: Re: [Kea-users] Intermittent "Address already in use" Error When Creating DHCP Reservations (Micetro 25.2.1, Kea 2.6.4)


HINWEIS: Externer Absender


Hi Tom,

This sounds like you may have the HA hook and the kea-ctrl-agent attempting to use the same IP/port combo.  This creates a race condition where sometimes kea-dhcp4 or kea-dhcp6 wins and sometimes kea-ctrl-agent wins.  This assumes you have the parameters "enable-multi-threading" and "http-dedicated-listener" set true (which is the default in 2.6.x). Confirm that your IP address / port specified in your kea-ctrl-agent configuration do not match the "url"
specified in the HA configuration for each peer in that case.

More information:
https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.6.4/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!MnVZWQ!olxlPMyZuFhSqoRAoK3-j0dW2FXkQHgZmwF1HOZ2s2xmpedFVHVx7ADgOYXwJuH1DFSjx8PCItEjepj3jwAv0FGTTg$

Thank you,
Darren Ankney

On Fri, Jul 10, 2026 at 4:01 AM Zenz, Thomas via Kea-users <kea-users at lists.isc.org> wrote:
>
> Dear ISC KEA Community,
>
> We’re encountering an intermittent issue when creating DHCP reservations in our Micetro environment (version 25.2.1) running on Kea (2.6.4) in a high-availability (HA) setup.
>
> The same request succeeds at times but fails unpredictably with the following server response:
>
>
>
> {'error': {'code': 5376, 'message': 'Error creating a reservation:
> Unable to update the configuration due to an error: Error initializing
> hooks: CmdHttpListener::run failed: unable to setup TCP acceptor for
> listening to the incoming HTTP requests: bind: Address already in
> use.'}}
>
>
>
> The HTTP response code is 400 (Bad Request), which is misleading since the request itself is valid (identical requests succeed at other times).
>
>
>
> Additional observations:
>
> During the failure, the primary server appears to be in standby mode (as if it were the secondary).
> The only workaround we’ve found is manually running keactrl reload on the primary server to restore normal operation.
>
> Our logs suggest the issue stems from a resource conflict (e.g., TCP port exhaustion or a lingering listener), but we’re unsure how to diagnose or resolve it.
>
> Has anyone encountered this behavior before? Any insights or troubleshooting steps would be greatly appreciated.
>
> Thank you in advance!
>
>
>
> Tom
>
>
> Diese E-Mail kann vertrauliche Informationen enthalten und irrtümlich an Sie gelangt sein. In diesem Fall informieren Sie bitte sofort die Absenderin bzw. den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder von Teilen dieser Mail sind nicht gestattet.
>
> Informationen gemäß DSGVO zur Verarbeitung personenbezogener Daten durch die OeNB finden Sie unter http://www.oenb.at/datenschutz .
> --
> ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!MnVZWQ!olxlPMyZuFhSqoRAoK3-j0dW2FXkQHgZmwF1HOZ2s2xmpedFVHVx7ADgOYXwJuH1DFSjx8PCItEjepj3jwCNLJsp7g$  for more information.
>
> To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!MnVZWQ!olxlPMyZuFhSqoRAoK3-j0dW2FXkQHgZmwF1HOZ2s2xmpedFVHVx7ADgOYXwJuH1DFSjx8PCItEjepj3jwBGv2LTxA$ .
> Kea-users at lists.isc.org
--
ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!MnVZWQ!olxlPMyZuFhSqoRAoK3-j0dW2FXkQHgZmwF1HOZ2s2xmpedFVHVx7ADgOYXwJuH1DFSjx8PCItEjepj3jwCNLJsp7g$  for more information.

To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!MnVZWQ!olxlPMyZuFhSqoRAoK3-j0dW2FXkQHgZmwF1HOZ2s2xmpedFVHVx7ADgOYXwJuH1DFSjx8PCItEjepj3jwBGv2LTxA$ .
Kea-users at lists.isc.org
_________________
Diese Nachricht wurde vom OeNB Mailserver TLS verschlüsselt empfangen

Diese E-Mail kann vertrauliche Informationen enthalten und irrtümlich an Sie gelangt sein. In diesem Fall informieren Sie bitte sofort die Absenderin bzw. den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail oder von Teilen dieser Mail sind nicht gestattet.

Informationen gemäß DSGVO zur Verarbeitung personenbezogener Daten durch die OeNB finden Sie unter www.oenb.at/datenschutz<https://www.oenb.at/datenschutz>.

More information about the Kea-users mailing list