[Kea-users] High availability with a database cluster issue
Sébastien Nauer
snauer at hotcity.lu
Thu Jan 9 09:51:43 UTC 2025
Hello Daren,
Thank you for your comments.
We use MariaDB v11.4.3 which seems to be fully compatible with KEA.
In MariaDB logs, we only see such entries :
> 2025-01-09 9:06:50 8232949 [Warning] Aborted connection 8232949 to db:
'kea' user: 'kea_user' host: '1.2.3.4' real ip: '5.6.7.8' (Got an error
reading communication packets)
As we don't see such errors by connecting directly a KEA node to a MariaDB
node, it seems that the issue only occurs when using HAPROXY in front of
the GALERA Cluster.
Also, according the following link, it seems that we should have to
configure 4 KEA instances (as we use 4 MariaDB instances which each KEA
instance being connected to one MariaDB node) ???
https://gitlab.isc.org/isc-projects/kea/-/issues/1746
For troubleshooting this issue, we currently configured HAPROXY with only 1
MariaDB node and source IP stickiness, the aims being :
* to persist the KEA database connection with the same MariaDB node
* to be able to connect KEA to another MariaDB node if the current node
fails
We can confirm that the error occurs in such a simple situation (with only
one database node) and we don't understand why KEA generates
DATABASE_MYSQL_FATAL_ERROR messages.
Could you confirm if we can use HAPROXY in such a manner ?
Are there specific settings to activate in KEA and/or HAPROXY ?
Or finaly, is it better not to use HAPROXY in such a scenario ?
Any advice for configuring this kind of KEA cluster will be appreciated.
Kind regards.
Le mer. 8 janv. 2025 à 22:59, Darren Ankney <darren.ankney at gmail.com> a
écrit :
> Hi Sébastien,
>
> The frequent disconnections could be anything. Is there anything in the
> MySQL logs? Also, is this MySQL or MariaDB? MySQL 8 has some performance
> issues that affect Kea pretty badly (See here:
> https://kea.readthedocs.io/en/kea-2.6.1/arm/admin.html#mysql-5-7-vs-mysql-8-vs-mariadb-10-and-11
> )
>
> If you haven't already found them, there are a couple of articles about
> MySQL clusters and Kea:
> https://kb.isc.org/docs/getting-started-with-galera-or-percona-for-kea
> https://kb.isc.org/docs/experimental-mysql-clusters-for-kea (this one
> might not be relevant)
>
> Thank you,
> Darren Ankney
>
> On Wed, Jan 8, 2025 at 4:51 AM Sébastien Nauer via Kea-users <
> kea-users at lists.isc.org> wrote:
>
>> Hello,
>>
>> We are experiencing issues with configuring a KEA cluster in
>> high-availability mode with a GALERA Cluster database backend.
>>
>> As we'd like to use the database cluster for managing leases
>> synchronization :
>>
>> - we configured KEA High Availability mode like this for disabling
>> lease sync & update :
>>
>> "high-availability": [
>> {
>> "this-server-name": "kea01",
>> "mode": "load-balancing",
>> "heartbeat-delay": 10000,
>> "max-response-delay": 10000,
>> "max-ack-delay": 5000,
>> "max-unacked-clients": 5,
>> "send-lease-updates": false,
>> "sync-leases": false,
>> "sync-page-limit": 10000,
>> "sync-timeout": 30000,
>> "multi-threading": {
>> "enable-multi-threading": true,
>> "http-dedicated-listener": true,
>> "http-listener-threads": 0,
>> "http-client-threads": 0
>> },
>> "peers": [
>> {
>> "name": "kea0&",
>> "url": "http://1.2.3.4:8080/",
>> "role": "primary",
>> "auto-failover": true
>> },
>> {
>> "name": "kea04",
>> "url": "http://5.6.7.8:8080/",
>> "role": "secondary",
>> "auto-failover": true
>> }
>> ]
>> }
>> ]
>>
>> - we configured the lease database like this for connecting to the
>> database cluster :
>>
>> "lease-database": {
>> "type": "mysql",
>> "name": "kea",
>> "user": "KEA_USER",
>> "password": "KEA_PASSWORD",
>> "host": "1.1.1.1",
>> "max-reconnect-tries" : 10,
>> "reconnect-wait-time" : 30000
>> },
>>
>> - we tried to connect to the database backend :
>> - through a VIP with load balancing
>> - through HAPROXY with / without load balancing
>> -
>>
>> When we start KEA service and the servers start, KEA logs show a lot of
>> DATABASE_MYSQL_FATAL_ERROR messages like this :
>>
>> 2025-01-07 16:18:42.483 ERROR [kea-dhcp4.database/4349.0x34cf8b308500]
>> DATABASE_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: unable to
>> execute for <SELECT h.host_id, h.dhcp_identifier, h.dhcp_identifier_type,
>> h.dhcp4_subnet_id, h.dhcp6_subnet_id, h.ipv4_address, h.hostname,
>> h.dhcp4_client_classes, h.dhcp6_client_classes, h.user_context,
>> h.dhcp4_next_server, h.dhcp4_server_hostname, h.dhcp4_boot_file_name,
>> h.auth_key, o.option_id, o.code, o.value, o.formatted_value, o.space,
>> o.persistent, o.cancelled, o.user_context FROM hosts AS h LEFT JOIN
>> dhcp4_options AS o ON h.host_id = o.host_id WHERE h.dhcp4_subnet_id = ? AND
>> h.dhcp_identifier_type = ? AND h.dhcp_identifier = ? ORDER BY h.host_id,
>> o.option_id>, reason: Server has gone away (error code: 2006).
>> 2025-01-07 16:18:43.533 INFO [kea-dhcp4.dhcp4/4349.0x34cf8aa12000]
>> DHCP4_DB_RECONNECT_LOST_CONNECTION database connection lost.
>> 2025-01-07 16:18:43.533 INFO [kea-dhcp4.hosts/4349.0x34cf8aa12000]
>> DHCPSRV_MYSQL_HOST_DB opening MySQL hosts database: host=10.194.50.12
>> max-reconnect-tries=10 name=prod_kea password=*****
>> reconnect-wait-time=30000 type=mysql universe=4 user=keaCluster
>> 2025-01-07 16:18:43.586 INFO [kea-dhcp4.dhcp4/4349.0x34cf8aa12000]
>> DHCP4_DB_RECONNECT_SUCCEEDED database connection recovered.
>>
>> As we noticed that the database connection is frequently lost for no
>> apparent reason we suspect that there is an issue with using persistent
>> database connections through a VIP with load balancing (maybe related to
>> multithreading also)
>>
>> Has anyone successfully configured such a setup ? How ?
>> What do you suggest for avoiding this error message and configuring a
>> high availability DHCP cluster with a high availability database cluster ?
>>
>> Any feedback from the community will be appreciated !
>> Kind regards
>>
>>
>>
>> --
>>
>> [image: Inline-Bild 1]
>>
>> *Sébastien NAUER <snauer at hotcity.lu>*| Technical Manager
>> 11, Avenue Guillaume | L-1651 Luxembourg <http://goo.gl/maps/GqQcf>
>> Phone: (+352) 2663 2660 | Fax: (+352) 2663 2665
>> Facebook <https://www.facebook.com/hotcity.lu> | Twitter
>> <https://twitter.com/hotcity_wifi>
>> --
>> ISC funds the development of this software with paid support
>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> information.
>>
>> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>>
>> Kea-users mailing list
>> Kea-users at lists.isc.org
>> https://lists.isc.org/mailman/listinfo/kea-users
>>
>
--
[image: Inline-Bild 1]
*Sébastien NAUER <snauer at hotcity.lu>*| Technical Manager
11, Avenue Guillaume | L-1651 Luxembourg <http://goo.gl/maps/GqQcf>
Phone: (+352) 2663 2660 | Fax: (+352) 2663 2665
Facebook <https://www.facebook.com/hotcity.lu> | Twitter
<https://twitter.com/hotcity_wifi>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20250109/5f32d617/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Logo_signature.png
Type: image/png
Size: 6124 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20250109/5f32d617/attachment-0001.png>
More information about the Kea-users
mailing list