[stork-users] Issues with stork-agent

Thu Jun 13 15:53:44 UTC 2024

David,

I'm glad to hear that your main problem with the statistics has been 
resolved.

The Stork agent produces messages about a problem connecting to some 
daemons and skipping their logs fetch because your Kea CA configuration 
specifies some daemons that are not running.
The Stork agent scans the Kea CA config first and then looks for the Kea 
daemons. In your case, the config specifies control sockets for the 
DHCPv4, DHCPv6, and D2 daemons. The last two are probably not running in 
your environment.

To fix the problem, remove the unnecessary entries from the 
`control-sockets` node in the Kea CA config.

We discussed your problem in our team and agreed that similar problems 
shouldn't happen even if not all configured daemons are running. Kea is 
shipped with this configuration, and Stork should handle it gracefully. 
Therefore, we opened 
https://gitlab.isc.org/isc-projects/stork/-/issues/1422 to improve the 
situation.

Regards,
Slawek

On 07/06/2024 12:19, David Rickard wrote:
> Slawek
> 
> Thank you for your reply.
> 
>> Kea returns status 0 on success. Any other status is an error.
> 
> Of course! I can't believe I missed that.
> 
>> I'm a bit confused. Could you confirm that I understand you correctly?
>> You have the Kea DHCPv4 daemon, Kea Control Agent, Stork agent and Stork
>> server? You can see all the DHCPv4 data in the Stork UI.
> 
> Correct. I see everything except consumption of the subnets
> 
>> The problem is that you get an error message about the unsuccessful
>> result of the config-get command, but otherwise, everything seems to
>> work properly, right?
> 
> Yes correct.
> 
>> Some questions to you:
>>
>> 1. Do you have DHCPv6 daemon running?
> No
>> 2. What control sockets do have specified in the
>> /etc/kea/kea-ctrl-agent.conf file?
> 
> The defaults, I don't think I changed these:
> 
> "control-sockets": {
>          "dhcp4": {
>              "socket-type": "unix",
>              "socket-name": "/tmp/kea4-ctrl-socket"
>          },
>          "dhcp6": {
>              "socket-type": "unix",
>              "socket-name": "/tmp/kea6-ctrl-socket"
>          },
>          "d2": {
>              "socket-type": "unix",
>              "socket-name": "/tmp/kea-ddns-ctrl-socket"
>          }
> 
>> 3. Can you browse the Kea logs in Stork UI? Go to the application page
>> and click on the log path - it works only if you log to the file, not to
>> the standard out.
> 
> I didn't try this, as I did the next thing first...
> 
>> You need to load the stat_cmds hook library in Kea to see the
>> utilizations in Stork.
> 
> I loaded the library, and now I see utilisation in stork!
> 
>> Does the Stork agent user (the user that runs the Stork agent) have the
>> right to read the log files?
> 
> It didn't. I added stork-agent to the kea group, and confirmed it can read the logs by using su and running as the user, and I could tail the logs. I can also see the logs in Stork. I didn't look for them last time round so I'm not sure if that changed. But the two little changes for the hook, and the permissions do seem to have fixed things. Stork now shows usage and graphs. Thank you for your help!
> 
> However, every 30 seconds the syslog contains two entries as follows still:
> 
> level="warning" msg="Skipped refreshing viewable log files because config-get returned unsuccessful result" file="              kea.go:68
> 
> So whilst it is functionally working as far as I can see, it's still spamming the logs. Perhaps something to do with the ctrl agent maybe?
> 
> Regards
> 
> David Rickard
> Head of Technology
> DTS - Infrastructure Team
> Buckinghamshire New University
> From: Stork-users <stork-users-bounces at lists.isc.org> On Behalf Of stork-users-request at lists.isc.org
> Sent: Wednesday, June 5, 2024 1:00 PM
> To: stork-users at lists.isc.org
> Subject: Stork-users Digest, Vol 49, Issue 1
> 
> CAUTION: This email originated from outside of the organisation. Do not click links, open attachments or respond unless you recognise the sender and know that the content is safe.
> 
> Send Stork-users mailing list submissions to
> mailto:stork-users at lists.isc.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.isc.org/mailman/listinfo/stork-users
> or, via email, send a message with subject or body 'help' to
> mailto:stork-users-request at lists.isc.org
> 
> You can reach the person managing the list at
> mailto:stork-users-owner at lists.isc.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Stork-users digest..."
> 
> 
> Today's Topics:
> 
> 1. Re: Issues with stork-agent (Slawek Figiel)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 4 Jun 2024 14:21:45 +0200
> From: Slawek Figiel <mailto:slawek at isc.org>
> To: mailto:stork-users at lists.isc.org
> Subject: Re: [stork-users] Issues with stork-agent
> Message-ID: <mailto:f5ae59d7-7a0a-4f85-9fa2-4454bdcf9ed6 at isc.org>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> 
> Hello David!
> 
> Thank you for your report. Let's try to recognize and solve your problem.
> 
>> This suggests (to me) that if the result is greater than zero (which
> it is) that it throws an error. Is this right? This is certainly what
> seems to be happening. The kea-ctrl logs show it happily responding to
> the config-get responses.
> 
> Kea returns status 0 on success. Any other status is an error.
> 
>> Stork is reporting the full configuration, and all the subnets of the
> attached agent.
> 
> I'm a bit confused. Could you confirm that I understand you correctly?
> You have the Kea DHCPv4 daemon, Kea Control Agent, Stork agent and Stork
> server? You can see all the DHCPv4 data in the Stork UI.
> 
> The problem is that you get an error message about the unsuccessful
> result of the config-get command, but otherwise, everything seems to
> work properly, right?
> 
> Some questions to you:
> 
> 1. Do you have DHCPv6 daemon running?
> 2. What control sockets do have specified in the
> /etc/kea/kea-ctrl-agent.conf file?
> 3. Can you browse the Kea logs in Stork UI? Go to the application page
> and click on the log path - it works only if you log to the file, not to
> the standard out.
> 
>> But it shows no utilisation in the subnet I am testing with - I only
> have a single client using the Kea servers right now.
> 
> You need to load the stat_cmds hook library in Kea to see the
> utilizations in Stork.
> 
>> When I checked the logs on the server the agent is trying repeatedly
> (many times per minute) to get-config against Kea, with no success.
> 
> Does the Stork agent user (the user that runs the Stork agent) have the
> right to read the log files?
> 
> Best regards,
> Slawek Figiel
> 
> 
> On 31/05/2024 17:33, David Rickard wrote:
>> Hello,
>>
>> I was initially going to open an issue on gitlab, but it said have you
>> asked here, so I thought I would. As I already started filling in the
>> issue template, I?ve included that here below.
>>
>> **Describe the bug** stork-agent is configured and operational on a
>> running kea-dhcp server. The agent is repeatedly polling the kea-ctrl
>> service with config-get, but fails with an error: level="warning"
>> msg="Skipped refreshing viewable log files because config-get returned
>> unsuccessful result". If I manually issue a config-get to kea-ctrl (via
>> the socket) I received a full config, with loggers. Upon further
>> inspection of the source, I see these lines in agent/kea.go:
>>
>> `if response.Result > 0 { log.Warn("Skipped refreshing viewable log
>> files because config-get returned unsuccessful result") return nil }`
>>
>> This suggests (to me) that if the result is greater than zero (which it
>> is) that it throws an error. Is this right? This is certainly what seems
>> to be happening. The kea-ctrl logs show it happily responding to the
>> config-get responses.
>>
>> **To Reproduce** Steps to reproduce the behavior:
>>
>> * Server is RHEL 8. Installed kea 2.4.1, stork-agent is version 1.16.0
>> from Cloudsmith repos, and registered to a remote stork server.
>>
>> * Built second server as HA hot failover.
>>
>> * kea is configured thus (subnets have been omitted for clarity):
>>
>> {
>>
>> "Dhcp4": {
>>
>> ??? "interfaces-config": {
>>
>> ??????? "interfaces": [ "*" ],
>>
>> ??????? "dhcp-socket-type": "udp"
>>
>> ??? },
>>
>> ??? "control-socket": {
>>
>> ??????? "socket-type": "unix",
>>
>> ??????? "socket-name": "/tmp/kea4-ctrl-socket"
>>
>> ??? },
>>
>> ??? "multi-threading": {
>>
>> ?????? "enable-multi-threading": true,
>>
>> ?????? "thread-pool-size": 12,
>>
>> ?????? "packet-queue-size": 792
>>
>> ??? },
>>
>> ??? "hooks-libraries": [{
>>
>> ??????? "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
>>
>> ??????? "parameters": { }
>>
>> ??? }, {
>>
>> ??????? "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
>>
>> ??????? "parameters": {
>>
>> ??????????? "high-availability": [{
>>
>> ??????????????? "this-server-name": "NODE1",
>>
>> ??????????????? "mode": "hot-standby",
>>
>> ??????????????? "heartbeat-delay": 10000,
>>
>> ??????????????? "max-response-delay": 60000,
>>
>> ??????????????? "max-ack-delay": 5000,
>>
>> ????????? ??????"max-unacked-clients": 1,
>>
>> ??????????????????????????????? "send-lease-updates": false,
>>
>> ??????????????????????????????? "sync-leases": false,
>>
>> ??????????????? "peers": [{
>>
>> ??????????????????? "name": "NODE1",
>>
>> ??????????????????? "url": "http://NODE1_IP:8001/",
>>
>> ??????????????????? "role": "primary"
>>
>> ????????????? ??}, {
>>
>> ??????????????????? "name": "NODE2",
>>
>> ??????????????????? "url": "http://NODE2_IP:8001/",
>>
>> ??????????????????? "role": "standby"
>>
>> ??????????????? }]
>>
>> ??????????? }]
>>
>> ??????? }
>>
>> ??? }],
>>
>> ??? "lease-database": {
>>
>> ??????? "type": "mysql",
>>
>> ??????? "name": "kea_dhcp",
>>
>> ??????? "user": "kea_dhcp",
>>
>> ??????? "password": "somepassword",
>>
>> ??????? "host": "localhost",
>>
>> ??????? "port": 3306,
>>
>> ??????? "lfc-interval": 3600
>>
>> ??? },
>>
>> ??? "hosts-database": {
>>
>> ??????? "type": "mysql",
>>
>> ? ??????"name": "kea_dhcp",
>>
>> ??????? "user": "kea_dhcp",
>>
>> ??????? "password": "somepassword",
>>
>> ??????? "host": "localhost",
>>
>> ??????? "port": 3306
>>
>> ??? },
>>
>> ??? "expired-leases-processing": {
>>
>> ??????? "reclaim-timer-wait-time": 10,
>>
>> ??????? "flush-reclaimed-timer-wait-time": 25,
>>
>> ??????? "hold-reclaimed-time": 3600,
>>
>> ??????? "max-reclaim-leases": 100,
>>
>> ??????? "max-reclaim-time": 250,
>>
>> ??????? "unwarned-reclaim-cycles": 5
>>
>> ??? },
>>
>> ??? "renew-timer": 900,
>>
>> ??? "rebind-timer": 1800,
>>
>> ??? "valid-lifetime": 3600,
>>
>> ??? "option-data": [
>>
>> ??????? {
>>
>> ??????????? "name": "domain-name-servers",
>>
>> ??????????? "data": "NODE1_IP, NODE2_IP"
>>
>> ??????? },
>>
>> ??????? {
>>
>> ??????????? "code": 15,
>>
>> ??????????? "data": "mydomain"
>>
>> ??????? }
>>
>> ??? ],
>>
>> // Subnet4
>>
>> ??? "loggers": [
>>
>> ????? {
>>
>> ??????? "name": "kea-dhcp4",
>>
>> ??????? "severity": "INFO",
>>
>> ??????? "output_options": [
>>
>> ????????? {
>>
>> ??????????? "output": "/var/log/kea/dhcp4.log"
>>
>> ????????? }
>>
>> ??????? ]
>>
>> ????? },
>>
>> ????? {
>>
>> ??????? "name": "kea-dhcp4.dhcpsrv",
>>
>> ??????? "severity": "INFO",
>>
>> ??????? "output_options": [
>>
>> ????????? {
>>
>> ??????????? "output": "/var/log/kea/dhcp4-dhcpsrv.log"
>>
>> ????????? }
>>
>> ??????? ]
>>
>> ????? },
>>
>> ????? {
>>
>> ??????? "name": "kea-dhcp4.leases",
>>
>> ??????? "severity": "INFO",
>>
>> ??????? "output_options": [
>>
>> ? ????????{
>>
>> ??????????? "output": "/var/log/kea/dhcp4-leases.log"
>>
>> ????????? }
>>
>> ??????? ]
>>
>> ????? }
>>
>> ??? ]
>>
>> }
>>
>> }
>>
>> * stork-agent is configured thus:
>>
>> ### the IP or hostname to listen on for incoming Stork server connections
>>
>> STORK_AGENT_HOST=[STORK SERVER IP]
>>
>> ### the TCP port to listen on for incoming Stork server connections
>>
>> STORK_AGENT_PORT=8082
>>
>> ### listen for commands from the Stork server only, but not for
>> Prometheus requests
>>
>> STORK_AGENT_LISTEN_STORK_ONLY=true
>>
>> **Expected behavior** Stork-agent should be able to read the config, and
>> then read the log files to get live information on leases, etc.
>>
>> **Environment:**
>>
>> - Kea version: 2.4.1, from Cloudsmith repo.
>>
>> - Stork: 1.16.0, from Cloudsmith repo.
>>
>> - OS: RHEL 8.9
>>
>> - Kea: RPM release, so default.
>>
>> - Kea: lease_cmds, ha
>>
>> **Additional Information** Stork is reporting the full configuration,
>> and all the subnets of the attached agent. But it shows no utilisation
>> in the subnet I am testing with - I only have a single client using the
>> Kea servers right now. Searching the DHCP Leases in Stork shows the
>> client, but that's the only place. When I checked the logs on the server
>> the agent is trying repeatedly (many times per minute) to get-config
>> against Kea, with no success. But it must be having some success as
>> Stork does report so much information.
>>
>> I have looked over the configs repeatedly and unless I?ve missed
>> something obvious, I?m at a bit of a loss.
>>
>> I?m also hopeful resolving this will stop the log spam!
>>
>> Thanks in advance.
>>
>> *David Rickard*
>>
>> Head of Technology
>>
>> DTS Directorate
>>
>> Buckinghamshire New University
>>
>>
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> Stork-users mailing list
> mailto:Stork-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/stork-users
> 
> 
> ------------------------------
> 
> End of Stork-users Digest, Vol 49, Issue 1
> ******************************************
> 