<div dir="ltr"><div>The certs have been regenerated on the node, for what it is worth <br></div><div><br></div><div>root@server-kea-node1:/var/lib/stork-agent# ls -lah certs/<br>total 20K<br>drwx------ 2 stork-agent root 4.0K May 7 11:47 .<br>drwxr-xr-x 4 stork-agent root 4.0K May 6 19:08 ..<br>-rw------- 1 stork-agent stork-agent 664 May 7 11:47 ca.pem<br>-rw------- 1 stork-agent stork-agent 656 May 7 11:47 cert.pem<br>-rw------- 1 stork-agent stork-agent 241 May 7 11:47 key.pem</div><div><br></div><div>but it seems that the Stork Server side is holding onto old certs? Not sure where they would be stored - likely in the backend DB, but I do not want to delete things at random. <br></div><div><br></div><div>Regards</div><div><br></div><div>Marek<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 7, 2024 at 5:56 AM Marek Hajduczenia <<a href="mailto:mxhajduczenia@gmail.com">mxhajduczenia@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi Slawek, <br></div><div><br></div><div>That has not solved my problem. I went through the following process <br></div><div><br></div><div>1. Remove the previous registration for .130 machine at Stork GUI (Action > Remove)</div><div>2. Remove all content from
/var/lib/stork-agen/certs and
/var/lib/stork-agen/tokens</div><div>3. Re-run registration <br></div><div><br></div><div>root@server-kea-node1:/var/lib/stork-agent/tokens# sudo su stork-agent -s /bin/sh -c 'stork-agent register --server-url <a href="http://172.17.129.251:8080" target="_blank">http://172.17.129.251:8080</a>'<br>>>>> Server access token (optional): <br>>>>> IP address or FQDN of the host with Stork Agent (for the Stork Server connection) [server-kea-node1]: 172.17.129.130<br>>>>> Port number that Stork Agent will listen on [8080]: <br>INFO[2024-05-07 11:47:14] register.go:81 There are no agent certificates - they will be generated. <br>INFO[2024-05-07 11:47:14] register.go:406 ============================================================================= <br>INFO[2024-05-07 11:47:14] register.go:407 AGENT TOKEN: B777710F0547C3EA237002537E4B18202F888F4D0F6C2C00BA105167DE1688CE <br>INFO[2024-05-07 11:47:14] register.go:408 ============================================================================= <br>INFO[2024-05-07 11:47:14] register.go:411 Authorize the machine in the Stork web UI <br>INFO[2024-05-07 11:47:14] register.go:425 Try to register agent in Stork Server <br>INFO[2024-05-07 11:47:14] register.go:262 Machine registered <br>INFO[2024-05-07 11:47:14] register.go:283 Stored agent-signed cert and CA cert <br>INFO[2024-05-07 11:47:14] main.go:215 Registration completed successfully <br></div><div><br></div><div>4. I am back where I was <br></div><div><br></div><div><img src="cid:ii_lvwbw3ni0" alt="image.png" width="561" height="228"><br></div><div><br></div><div>I did restart the local Stork agent but that did not change anything <br></div><div><br></div><div>root@server-kea-node1:/var/lib/stork-agent/tokens# service isc-kea-ctrl-agent restart<br>root@server-kea-node1:/var/lib/stork-agent/tokens# service isc-kea-ctrl-agent status<br>● isc-kea-ctrl-agent.service - Kea Control Agent<br> Loaded: loaded (/lib/systemd/system/isc-kea-ctrl-agent.service; enabled; vendor preset: enabled)<br> Active: active (running) since Tue 2024-05-07 11:50:16 UTC; 3s ago<br> Docs: man:kea-ctrl-agent(8)<br> Main PID: 10543 (kea-ctrl-agent)<br> Tasks: 5 (limit: 9343)<br> Memory: 1.4M<br> CPU: 7ms<br> CGroup: /system.slice/isc-kea-ctrl-agent.service<br> └─10543 /usr/sbin/kea-ctrl-agent -c /etc/kea/kea-ctrl-agent.conf<br><br>May 07 11:50:16 server-kea-node1 systemd[1]: isc-kea-ctrl-agent.service: Deactivated successfully.<br>May 07 11:50:16 server-kea-node1 systemd[1]: Stopped Kea Control Agent.<br>May 07 11:50:16 server-kea-node1 systemd[1]: isc-kea-ctrl-agent.service: Consumed 48.595s CPU time.<br>May 07 11:50:16 server-kea-node1 systemd[1]: Started Kea Control Agent.</div><div><br></div><div>For what is worth, the message in the logs has changed<br></div><div><br></div>May 7 11:54:39 server-kea-control stork-server[719]: time="2024-05-07 11:54:39" level="info" msg="Completed pulling lease stats from Kea apps: 0/1 succeeded" file=" statspuller.go:71 "<br>May 7 11:54:39 server-kea-control stork-server[719]: time="2024-05-07 11:54:39" level="warning" msg="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"" file=" manager.go:124 " agent="<a href="http://172.17.129.130:8080" target="_blank">172.17.129.130:8080</a>"<br>May 7 11:54:39 server-kea-control stork-server[719]: time="2024-05-07 11:54:39" level="warning" msg="Failed to get state from the Stork agent; the agent is still not responding" file=" grpcli.go:326 " agent="<a href="http://172.17.129.130:8080" target="_blank">172.17.129.130:8080</a>"<br>May 7 11:54:39 server-kea-control stork-server[719]: time="2024-05-07 11:54:39" level="warning" msg="failed to get state from agent <a href="http://172.17.129.130:8080" target="_blank">172.17.129.130:8080</a>: grpc manager is unable to re-establish connection with the agent <a href="http://172.17.129.130:8080" target="_blank">172.17.129.130:8080</a>: rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"" file=" statepuller.go:247 "<br></div><div dir="ltr"><br></div><div>Not sure whether it for the better or worse</div><div><br></div><div>Regards</div><div><br></div><div>Marek<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 7, 2024 at 4:06 AM Slawek Figiel <<a href="mailto:slawek@isc.org" target="_blank">slawek@isc.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Marek!<br>
<br>
Stork server reports that the agent introduced itself with a "bad <br>
certificate." Several reasons may cause it. I think you should remove <br>
the existing cert files and re-register the agent. Please do the <br>
following steps:<br>
<br>
1. On the agent machine, remove the files in the `/var/lib/stork-agent` <br>
directory (you need to remove all files from the `certs` and `tokens` <br>
subdirectories)<br>
2. If you manually registered the agent (by the `register` command, you <br>
need to call it again and restart the agent. If you used the <br>
self-registration flow, just restart the agent.<br>
3. Open the Stork UI, go to the machines list, switch to the <br>
"Unauthorized" tab, and re-authorize the agent.<br>
<br>
I hope it'll solve your problem.<br>
Don't hesitate to ask for more details if you have any questions.<br>
<br>
Regards,<br>
Slawek Figiel<br>
<br>
On 07/05/2024 00:05, <a href="mailto:mxhajduczenia@gmail.com" target="_blank">mxhajduczenia@gmail.com</a> wrote:<br>
> Dear Forum,<br>
> <br>
> I had two nodes added to Stork: .130 and .131 and they were working <br>
> correctly. Node .130 had a kernel failure due to changes I was trying to <br>
> apply and I did not make a copy, unfortunately. Long story short, I had <br>
> to re-install node .130, and then I wanted to add it back to Stork<br>
> <br>
> No matter what I do, I am getting the error shown above, i.e., Cannot <br>
> get state of machine.<br>
> <br>
> Syslog review shows only one error message following two warning messages.<br>
> <br>
> May 6 21:58:38 server-kea-control stork-server[719]: time="2024-05-06 <br>
> 21:58:38" level="warning" msg="rpc error: code = Unavailable desc = <br>
> connection error: desc = \"error reading server preface: remote error: <br>
> tls: bad certificate\"" file=" manager.go:124 " <br>
> agent="<a href="http://172.17.129.130:8080" rel="noreferrer" target="_blank">172.17.129.130:8080</a>"<br>
> <br>
> May 6 21:58:38 server-kea-control stork-server[719]: time="2024-05-06 <br>
> 21:58:38" level="warning" msg="Failed to get state from the Stork agent; <br>
> the agent is still not responding" file=" grpcli.go:326 " <br>
> agent="<a href="http://172.17.129.130:8080" rel="noreferrer" target="_blank">172.17.129.130:8080</a>"<br>
> <br>
> May 6 21:58:38 server-kea-control stork-server[719]: time="2024-05-06 <br>
> 21:58:38" level="warning" msg="failed to get state from agent <br>
> <a href="http://172.17.129.130:8080" rel="noreferrer" target="_blank">172.17.129.130:8080</a>: grpc manager is unable to re-establish connection <br>
> with the agent <a href="http://172.17.129.130:8080" rel="noreferrer" target="_blank">172.17.129.130:8080</a>: rpc error: code = Unavailable desc = <br>
> connection error: desc = \"error reading server preface: remote error: <br>
> tls: bad certificate\"" file=" statepuller.go:247 "<br>
> <br>
> I suspect that the TLS certificate does to get cleared when the machine <br>
> is removed and a machine with the same IP address is re-added.<br>
> <br>
> I did not find a remedy for it for now and I do not fancy a complete <br>
> re-install of Stork if I can avoid it.<br>
> <br>
> Any suggestions on how to fix it?<br>
> <br>
> Regards<br>
> <br>
> Marek<br>
> <br>
> <br>
-- <br>
Stork-users mailing list<br>
<a href="mailto:Stork-users@lists.isc.org" target="_blank">Stork-users@lists.isc.org</a><br>
<a href="https://lists.isc.org/mailman/listinfo/stork-users" rel="noreferrer" target="_blank">https://lists.isc.org/mailman/listinfo/stork-users</a><br>
</blockquote></div></div>
</blockquote></div>