<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoPlainText>As far as 1, i.e., connectivity between hosts, is concerned - they are on the same VLAN (L2) segment and can reach each other without any problems. Notice the very small RTT proving the point – they are hosted on the same cluster, so they are bridged locally within the host, hence sum 0.1ms RTT. No issues here AFAIK<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><span style='font-family:"Courier New";background:yellow;mso-highlight:yellow'>root@server-kea-node1:/etc/kea# ip a</span><span style='font-family:"Courier New"'><o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet 127.0.0.1/8 scope host lo<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet6 ::1/128 scope host <o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>2: enp6s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> link/ether bc:24:11:fe:9b:ff brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet 172.17.129.130/25 brd 172.17.129.255 scope global enp6s18<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet6 2600:6ce4:0:42::130/64 scope global <o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet6 fe80::be24:11ff:fefe:9bff/64 scope link <o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>root@server-kea-node1:/etc/kea# ping 172.17.129.133<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>PING 172.17.129.133 (172.17.129.133) 56(84) bytes of data.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.133: icmp_seq=1 ttl=64 time=0.093 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.133: icmp_seq=2 ttl=64 time=0.077 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.133: icmp_seq=3 ttl=64 time=0.080 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.133: icmp_seq=4 ttl=64 time=0.086 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.133: icmp_seq=5 ttl=64 time=0.084 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>^C<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>--- 172.17.129.133 ping statistics ---<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>5 packets transmitted, 5 received, 0% packet loss, time 4083ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>rtt min/avg/max/mdev = 0.077/0.084/0.093/0.005 ms<o:p></o:p></span></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><span style='font-family:"Courier New";background:yellow;mso-highlight:yellow'>root@server-kea-control:/etc/stork# ip a</span><span style='font-family:"Courier New"'><o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet 127.0.0.1/8 scope host lo<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet6 ::1/128 scope host <o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>2: enp6s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> link/ether bc:24:11:94:c3:52 brd ff:ff:ff:ff:ff:ff<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet 172.17.129.133/25 brd 172.17.129.255 scope global enp6s18<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet6 2600:6ce4:0:42::133/64 scope global <o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> inet6 fe80::be24:11ff:fe94:c352/64 scope link <o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'> valid_lft forever preferred_lft forever<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>root@server-kea-control:/etc/stork# ping 172.17.129.130<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>PING 172.17.129.130 (172.17.129.130) 56(84) bytes of data.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.130: icmp_seq=1 ttl=64 time=0.076 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.130: icmp_seq=2 ttl=64 time=0.086 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.130: icmp_seq=3 ttl=64 time=0.092 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>64 bytes from 172.17.129.130: icmp_seq=4 ttl=64 time=0.074 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>^C<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>--- 172.17.129.130 ping statistics ---<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>4 packets transmitted, 4 received, 0% packet loss, time 3052ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>rtt min/avg/max/mdev = 0.074/0.082/0.092/0.007 ms<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoPlainText>Neither machine has firewall running on them. <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>As far as 2 is concerned, I seem to be able to curl all content and wget does return 200 code as expected. <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><span style='font-family:"Courier New"'>root@server-kea-control:/tmp# wget <a href="http://172.17.129.130:9547/metrics">http://172.17.129.130:9547/metrics</a><o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>--2024-05-08 14:18:08-- <a href="http://172.17.129.130:9547/metrics">http://172.17.129.130:9547/metrics</a><o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>Connecting to 172.17.129.130:9547... connected.<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New";background:yellow;mso-highlight:yellow'>HTTP request sent, awaiting response... 200 OK</span><span style='font-family:"Courier New"'><o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>Length: unspecified [text/plain]<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>metrics: Read-only file system<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>Cannot write to ‘metrics’ (Read-only file system)<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'><a href="mailto:.root@server-kea-">.root@server-kea-</a>control:/tmp# curl <a href="http://172.17.129.130:9547/metrics">http://172.17.129.130:9547/metrics</a><o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_addresses_assigned_total Assigned addresses<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_addresses_assigned_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="1"} 2<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="100"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="2"} 17<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="3"} 2<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="300"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="4"} 2<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="400"} 117<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_assigned_total{subnet="401"} 3<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_addresses_declined_reclaimed_total Declined addresses that were reclaimed<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_addresses_declined_reclaimed_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="1"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="100"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="2"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="3"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="300"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="4"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_reclaimed_total{subnet="401"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_addresses_declined_total Declined counts<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_addresses_declined_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="1"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="100"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="2"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="3"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="300"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="4"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_declined_total{subnet="401"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_addresses_reclaimed_total Expired addresses that were reclaimed<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_addresses_reclaimed_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="1"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="100"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="2"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="3"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="300"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="4"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_reclaimed_total{subnet="401"} 24<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_addresses_total Size of subnet address pool<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_addresses_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="1"} 13<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="100"} 60<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="2"} 92<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="3"} 61<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="300"} 11<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="4"} 8<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_addresses_total{subnet="401"} 13<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_cumulative_addresses_assigned_total Cumulative number of assigned addresses since server startup<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_cumulative_addresses_assigned_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="1"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="100"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="2"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="3"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="300"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="4"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_cumulative_addresses_assigned_total{subnet="401"} 24<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_global4_addresses_declined_reclaimed_total Declined addresses that were reclaimed for all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_global4_addresses_declined_reclaimed_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_global4_addresses_declined_reclaimed_total 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_global4_addresses_declined_total Declined counts from all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_global4_addresses_declined_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_global4_addresses_declined_total 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_global4_addresses_reclaimed_total Expired addresses that were reclaimed for all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_global4_addresses_reclaimed_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_global4_addresses_reclaimed_total 24<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_global4_cumulative_addresses_assigned_total Cumulative number of assigned addresses since server startup from all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_global4_cumulative_addresses_assigned_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_global4_cumulative_addresses_assigned_total 24<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_packets_received_total Packets received<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_packets_received_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="ack"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="decline"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="discover"} 21850<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="drop"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="inform"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="nak"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="offer"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="parse-failed"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="release"} 24<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="request"} 266<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_received_total{operation="unknown"} 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp4_packets_sent_total Packets sent<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp4_packets_sent_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_sent_total{operation="ack"} 72<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_sent_total{operation="nak"} 194<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp4_packets_sent_total{operation="offer"} 304<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp6_global6_addresses_declined_reclaimed_total Declined addresses that were reclaimed for all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp6_global6_addresses_declined_reclaimed_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp6_global6_addresses_declined_reclaimed_total 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp6_global6_addresses_declined_total Declined counts from all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp6_global6_addresses_declined_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp6_global6_addresses_declined_total 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp6_global6_addresses_reclaimed_total Expired addresses that were reclaimed for all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp6_global6_addresses_reclaimed_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp6_global6_addresses_reclaimed_total 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp6_global6_cumulative_nas_assigned_total Cumulative number of assigned NA addresses since server startup from all subnets<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp6_global6_cumulative_nas_assigned_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp6_global6_cumulative_nas_assigned_total 0<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># HELP kea_dhcp6_global6_cumulative_pds_assigned_total Cumulative number of assigned PD prefixes since server startup<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'># TYPE kea_dhcp6_global6_cumulative_pds_assigned_total gauge<o:p></o:p></span></p><p class=MsoPlainText><span style='font-family:"Courier New"'>kea_dhcp6_global6_cumulative_pds_assigned_total 0<o:p></o:p></span></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Please note that the Stork server (133) was able to read this Stork agent (130) before the agent machine had kernel failure and had to be reinstalled. After I removed the previous registration of 130 agent from 133 server, I am unable to re-add the newly installed 130 agent with the same IP as before. <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Regards<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Marek<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>-----Original Message-----<br>From: Slawek Figiel <slawek@isc.org> <br>Sent: Wednesday, May 8, 2024 7:44 AM<br>To: Marek Hajduczenia <mxhajduczenia@gmail.com><br>Cc: stork-users@lists.isc.org<br>Subject: Re: [stork-users] Removing a node and re-adding it back causes a certificate error</p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Marek,<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>your logs show the connection from the Stork agent (172.17.129.130) to the Stork server (172.17.129.133) is established properly. The problem is the Stork server (172.17.129.133) cannot reach the Stork agent (172.17.129.130).<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Please, try to perform the below tests:<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>1. From the 172.17.129.133 host ping the 172.17.129.130 host. Does it work?<o:p></o:p></p><p class=MsoPlainText>2. From the 172.17.129.133 open/fetch the <a href="http://172.17.129.130:9547/metrics"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:9547/metrics</span></a> . Does it return HTTP 200 OK status and some metrics? If you specified the "--listen-stork-only" flag (or "STORK_AGENT_LISTEN_STORK_ONLY" environment variable), remove it temporarily.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Please verify the ports opened by your containers/VMs (default values, adjust them if you specified the custom ones in configuration):<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>- Stork server: 8080 (HTTP)<o:p></o:p></p><p class=MsoPlainText>- Stork agent: 8080 (GRPC), 9547 (HTTP)<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Regards,<o:p></o:p></p><p class=MsoPlainText>Slawek<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>On 07/05/2024 18:47, Marek Hajduczenia wrote:<o:p></o:p></p><p class=MsoPlainText>> I did go with the recommendation and even though I am 100% sure I have <o:p></o:p></p><p class=MsoPlainText>> IP reachability, the registration process with server token fails.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> root@server-kea-node1:/home/ace# ping 172.17.129.133 PING <o:p></o:p></p><p class=MsoPlainText>> 172.17.129.133 (172.17.129.133) 56(84) bytes of data.<o:p></o:p></p><p class=MsoPlainText>> 64 bytes from 172.17.129.133 <<a href="http://172.17.129.133"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133</span></a>>: icmp_seq=1 <o:p></o:p></p><p class=MsoPlainText>> ttl=64<o:p></o:p></p><p class=MsoPlainText>> time=0.074 ms<o:p></o:p></p><p class=MsoPlainText>> 64 bytes from 172.17.129.133 <<a href="http://172.17.129.133"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133</span></a>>: icmp_seq=2 <o:p></o:p></p><p class=MsoPlainText>> ttl=64<o:p></o:p></p><p class=MsoPlainText>> time=0.063 ms<o:p></o:p></p><p class=MsoPlainText>> 64 bytes from 172.17.129.133 <<a href="http://172.17.129.133"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133</span></a>>: icmp_seq=3 <o:p></o:p></p><p class=MsoPlainText>> ttl=64<o:p></o:p></p><p class=MsoPlainText>> time=0.147 ms<o:p></o:p></p><p class=MsoPlainText>> ^C<o:p></o:p></p><p class=MsoPlainText>> --- 172.17.129.133 ping statistics ---<o:p></o:p></p><p class=MsoPlainText>> 3 packets transmitted, 3 received, 0% packet loss, time 2054ms rtt <o:p></o:p></p><p class=MsoPlainText>> min/avg/max/mdev = 0.063/0.094/0.147/0.037 ms <o:p></o:p></p><p class=MsoPlainText>> root@server-kea-node1:/home/ace# sudo su stork-agent -s /bin/sh -c <o:p></o:p></p><p class=MsoPlainText>> 'stork-agent register --server-url <a href="http://172.17.129.133:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133:8080</span></a> <o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.133:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133:8080</span></a>>'<o:p></o:p></p><p class=MsoPlainText>> >>>> Server access token (optional):<o:p></o:p></p><p class=MsoPlainText>> >>>> IP address or FQDN of the host with Stork Agent (for the Stork <o:p></o:p></p><p class=MsoPlainText>> Server connection) [server-kea-node1]: 172.17.129.130 >>>> Port <o:p></o:p></p><p class=MsoPlainText>> number that Stork Agent will listen on [8080]:<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:84 Forced agent <o:p></o:p></p><p class=MsoPlainText>> certificates regeneration.<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:406 <o:p></o:p></p><p class=MsoPlainText>> ======================================================================<o:p></o:p></p><p class=MsoPlainText>> =======<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:407 AGENT TOKEN: <o:p></o:p></p><p class=MsoPlainText>> E9EE6D836E249B0E9A8898E638DECFCAD35A6577A70672E8F639D4A46CEBC211<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:408 <o:p></o:p></p><p class=MsoPlainText>> ======================================================================<o:p></o:p></p><p class=MsoPlainText>> =======<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:413 Machine will be <o:p></o:p></p><p class=MsoPlainText>> automatically registered using the server token<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:414 Agent token is <o:p></o:p></p><p class=MsoPlainText>> printed above for informational purposes only<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:415 User does not need <o:p></o:p></p><p class=MsoPlainText>> to copy or verify the agent token during registration via the server <o:p></o:p></p><p class=MsoPlainText>> token<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:416 It will be sent to <o:p></o:p></p><p class=MsoPlainText>> the server but it is not directly used in this type of machine <o:p></o:p></p><p class=MsoPlainText>> registration<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:425 Try to register <o:p></o:p></p><p class=MsoPlainText>> agent in Stork Server<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:262 Machine registered<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:44:26] register.go:283 Stored <o:p></o:p></p><p class=MsoPlainText>> agent-signed cert and CA cert<o:p></o:p></p><p class=MsoPlainText>> ERRO[2024-05-07 16:44:26] register.go:454 Retrying ping 1/3 <o:p></o:p></p><p class=MsoPlainText>> due to error error="problem pinging machine: Cannot <o:p></o:p></p><p class=MsoPlainText>> ping machine"<o:p></o:p></p><p class=MsoPlainText>> ERRO[2024-05-07 16:44:28] register.go:454 Retrying ping 2/3 <o:p></o:p></p><p class=MsoPlainText>> due to error error="problem pinging machine: Cannot <o:p></o:p></p><p class=MsoPlainText>> ping machine"<o:p></o:p></p><p class=MsoPlainText>> ERRO[2024-05-07 16:44:32] register.go:459 Cannot ping <o:p></o:p></p><p class=MsoPlainText>> machine<o:p></o:p></p><p class=MsoPlainText>> error="problem pinging machine: Cannot ping <o:p></o:p></p><p class=MsoPlainText>> machine"<o:p></o:p></p><p class=MsoPlainText>> FATA[2024-05-07 16:44:32] main.go:217 Registration <o:p></o:p></p><p class=MsoPlainText>> failed<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> I did try to add the --server-token flag but the net result is the <o:p></o:p></p><p class=MsoPlainText>> same<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> root@server-kea-node1:/home/ace# sudo su stork-agent -s /bin/sh -c <o:p></o:p></p><p class=MsoPlainText>> 'stork-agent register --server-url <a href="http://172.17.129.133:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133:8080</span></a> <o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.133:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.133:8080</span></a>> --server-token <o:p></o:p></p><p class=MsoPlainText>> OQYuMxkWmc3dySolt6uytLY4NrSkLWpo'<o:p></o:p></p><p class=MsoPlainText>> >>>> IP address or FQDN of the host with Stork Agent (for the Stork <o:p></o:p></p><p class=MsoPlainText>> Server connection) [server-kea-node1]: 172.17.129.130 >>>> Port <o:p></o:p></p><p class=MsoPlainText>> number that Stork Agent will listen on [8080]:<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:84 Forced agent <o:p></o:p></p><p class=MsoPlainText>> certificates regeneration.<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:406 <o:p></o:p></p><p class=MsoPlainText>> ======================================================================<o:p></o:p></p><p class=MsoPlainText>> =======<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:407 AGENT TOKEN: <o:p></o:p></p><p class=MsoPlainText>> D43AA9AA37F03B1D24A0ADC9CB23E4137FCC284429A1CC87AE397CC78E3DE4FC<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:408 <o:p></o:p></p><p class=MsoPlainText>> ======================================================================<o:p></o:p></p><p class=MsoPlainText>> =======<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:413 Machine will be <o:p></o:p></p><p class=MsoPlainText>> automatically registered using the server token<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:414 Agent token is <o:p></o:p></p><p class=MsoPlainText>> printed above for informational purposes only<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:415 User does not need <o:p></o:p></p><p class=MsoPlainText>> to copy or verify the agent token during registration via the server <o:p></o:p></p><p class=MsoPlainText>> token<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:416 It will be sent to <o:p></o:p></p><p class=MsoPlainText>> the server but it is not directly used in this type of machine <o:p></o:p></p><p class=MsoPlainText>> registration<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:425 Try to register <o:p></o:p></p><p class=MsoPlainText>> agent in Stork Server<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:262 Machine registered<o:p></o:p></p><p class=MsoPlainText>> INFO[2024-05-07 16:46:52] register.go:283 Stored <o:p></o:p></p><p class=MsoPlainText>> agent-signed cert and CA cert<o:p></o:p></p><p class=MsoPlainText>> ERRO[2024-05-07 16:46:52] register.go:454 Retrying ping 1/3 <o:p></o:p></p><p class=MsoPlainText>> due to error error="problem pinging machine: Cannot <o:p></o:p></p><p class=MsoPlainText>> ping machine"<o:p></o:p></p><p class=MsoPlainText>> ERRO[2024-05-07 16:46:54] register.go:454 Retrying ping 2/3 <o:p></o:p></p><p class=MsoPlainText>> due to error error="problem pinging machine: Cannot <o:p></o:p></p><p class=MsoPlainText>> ping machine"<o:p></o:p></p><p class=MsoPlainText>> ERRO[2024-05-07 16:46:58] register.go:459 Cannot ping <o:p></o:p></p><p class=MsoPlainText>> machine<o:p></o:p></p><p class=MsoPlainText>> error="problem pinging machine: Cannot ping <o:p></o:p></p><p class=MsoPlainText>> machine"<o:p></o:p></p><p class=MsoPlainText>> FATA[2024-05-07 16:46:58] main.go:217 Registration <o:p></o:p></p><p class=MsoPlainText>> failed<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> Regards<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> Marek<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> On Tue, May 7, 2024 at 10:38 AM Slawek Figiel <slawek@isc.org <o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:slawek@isc.org"><span style='color:windowtext;text-decoration:none'>mailto:slawek@isc.org</span></a>>> wrote:<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> Marek,<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> it is interesting case. But don't worry I'm sure we will find the cause<o:p></o:p></p><p class=MsoPlainText>> of the problem soon.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> I see you performed the manual registration using the "register"<o:p></o:p></p><p class=MsoPlainText>> command. Could you use this command again, but this time provide the<o:p></o:p></p><p class=MsoPlainText>> `--server-token` flag? Your server token is on the machines page.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> An additional check is performed when the `--server-token` flag is<o:p></o:p></p><p class=MsoPlainText>> used.<o:p></o:p></p><p class=MsoPlainText>> After the successful registration, the server sends the Ping request<o:p></o:p></p><p class=MsoPlainText>> over the GRPC protocol to the agent. It verifies whether the provided<o:p></o:p></p><p class=MsoPlainText>> agent host is accessible from the server machine.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> If the operation fails, you must check your network configuration and<o:p></o:p></p><p class=MsoPlainText>> the IP address provided as the agent host.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> I'm waiting for your feedback.<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> Regards,<o:p></o:p></p><p class=MsoPlainText>> Slawek<o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p><p class=MsoPlainText>> On 07/05/2024 18:25, Marek Hajduczenia wrote:<o:p></o:p></p><p class=MsoPlainText>> > Inline, please, with [mh0507] tags<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > -----Original Message-----<o:p></o:p></p><p class=MsoPlainText>> > From: Slawek Figiel <<a href="mailto:slawek@isc.org%20%3cmailto:slawek@isc.org"><span style='color:windowtext;text-decoration:none'>slawek@isc.org <mailto:slawek@isc.org</span></a>>><o:p></o:p></p><p class=MsoPlainText>> > Sent: Tuesday, May 7, 2024 10:21 AM<o:p></o:p></p><p class=MsoPlainText>> > To: Marek Hajduczenia <mxhajduczenia@gmail.com<o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:mxhajduczenia@gmail.com"><span style='color:windowtext;text-decoration:none'>mailto:mxhajduczenia@gmail.com</span></a>>><o:p></o:p></p><p class=MsoPlainText>> > Cc: <a href="mailto:stork-users@lists.isc.org"><span style='color:windowtext;text-decoration:none'>stork-users@lists.isc.org</span></a> <<a href="mailto:stork-users@lists.isc.org"><span style='color:windowtext;text-decoration:none'>mailto:stork-users@lists.isc.org</span></a>><o:p></o:p></p><p class=MsoPlainText>> > Subject: Re: [stork-users] Removing a node and re-adding it back<o:p></o:p></p><p class=MsoPlainText>> causes a certificate error<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > Marek,<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > > That has not solved my problem. I went through the<o:p></o:p></p><p class=MsoPlainText>> following process<o:p></o:p></p><p class=MsoPlainText>> > ><o:p></o:p></p><p class=MsoPlainText>> > > 1. Remove the previous registration for .130 machine at<o:p></o:p></p><p class=MsoPlainText>> Stork GUI<o:p></o:p></p><p class=MsoPlainText>> > > (Action > Remove)<o:p></o:p></p><p class=MsoPlainText>> > > 2. Remove all content from /var/lib/stork-agen/certs and<o:p></o:p></p><p class=MsoPlainText>> > > /var/lib/stork-agen/tokens<o:p></o:p></p><p class=MsoPlainText>> > > 3. Re-run registration<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > Did you re-authorize the machine? (Machines => Unathorized =><o:p></o:p></p><p class=MsoPlainText>> Click the Authorize button). I suppose yes but I would like to<o:p></o:p></p><p class=MsoPlainText>> double-check.<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > [mh0507] Correct, I did re-authorize the machine, that is part of<o:p></o:p></p><p class=MsoPlainText>> the standard work flow covered in the documentation for Stork.<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > > I am back where I was<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > Hmm... Could you verify if the Stork server and Stork agent<o:p></o:p></p><p class=MsoPlainText>> versions are the same? You can check them by `stork-server<o:p></o:p></p><p class=MsoPlainText>> --version` and `stork-agent --version` commands.<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > [mh0507] As requested, they are both on 1.16.0 as shown below.<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > root@server-kea-control:/etc/stork# stork-server --version<o:p></o:p></p><p class=MsoPlainText>> > 1.16.0<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > root@server-kea-node1:/var/lib/stork-agent# stork-agent --version<o:p></o:p></p><p class=MsoPlainText>> > 1.16.0<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > Slawek<o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> > On 07/05/2024 16:23, Marek Hajduczenia wrote:<o:p></o:p></p><p class=MsoPlainText>> >> The certs have been regenerated on the node, for what it is worth<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> root@server-kea-node1:/var/lib/stork-agent# ls -lah certs/ total 20K<o:p></o:p></p><p class=MsoPlainText>> >> drwx------ 2 stork-agent root 4.0K May 7 11:47 .<o:p></o:p></p><p class=MsoPlainText>> >> drwxr-xr-x 4 stork-agent root 4.0K May 6 19:08 ..<o:p></o:p></p><p class=MsoPlainText>> >> -rw------- 1 stork-agent stork-agent 664 May 7 11:47 ca.pem<o:p></o:p></p><p class=MsoPlainText>> >> -rw------- 1 stork-agent stork-agent 656 May 7 11:47 cert.pem<o:p></o:p></p><p class=MsoPlainText>> >> -rw------- 1 stork-agent stork-agent 241 May 7 11:47 key.pem<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> but it seems that the Stork Server side is holding onto old<o:p></o:p></p><p class=MsoPlainText>> certs? Not<o:p></o:p></p><p class=MsoPlainText>> >> sure where they would be stored - likely in the backend DB, but I do<o:p></o:p></p><p class=MsoPlainText>> >> not want to delete things at random.<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Regards<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Marek<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> On Tue, May 7, 2024 at 5:56 AM Marek Hajduczenia<o:p></o:p></p><p class=MsoPlainText>> >> <<a href="mailto:mxhajduczenia@gmail.com%20%3cmailto:mxhajduczenia@gmail.com"><span style='color:windowtext;text-decoration:none'>mxhajduczenia@gmail.com <mailto:mxhajduczenia@gmail.com</span></a>><o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:mxhajduczenia@gmail.com%20%3cmailto:mxhajduczenia@gmail.com"><span style='color:windowtext;text-decoration:none'>mailto:mxhajduczenia@gmail.com <mailto:mxhajduczenia@gmail.com</span></a>>>><o:p></o:p></p><p class=MsoPlainText>> wrote:<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Hi Slawek,<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> That has not solved my problem. I went through the following<o:p></o:p></p><p class=MsoPlainText>> >> process<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> 1. Remove the previous registration for .130 machine at<o:p></o:p></p><p class=MsoPlainText>> Stork GUI<o:p></o:p></p><p class=MsoPlainText>> >> (Action > Remove)<o:p></o:p></p><p class=MsoPlainText>> >> 2. Remove all content from /var/lib/stork-agen/certs and<o:p></o:p></p><p class=MsoPlainText>> >> /var/lib/stork-agen/tokens<o:p></o:p></p><p class=MsoPlainText>> >> 3. Re-run registration<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> root@server-kea-node1:/var/lib/stork-agent/tokens# sudo su<o:p></o:p></p><p class=MsoPlainText>> >> stork-agent -s /bin/sh -c 'stork-agent register --server-url<o:p></o:p></p><p class=MsoPlainText>> >> <a href="http://172.17.129.251:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.251:8080</span></a> <<a href="http://172.17.129.251:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.251:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.251:8080%20%3chttp:/172.17.129.251:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.251:8080 <http://172.17.129.251:8080</span></a>>>'<o:p></o:p></p><p class=MsoPlainText>> >> >>>> Server access token (optional):<o:p></o:p></p><p class=MsoPlainText>> >> >>>> IP address or FQDN of the host with Stork Agent (for<o:p></o:p></p><p class=MsoPlainText>> the Stork<o:p></o:p></p><p class=MsoPlainText>> >> Server connection) [server-kea-node1]: 172.17.129.130<o:p></o:p></p><p class=MsoPlainText>> >> >>>> Port number that Stork Agent will listen on [8080]:<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:81 There<o:p></o:p></p><p class=MsoPlainText>> are no<o:p></o:p></p><p class=MsoPlainText>> >> agent certificates - they will be generated.<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:406<o:p></o:p></p><p class=MsoPlainText>> >> <o:p></o:p></p><p class=MsoPlainText>> =============================================================================<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:407 AGENT<o:p></o:p></p><p class=MsoPlainText>> TOKEN:<o:p></o:p></p><p class=MsoPlainText>> >> <o:p></o:p></p><p class=MsoPlainText>> B777710F0547C3EA237002537E4B18202F888F4D0F6C2C00BA105167DE1688CE<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:408<o:p></o:p></p><p class=MsoPlainText>> >> <o:p></o:p></p><p class=MsoPlainText>> =============================================================================<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:411 <o:p></o:p></p><p class=MsoPlainText>> Authorize the<o:p></o:p></p><p class=MsoPlainText>> >> machine in the Stork web UI<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:425 Try to<o:p></o:p></p><p class=MsoPlainText>> register<o:p></o:p></p><p class=MsoPlainText>> >> agent in Stork Server<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:262 Machine<o:p></o:p></p><p class=MsoPlainText>> registered<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] register.go:283 Stored<o:p></o:p></p><p class=MsoPlainText>> >> agent-signed cert and CA cert<o:p></o:p></p><p class=MsoPlainText>> >> INFO[2024-05-07 11:47:14] main.go:215 <o:p></o:p></p><p class=MsoPlainText>> Registration<o:p></o:p></p><p class=MsoPlainText>> >> completed successfully<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> 4. I am back where I was<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> image.png<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> I did restart the local Stork agent but that did not change<o:p></o:p></p><p class=MsoPlainText>> >> anything<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> root@server-kea-node1:/var/lib/stork-agent/tokens# service<o:p></o:p></p><p class=MsoPlainText>> >> isc-kea-ctrl-agent restart<o:p></o:p></p><p class=MsoPlainText>> >> root@server-kea-node1:/var/lib/stork-agent/tokens# service<o:p></o:p></p><p class=MsoPlainText>> >> isc-kea-ctrl-agent status<o:p></o:p></p><p class=MsoPlainText>> >> ● isc-kea-ctrl-agent.service - Kea Control Agent<o:p></o:p></p><p class=MsoPlainText>> >> Loaded: loaded<o:p></o:p></p><p class=MsoPlainText>> >> (/lib/systemd/system/isc-kea-ctrl-agent.service; enabled;<o:p></o:p></p><p class=MsoPlainText>> vendor<o:p></o:p></p><p class=MsoPlainText>> >> preset: enabled)<o:p></o:p></p><p class=MsoPlainText>> >> Active: active (running) since Tue 2024-05-07<o:p></o:p></p><p class=MsoPlainText>> 11:50:16 UTC; 3s ago<o:p></o:p></p><p class=MsoPlainText>> >> Docs: man:kea-ctrl-agent(8)<o:p></o:p></p><p class=MsoPlainText>> >> Main PID: 10543 (kea-ctrl-agent)<o:p></o:p></p><p class=MsoPlainText>> >> Tasks: 5 (limit: 9343)<o:p></o:p></p><p class=MsoPlainText>> >> Memory: 1.4M<o:p></o:p></p><p class=MsoPlainText>> >> CPU: 7ms<o:p></o:p></p><p class=MsoPlainText>> >> CGroup: /system.slice/isc-kea-ctrl-agent.service<o:p></o:p></p><p class=MsoPlainText>> >> └─10543 /usr/sbin/kea-ctrl-agent -c<o:p></o:p></p><p class=MsoPlainText>> >> /etc/kea/kea-ctrl-agent.conf<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> May 07 11:50:16 server-kea-node1 systemd[1]:<o:p></o:p></p><p class=MsoPlainText>> >> isc-kea-ctrl-agent.service: Deactivated successfully.<o:p></o:p></p><p class=MsoPlainText>> >> May 07 11:50:16 server-kea-node1 systemd[1]: Stopped Kea<o:p></o:p></p><p class=MsoPlainText>> Control Agent.<o:p></o:p></p><p class=MsoPlainText>> >> May 07 11:50:16 server-kea-node1 systemd[1]:<o:p></o:p></p><p class=MsoPlainText>> >> isc-kea-ctrl-agent.service: Consumed 48.595s CPU time.<o:p></o:p></p><p class=MsoPlainText>> >> May 07 11:50:16 server-kea-node1 systemd[1]: Started Kea<o:p></o:p></p><p class=MsoPlainText>> Control Agent.<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> For what is worth, the message in the logs has changed<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> May 7 11:54:39 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-07 11:54:39" level="info" msg="Completed<o:p></o:p></p><p class=MsoPlainText>> pulling lease<o:p></o:p></p><p class=MsoPlainText>> >> stats from Kea apps: 0/1 succeeded" file=" <o:p></o:p></p><p class=MsoPlainText>> statspuller.go:71 "<o:p></o:p></p><p class=MsoPlainText>> >> May 7 11:54:39 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-07 11:54:39" level="warning" msg="rpc error:<o:p></o:p></p><p class=MsoPlainText>> code =<o:p></o:p></p><p class=MsoPlainText>> >> Unavailable desc = connection error: desc = \"error reading<o:p></o:p></p><p class=MsoPlainText>> server<o:p></o:p></p><p class=MsoPlainText>> >> preface: remote error: tls: bad certificate\"" file="<o:p></o:p></p><p class=MsoPlainText>> >> manager.go:124 " agent="172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> >> <<a href="http://172.17.129.130:8080%20%3chttp:/172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080 <http://172.17.129.130:8080</span></a>>>"<o:p></o:p></p><p class=MsoPlainText>> >> May 7 11:54:39 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-07 11:54:39" level="warning" msg="Failed to<o:p></o:p></p><p class=MsoPlainText>> get state<o:p></o:p></p><p class=MsoPlainText>> >> from the Stork agent; the agent is still not responding" file="<o:p></o:p></p><p class=MsoPlainText>> >> grpcli.go:326 " agent="172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> >> <<a href="http://172.17.129.130:8080%20%3chttp:/172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080 <http://172.17.129.130:8080</span></a>>>"<o:p></o:p></p><p class=MsoPlainText>> >> May 7 11:54:39 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-07 11:54:39" level="warning" msg="failed to<o:p></o:p></p><p class=MsoPlainText>> get state<o:p></o:p></p><p class=MsoPlainText>> >> from agent 172.17.129.130:8080 <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080%20%3chttp:/172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080 <http://172.17.129.130:8080</span></a>>>: grpc<o:p></o:p></p><p class=MsoPlainText>> >> manager is unable to re-establish connection with the agent<o:p></o:p></p><p class=MsoPlainText>> >> 172.17.129.130:8080 <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080%20%3chttp:/172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080 <http://172.17.129.130:8080</span></a>>>: rpc<o:p></o:p></p><p class=MsoPlainText>> error: code =<o:p></o:p></p><p class=MsoPlainText>> >> Unavailable desc = connection error: desc = \"error reading<o:p></o:p></p><p class=MsoPlainText>> server<o:p></o:p></p><p class=MsoPlainText>> >> preface: remote error: tls: bad certificate\"" file="<o:p></o:p></p><p class=MsoPlainText>> >> statepuller.go:247 "<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Not sure whether it for the better or worse<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Regards<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Marek<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> On Tue, May 7, 2024 at 4:06 AM Slawek Figiel<o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:slawek@isc.org%20%3cmailto:slawek@isc.org"><span style='color:windowtext;text-decoration:none'>slawek@isc.org <mailto:slawek@isc.org</span></a>><o:p></o:p></p><p class=MsoPlainText>> >> <<a href="mailto:slawek@isc.org%20%3cmailto:slawek@isc.org"><span style='color:windowtext;text-decoration:none'>mailto:slawek@isc.org <mailto:slawek@isc.org</span></a>>>> wrote:<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Hello Marek!<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Stork server reports that the agent introduced itself<o:p></o:p></p><p class=MsoPlainText>> with a "bad<o:p></o:p></p><p class=MsoPlainText>> >> certificate." Several reasons may cause it. I think you<o:p></o:p></p><p class=MsoPlainText>> should<o:p></o:p></p><p class=MsoPlainText>> >> remove<o:p></o:p></p><p class=MsoPlainText>> >> the existing cert files and re-register the agent.<o:p></o:p></p><p class=MsoPlainText>> Please do the<o:p></o:p></p><p class=MsoPlainText>> >> following steps:<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> 1. On the agent machine, remove the files in the<o:p></o:p></p><p class=MsoPlainText>> >> `/var/lib/stork-agent`<o:p></o:p></p><p class=MsoPlainText>> >> directory (you need to remove all files from the<o:p></o:p></p><p class=MsoPlainText>> `certs` and<o:p></o:p></p><p class=MsoPlainText>> >> `tokens`<o:p></o:p></p><p class=MsoPlainText>> >> subdirectories)<o:p></o:p></p><p class=MsoPlainText>> >> 2. If you manually registered the agent (by the `register`<o:p></o:p></p><p class=MsoPlainText>> >> command, you<o:p></o:p></p><p class=MsoPlainText>> >> need to call it again and restart the agent. If you<o:p></o:p></p><p class=MsoPlainText>> used the<o:p></o:p></p><p class=MsoPlainText>> >> self-registration flow, just restart the agent.<o:p></o:p></p><p class=MsoPlainText>> >> 3. Open the Stork UI, go to the machines list, switch<o:p></o:p></p><p class=MsoPlainText>> to the<o:p></o:p></p><p class=MsoPlainText>> >> "Unauthorized" tab, and re-authorize the agent.<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> I hope it'll solve your problem.<o:p></o:p></p><p class=MsoPlainText>> >> Don't hesitate to ask for more details if you have any<o:p></o:p></p><p class=MsoPlainText>> questions.<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> Regards,<o:p></o:p></p><p class=MsoPlainText>> >> Slawek Figiel<o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> >> On 07/05/2024 00:05, <a href="mailto:mxhajduczenia@gmail.com"><span style='color:windowtext;text-decoration:none'>mxhajduczenia@gmail.com</span></a><o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:mxhajduczenia@gmail.com"><span style='color:windowtext;text-decoration:none'>mailto:mxhajduczenia@gmail.com</span></a>><o:p></o:p></p><p class=MsoPlainText>> >> <mailto:mxhajduczenia@gmail.com<o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:mxhajduczenia@gmail.com"><span style='color:windowtext;text-decoration:none'>mailto:mxhajduczenia@gmail.com</span></a>>> wrote:<o:p></o:p></p><p class=MsoPlainText>> >> > Dear Forum,<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > I had two nodes added to Stork: .130 and .131 and<o:p></o:p></p><p class=MsoPlainText>> they were<o:p></o:p></p><p class=MsoPlainText>> >> working<o:p></o:p></p><p class=MsoPlainText>> >> > correctly. Node .130 had a kernel failure due to<o:p></o:p></p><p class=MsoPlainText>> changes I<o:p></o:p></p><p class=MsoPlainText>> >> was trying to<o:p></o:p></p><p class=MsoPlainText>> >> > apply and I did not make a copy, unfortunately. Long<o:p></o:p></p><p class=MsoPlainText>> story<o:p></o:p></p><p class=MsoPlainText>> >> short, I had<o:p></o:p></p><p class=MsoPlainText>> >> > to re-install node .130, and then I wanted to add it<o:p></o:p></p><p class=MsoPlainText>> back to<o:p></o:p></p><p class=MsoPlainText>> >> Stork<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > No matter what I do, I am getting the error shown above,<o:p></o:p></p><p class=MsoPlainText>> >> i.e., Cannot<o:p></o:p></p><p class=MsoPlainText>> >> > get state of machine.<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > Syslog review shows only one error message following two<o:p></o:p></p><p class=MsoPlainText>> >> warning messages.<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > May 6 21:58:38 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-06<o:p></o:p></p><p class=MsoPlainText>> >> > 21:58:38" level="warning" msg="rpc error: code =<o:p></o:p></p><p class=MsoPlainText>> Unavailable<o:p></o:p></p><p class=MsoPlainText>> >> desc =<o:p></o:p></p><p class=MsoPlainText>> >> > connection error: desc = \"error reading server preface:<o:p></o:p></p><p class=MsoPlainText>> >> remote error:<o:p></o:p></p><p class=MsoPlainText>> >> > tls: bad certificate\"" file=" <o:p></o:p></p><p class=MsoPlainText>> manager.go:124 "<o:p></o:p></p><p class=MsoPlainText>> >> > agent="172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>> <http://172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>>>"<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > May 6 21:58:38 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-06<o:p></o:p></p><p class=MsoPlainText>> >> > 21:58:38" level="warning" msg="Failed to get state<o:p></o:p></p><p class=MsoPlainText>> from the<o:p></o:p></p><p class=MsoPlainText>> >> Stork agent;<o:p></o:p></p><p class=MsoPlainText>> >> > the agent is still not responding" file="<o:p></o:p></p><p class=MsoPlainText>> >> grpcli.go:326 "<o:p></o:p></p><p class=MsoPlainText>> >> > agent="172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>> <http://172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>>>"<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > May 6 21:58:38 server-kea-control stork-server[719]:<o:p></o:p></p><p class=MsoPlainText>> >> time="2024-05-06<o:p></o:p></p><p class=MsoPlainText>> >> > 21:58:38" level="warning" msg="failed to get state<o:p></o:p></p><p class=MsoPlainText>> from agent<o:p></o:p></p><p class=MsoPlainText>> >> > 172.17.129.130:8080 <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080%20%3chttp:/172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080 <http://172.17.129.130:8080</span></a>>>: grpc<o:p></o:p></p><p class=MsoPlainText>> >> manager is unable to re-establish connection<o:p></o:p></p><p class=MsoPlainText>> >> > with the agent 172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>><o:p></o:p></p><p class=MsoPlainText>> >> <http://172.17.129.130:8080<o:p></o:p></p><p class=MsoPlainText>> <<a href="http://172.17.129.130:8080"><span style='color:windowtext;text-decoration:none'>http://172.17.129.130:8080</span></a>>>: rpc error: code = Unavailable desc =<o:p></o:p></p><p class=MsoPlainText>> >> > connection error: desc = \"error reading server preface:<o:p></o:p></p><p class=MsoPlainText>> >> remote error:<o:p></o:p></p><p class=MsoPlainText>> >> > tls: bad certificate\"" file=" <o:p></o:p></p><p class=MsoPlainText>> statepuller.go:247 "<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > I suspect that the TLS certificate does to get<o:p></o:p></p><p class=MsoPlainText>> cleared when<o:p></o:p></p><p class=MsoPlainText>> >> the machine<o:p></o:p></p><p class=MsoPlainText>> >> > is removed and a machine with the same IP address is<o:p></o:p></p><p class=MsoPlainText>> re-added.<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > I did not find a remedy for it for now and I do not<o:p></o:p></p><p class=MsoPlainText>> fancy a<o:p></o:p></p><p class=MsoPlainText>> >> complete<o:p></o:p></p><p class=MsoPlainText>> >> > re-install of Stork if I can avoid it.<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > Any suggestions on how to fix it?<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > Regards<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> > Marek<o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> ><o:p></o:p></p><p class=MsoPlainText>> >> --<o:p></o:p></p><p class=MsoPlainText>> >> Stork-users mailing list<o:p></o:p></p><p class=MsoPlainText>> >> <a href="mailto:Stork-users@lists.isc.org"><span style='color:windowtext;text-decoration:none'>Stork-users@lists.isc.org</span></a> <<a href="mailto:Stork-users@lists.isc.org"><span style='color:windowtext;text-decoration:none'>mailto:Stork-users@lists.isc.org</span></a>><o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:Stork-users@lists.isc.org%20%3cmailto:Stork-users@lists.isc.org"><span style='color:windowtext;text-decoration:none'>mailto:Stork-users@lists.isc.org <mailto:Stork-users@lists.isc.org</span></a>>><o:p></o:p></p><p class=MsoPlainText>> >> <a href="https://lists.isc.org/mailman/listinfo/stork-users"><span style='color:windowtext;text-decoration:none'>https://lists.isc.org/mailman/listinfo/stork-users</span></a><o:p></o:p></p><p class=MsoPlainText>> <<a href="https://lists.isc.org/mailman/listinfo/stork-users"><span style='color:windowtext;text-decoration:none'>https://lists.isc.org/mailman/listinfo/stork-users</span></a>><o:p></o:p></p><p class=MsoPlainText>> >> <https://lists.isc.org/mailman/listinfo/stork-users<o:p></o:p></p><p class=MsoPlainText>> <<a href="https://lists.isc.org/mailman/listinfo/stork-users"><span style='color:windowtext;text-decoration:none'>https://lists.isc.org/mailman/listinfo/stork-users</span></a>>><o:p></o:p></p><p class=MsoPlainText>> >><o:p></o:p></p><p class=MsoPlainText>> ><o:p></o:p></p><p class=MsoPlainText>> <o:p></o:p></p></div></body></html>