[stork-users] HA-status unavailable

[Slawek Figiel]

Hello Andrey!

 > I checked the logs but couldn't identify the problem. What connection
 > should I look for? Is it stork server, stork agent or control agent that
 > establishes this connection and performs the check?

It works this way:

1. The Stork server prepares a command to Kea API.
2. The Stork server sends this command to a specific Stork agent.
3. The Stork agent passes the command to Kea Control Agent API.
4. Kea Control Agent passes the command to a particular Kea DHCP daemon.
5. Kea DHCP daemon returns HA status to Kea Control Agent.
6. Kea Control Agent returns the HA status to Stork agent.
7. Stork agent returns the HA status to Stork server.
8. Stork server saves the HA status in the database.

The important thing is the above process only monitors the HA status. 
Stork doesn't have any influence on how the HA works. The HA peers 
exchange their statuses directly, so it is Kea DHCP daemon to Kea DHCP 
daemon connection.

In the attached data, the primary peer is unavailable and offline but 
the secondary peer is in the standby state. It means it haven't detected 
that its partner is down.

It may mean there is a connectivity issue between Stork server and Kea 
DHCP daemon, one of the services (i.e. Stork agent or Kea Control Agent) 
is down, or the Kea DHCP daemon returns an unexpected response.

But you wrote the everything is OK from the primary peer perspective. 
So, there may be a problem with updating the Kea data on the Stork side.

Please, make the below checks:

1. Check the machine list (/machines), and machine pages (/machines/XX) 
related to the HA hosts. Does UI notify any errors?
2. Check the unauthorized machine list. Are there any machines listed?
3. Check the application pages for your HA peers. Does UI notify any errors?
4. Check the Stork server and Stork agent logs. The HA status is fetched 
one per minute by default. Do you observe any errors here?
5. Which Kea version do you use? Does the second HA pair use the same 
Kea version?

Regard,
Slawek Figiel

On 23/02/2024 18:05, Andrey Kostin wrote:
> Hi Stork-users,
> 
> I have two pairs of Kea-dhcp servers in ha-mode. In one pair the standby 
> server shows HA-status of the primary as unavailable (for both dhcp4 and 
> dhcp6).
> Here is how it's seen in stork web interface:
> High Availability
> 
> Local server
> Status time:    2024-02-23 11:52:04
> Status checked:    1 minutes ago
> Role:    standby
> Control status:    online
> Heartbeat status:    ok
> State:    hot-standby
> Scopes served:    none (standby server)
> Last in partner-down:    never
> Unacked clients:    n/a
> Connecting clients:    n/a
> Analyzed packets:    n/a
> 
> Remote server Kea at X.X.X.251
> Status time:    2024-02-23 11:52:03
> Status checked:    1 minutes ago
> Role:    primary
> Control status:    offline             <<<< Here is the problem
> Heartbeat status:    ok
> State:    unavailable             <<<< Here is the problem
> Scopes served:    none
> Last in partner-down:    never
> Unacked clients:    n/a
> Connecting clients:    n/a
> Analyzed packets:    n/a
> 
>  From primary server's perspective everything is ok. In another pair of 
> dhcp servers both looks mutually ok.
> In direct API call to the standby server's Kea control agent the status 
> looks ok:
> API call is successful
> Result code: 0
> {
>      "high-availability": [
>          {
>              "ha-mode": "hot-standby",
>              "ha-servers": {
>                  "local": {
>                      "role": "standby",
>                      "scopes": [],
>                      "state": "hot-standby"
>                  },
>                  "remote": {
>                      "age": 0,
>                      "analyzed-packets": 0,
>                      "communication-interrupted": false,
>                      "connecting-clients": 0,
>                      "in-touch": true,
>                      "last-scopes": [
>                          "keadhcp01"
>                      ],
>                      "last-state": "hot-standby",
>                      "role": "primary",
>                      "unacked-clients": 0,
>                      "unacked-clients-left": 0
>                  }
>              }
>          }
>      ],
>      "multi-threading-enabled": true,
>      "packet-queue-size": 28,
>      "packet-queue-statistics": [
>          1.0,
>          1.0,
>          1.0
>      ],
>      "pid": 20269,
>      "reload": 12092253,
>      "thread-pool-size": 4,
>      "uptime": 12426307
> }
> 
> I checked the logs but couldn't identify the problem. What connection 
> should I look for? Is it stork server, stork agent or control agent that 
> establishes this connection and performs the check?
> 
> Kind regards,
> Andrey