Agent gets unhealthy and shows error on Last Check-in on triggering multiple single or bulk privilege change API calls

[harshitgupta-qasource]

Kibana version:

VERSION: 9.3.0 SNAPSHOT
BUILD: 94838
COMMIT: 1fd010eed96bf0195cbf4490405fcc067c64746e

Preconditions:

1. A 9.3.0-SNAPSHOT Kibana cloud environment is available.
2. Multiple agents are installed with privileged (root) access.

Steps to reproduce:

1. Navigate to Developer tools.
2. Execute the following API: POST kbn:/api/fleet/agents/{agent_id}/privilege_level_change.
3. Trigger the above API multiple times for the same agent
4. Wait for 5 mins
5. Navigate to Fleet > Agents Tab
6. Observe that Agent gets unhealthy and shows error on Last Check-in

Expected behavior:

• API should prevent or reject multiple simultaneous or repeated privilege_level_change API calls for the same agent.

Screen shot:*

Linked Feature ticket:
#4973

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary review for this ticket is Done.

[cmacknz]

@michalpristas seems like the privilege change action on the agent needs to be idempotent

[michalpristas]

We could do that, but the question is how we would determine that the action is the same.

Once we are no longer running as root, we cannot reliably determine the intended user. At that point, the only correct options are to fail, or to continue and risk falsely succeeding when an action is triggered or received with credentials that differ from the current unelevated user.

I see failing as cleaner and safer approach.

In the normal flow, once root privileges are dropped, no actions should be delivered to the agent anyway, since the API and UI will catch this and stop propagation.

[cmacknz]

I see failing as cleaner and safer approach.

If we are going to fail, the agent cannot permanently be marked as degraded/failed after an action potentially sent in error or that is duplicated due to a bug. There has to be a way to clear the error. I think marking the action as failed is correct and marking the agent as failed is incorrect in this case.

since the API and UI will catch this and stop propagation.

In general the agent is the authority on it's current state, and in a bug free implementation the UI's knowledge of the agent is always delayed (by a checkin period, by an acknowledgement that hasn't arrived yet), so I don't think we can rely on the UI as a safe guard for basically anything.