zedagent: refactor LPS handling into LocalCmdAgent component #5191

milan-zededa · 2025-08-21T10:13:02Z

Description

Refactored the LPS-handling code inside zedagent into a separate component, LocalCmdAgent. This helps mitigate the complexity and unorganized structure of the already large zedagent package.
No functional changes were introduced; the code was reorganized, cleaned up, and better documented.
Improved thread-safety for variables that previously lacked proper protection, reducing potential race conditions.
Added user-facing documentation for the Local Profile Server (LPS).
Added initial developer-facing documentation for zedagent, with focus on LPS handling.

How to test and validate this PR

No functional changes were made, but the entire LPS functionality should be retested.
Deploy and configure LPS, then test all the endpoints: https://github.com/lf-edge/eve-api/blob/main/PROFILE.md

try to change local profile
try to enable/disable radio silence
check app info published to LPS
try to restart and purge app via LPS
check device info published to LPS
try to shutdown all apps, shutdown the entire device and reboot the device
if device has cellular modem with a GNSS receiver (with a good signal reception), check location published to LPS

The same commands should be also tested with the Local Operator Console (LOC) in air-gaped environment.

Note that I performed all these tests and additionally we have a small test suite in eden for LPS: https://github.com/lf-edge/eden/blob/master/tests/workflow/lps-loc.tests.txt

Changelog notes

Refactored Local Profile Server (LPS) handling in the zedagent microservice into a dedicated LocalCmdAgent component.
Added user-facing documentation for Local Profile Server (LPS).

PR Backports

This is just refactoring and does not need to be backported.

14.5-stable: No
13.4-stable: No

Checklist

I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't check them.

pkg/pillar/localcommand/devinfo.go

pkg/pillar/cmd/zedagent/parseconfig.go

milan-zededa · 2025-08-21T13:23:54Z

@christoph-zededa Btw. in my refactoring I added return statement here: https://github.com/milan-zededa/eve/blob/zedagent-lps-refactor/pkg/pillar/cmd/zedagent/localcommand.go#L43
I believe that it was missing unintentionally: https://github.com/lf-edge/eve/blob/master/pkg/pillar/cmd/zedagent/localinfo.go#L714-L725

christoph-zededa · 2025-08-21T13:39:13Z

@christoph-zededa Btw. in my refactoring I added return statement here: https://github.com/milan-zededa/eve/blob/zedagent-lps-refactor/pkg/pillar/cmd/zedagent/localcommand.go#L43 I believe that it was missing unintentionally: https://github.com/lf-edge/eve/blob/master/pkg/pillar/cmd/zedagent/localinfo.go#L714-L725

Sounds good. Once it is merged, I will do a backport of that fix.

issue has been found here: lf-edge#5191 (comment) Signed-off-by: Christoph Ostarek <[email protected]>

OhmSpectator

New docs are super useful, thanks! Could you please link them in https://github.com/lf-edge/eve/blob/master/docs/mkdocs/mkdocs.yml ?

milan-zededa · 2025-08-28T13:34:09Z

New docs are super useful, thanks! Could you please link them in https://github.com/lf-edge/eve/blob/master/docs/mkdocs/mkdocs.yml ?

Sure, added link to mkdocs.yml

pkg/pillar/cmd/zedagent/handlenetworkinstance.go

pkg/pillar/localcommand/agent.go

eriknordmark · 2025-08-29T23:02:39Z

pkg/pillar/localcommand/agent.go

+	throttled    bool
+}
+
+// newTaskTicker creates a new taskTicker with a randomized firing interval.


The research paper from the 1980's says you need 0.5 to 1.5 to avoid synchronization.

I can put there anything, in this PR I just kept what was already there:

https://github.com/lf-edge/eve/blob/master/pkg/pillar/cmd/zedagent/localinfo.go#L49-L50

https://github.com/lf-edge/eve/blob/master/pkg/pillar/cmd/zedagent/localinfo.go#L521-L522

etc. (at least it is now only in one place for all LPS tasks, so we do not need to change it in many places)

eriknordmark · 2025-08-29T23:06:45Z

pkg/pillar/localcommand/agent.go

+	if lc.beforeStart != nil {
+		lc.beforeStart()
+	}
+	locked := lc.taskMx.TryRLock()


Did we hold a (read) lock across http invocations in the old/current implementation?
Means the read lock can be held for minutes.

We didn't (but then we had some potential race conditions in the old implementation).
But I enhanced this now so that Pause will not be blocked by HTTP request. I added function:

// runInterruptible temporarily releases the task lock to allow Pause() to proceed, // runs the provided callback, then re-acquires the lock. Returns true if a pause // was triggered while the callback was running, indicating the caller should // discard or retry the operation. func (lc *taskControl) runInterruptible(callback func()) (wasPaused bool) { ...

Which is now used to run HTTP operations without lock being held and discard results if there was a Pause while the HTTP request was running.

eriknordmark · 2025-08-29T23:07:50Z

pkg/pillar/localcommand/appinfo.go

+// GetLocalAppRestartCmd returns the most recent locally issued restart
+// command for the given app, or an empty command if none exists.
+func (lc *LocalCmdAgent) GetLocalAppRestartCmd(appUUID uuid.UUID) types.AppInstanceOpsCmd {
+	lc.appCommandsMx.RLock()


Are the callers ok with this potentially blocking for minutes when the http call needs to time out?

This one is not locked during HTTP call execution.
Only taskMx was, but I changed it to avoid Pause being blocked by HTTP operations: #5191 (comment)

eriknordmark · 2025-08-29T23:09:44Z

pkg/pillar/cmd/zedagent/parseconfig.go

 	// from the primary controller is being applied. Or vice versa.
-	getconfigCtx.sideController.Lock()
-	defer getconfigCtx.sideController.Unlock()
+	resume := getconfigCtx.localCmdAgent.Pause()


Can't this block for minutes if a local task is waiting for http to time out while holding the read lock (from startTask)?

Good point -- fixed: #5191 (comment)

issue has been found here: lf-edge#5191 (comment) Signed-off-by: Christoph Ostarek <[email protected]>

eriknordmark

LGTM but please test with a config where the http post/get times due to unreachable/blocked connectivity to the server.

issue has been found here: #5191 (comment) Signed-off-by: Christoph Ostarek <[email protected]>

OhmSpectator · 2025-09-11T17:30:34Z

Hmmm... LPS/LOC tests fails: --- FAIL: TestEdenScripts/dev_local_info (1549.20s)

OhmSpectator

LOC/LPS tests are failing. Requesting a change to prevent occasional merging.

milan-zededa · 2025-09-16T09:59:41Z

LOC/LPS tests are failing. Requesting a change to prevent occasional merging.

I'm unable to reproduce these failures locally. So I temporarily added new commit with some extra log messages that might help. Could you please restart eden tests?

uncleDecart

kicking the tests!

OhmSpectator

Run the tests

milan-zededa · 2025-09-17T16:46:01Z

@uncleDecart / @OhmSpectator When you get a chance, please restart the eden tests again. I submitted a small change which should fix the failing tests (but for now keeping the temporary commit with extra logs).

OhmSpectator · 2025-09-17T16:54:17Z

They should start once the build is done

milan-zededa · 2025-09-17T21:13:41Z

Looks like the failing test is fixed, I removed the extra logs for troubleshooting.

- Refactored the LPS-handling code inside zedagent into a separate component, LocalCmdAgent. This helps mitigate the complexity and unorganized structure of the already large zedagent package. - No functional changes were introduced; the code was reorganized, cleaned up, and better documented. - Improved thread-safety for variables that previously lacked proper protection, reducing potential race conditions. - Added user-facing documentation for the Local Profile Server (LPS). - Added initial developer-facing documentation for zedagent, with focus on LPS handling. Signed-off-by: Milan Lenco <[email protected]>

milan-zededa · 2025-09-18T09:07:48Z

@OhmSpectator Finally everything is green. When you get a chance, please merge.

OhmSpectator · 2025-09-18T09:09:14Z

@OhmSpectator Finally everything is green. When you get a chance, please merge.

done

milan-zededa requested review from OhmSpectator, deitch, eriknordmark and rouming as code owners August 21, 2025 10:13

github-actions bot requested a review from uncleDecart August 21, 2025 10:13

milan-zededa force-pushed the zedagent-lps-refactor branch 3 times, most recently from ba5f340 to 4b3b3fc Compare August 21, 2025 10:53

christoph-zededa reviewed Aug 21, 2025

View reviewed changes

pkg/pillar/localcommand/devinfo.go Show resolved Hide resolved

christoph-zededa reviewed Aug 21, 2025

View reviewed changes

pkg/pillar/localcommand/devinfo.go Outdated Show resolved Hide resolved

christoph-zededa reviewed Aug 21, 2025

View reviewed changes

pkg/pillar/cmd/zedagent/parseconfig.go Outdated Show resolved Hide resolved

milan-zededa force-pushed the zedagent-lps-refactor branch from 4b3b3fc to 5929bd9 Compare August 21, 2025 12:58

milan-zededa force-pushed the zedagent-lps-refactor branch from 5929bd9 to 4cbb4cd Compare August 21, 2025 15:19

christoph-zededa added a commit to christoph-zededa/eve that referenced this pull request Aug 27, 2025

pillar/zedagent: fix shutdown after collectinfo

b5e5691

issue has been found here: lf-edge#5191 (comment) Signed-off-by: Christoph Ostarek <[email protected]>

OhmSpectator added the main-quest The fate of the project rests on this PR. Prioritise review to advance the storyline! label Aug 28, 2025

OhmSpectator reviewed Aug 28, 2025

View reviewed changes

milan-zededa force-pushed the zedagent-lps-refactor branch from 4cbb4cd to 030506a Compare August 28, 2025 13:33

github-actions bot requested a review from OhmSpectator August 28, 2025 13:34

milan-zededa force-pushed the zedagent-lps-refactor branch from 030506a to e5afc0c Compare August 28, 2025 13:37

eriknordmark reviewed Aug 29, 2025

View reviewed changes

pkg/pillar/cmd/zedagent/handlenetworkinstance.go Show resolved Hide resolved

eriknordmark reviewed Aug 29, 2025

View reviewed changes

pkg/pillar/localcommand/agent.go Show resolved Hide resolved

eriknordmark reviewed Aug 29, 2025

View reviewed changes

milan-zededa force-pushed the zedagent-lps-refactor branch from e5afc0c to 03d083b Compare September 1, 2025 08:38

github-actions bot requested a review from christoph-zededa September 1, 2025 08:38

milan-zededa force-pushed the zedagent-lps-refactor branch from 784a59e to fe8511c Compare September 3, 2025 11:35

github-actions bot requested a review from eriknordmark September 3, 2025 11:36

milan-zededa force-pushed the zedagent-lps-refactor branch 4 times, most recently from f0143a7 to fd53736 Compare September 3, 2025 12:14

christoph-zededa added a commit to christoph-zededa/eve that referenced this pull request Sep 3, 2025

pillar/zedagent: fix shutdown after collectinfo

3393111

issue has been found here: lf-edge#5191 (comment) Signed-off-by: Christoph Ostarek <[email protected]>

eriknordmark approved these changes Sep 4, 2025

View reviewed changes

OhmSpectator pushed a commit that referenced this pull request Sep 8, 2025

pillar/zedagent: fix shutdown after collectinfo

7ced0d8

issue has been found here: #5191 (comment) Signed-off-by: Christoph Ostarek <[email protected]>

OhmSpectator suggested changes Sep 12, 2025

View reviewed changes

github-actions bot requested review from OhmSpectator and eriknordmark September 16, 2025 09:59

milan-zededa force-pushed the zedagent-lps-refactor branch from ddf1d8c to b780715 Compare September 16, 2025 10:01

uncleDecart approved these changes Sep 16, 2025

View reviewed changes

OhmSpectator approved these changes Sep 16, 2025

View reviewed changes

milan-zededa force-pushed the zedagent-lps-refactor branch from b780715 to efecb3e Compare September 17, 2025 16:43

github-actions bot requested review from OhmSpectator and uncleDecart September 17, 2025 16:43

milan-zededa force-pushed the zedagent-lps-refactor branch from efecb3e to cda6d4f Compare September 17, 2025 16:44

milan-zededa force-pushed the zedagent-lps-refactor branch from cda6d4f to f9edbaa Compare September 17, 2025 21:12

milan-zededa force-pushed the zedagent-lps-refactor branch from f9edbaa to 1bc9707 Compare September 17, 2025 21:16

OhmSpectator merged commit 3ad0e22 into lf-edge:master Sep 18, 2025
73 of 75 checks passed

zedagent: refactor LPS handling into LocalCmdAgent component #5191

zedagent: refactor LPS handling into LocalCmdAgent component #5191

Uh oh!

Conversation

milan-zededa commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to test and validate this PR

Changelog notes

PR Backports

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

milan-zededa commented Aug 21, 2025

Uh oh!

christoph-zededa commented Aug 21, 2025

Uh oh!

OhmSpectator left a comment

Choose a reason for hiding this comment

Uh oh!

milan-zededa commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eriknordmark Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

milan-zededa Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

milan-zededa Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

milan-zededa Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eriknordmark Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

milan-zededa Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eriknordmark left a comment

Choose a reason for hiding this comment

Uh oh!

OhmSpectator commented Sep 11, 2025

Uh oh!

OhmSpectator left a comment

Choose a reason for hiding this comment

Uh oh!

milan-zededa commented Sep 16, 2025

Uh oh!

uncleDecart left a comment

Choose a reason for hiding this comment

Uh oh!

OhmSpectator left a comment

Choose a reason for hiding this comment

Uh oh!

milan-zededa commented Sep 17, 2025

Uh oh!

OhmSpectator commented Sep 17, 2025

Uh oh!

milan-zededa commented Sep 17, 2025

Uh oh!

milan-zededa commented Sep 18, 2025

Uh oh!

Uh oh!

milan-zededa commented Aug 21, 2025 •

edited

Loading

milan-zededa commented Aug 28, 2025 •

edited

Loading

milan-zededa Sep 1, 2025 •

edited

Loading

milan-zededa Sep 1, 2025 •

edited

Loading

milan-zededa Sep 1, 2025 •

edited

Loading

milan-zededa Sep 1, 2025 •

edited

Loading