ci: emulated perf workflow #7949

AndreCostaaa · 2025-03-18T21:40:30Z

This PR adds a new workflow to run performance benchmarks in an emulated ARM environment

LVGL runs on top of so3, a lightweight operating system. The docker image used is provided by so3 with a special config for theses tests. You can read all the details about the images used here. The file also explains how you can run this image locally

This PR adds:

Script Checks: Ensures the workflow scripts are working before execution.
Benchmark Workflow: Runs performance tests in an emulated setup.
Stable Timing: Since it’s emulated, results don’t depend on the runner’s hardware.
- I've personally tested running on a 2 Core Celeron N3060 @ 2.48 GHz, and while the app takes ~8 minutes to run, the results are the same as in my 12 Core Ryzen 5 3600 running @ 4.20 GHz.
- In a standard github runner, the whole workflow takes ~3 minutes to run
PR Comments: If this runs on a PR, it’ll compare results with previous benchmarks (if they exist) and comment on the findings.
Master: Stores results in a "pre-release" when merged to master.

Why This?

To make sure performance stays in check and we can track regressions easily. No more guessing if something slowed down

You can check this repository to see what it looks like

These two PRs 1 2 showcase the PR part. Including the comparison of results between master and PR 2.

This is the "pre-release" this workflow adds

Not sure if we need to add some documentation about this somewhere, i've tried to be very explicit and added some documentation to the different python scripts

Let me know what you think

@kisvegabor @liamHowatt (cc @daniel-rossier)

AndreCostaaa · 2025-03-18T22:26:38Z

Seem like there’s a permission problem when trying to write the PR comment

AndreCostaaa · 2025-03-19T15:22:19Z

I was able to figure it out

When running a workflow triggered by the pull_request event no write permissions are given to the GITHUB_TOKEN

So in order to make it work i've separated the "running the benchmark" and the "checking results" parts
The results are passed as artifacts between the workflows

Right now the second workflow isn't triggered because the workflow file must exist in the master branch which it doesn't... yet. I could split this PR into two :

Add the workflow that checks the results, creates the PR comment and the Pre-Release
Add the workflow that runs the benchmark and exports the results

This way, when PR 2 is created, the workflow will already exist in master and it will be triggered, causing it to create the PR comment and the "baseline" results

Let me know what you prefer !

daniel-rossier · 2025-03-21T16:55:09Z

I think we need @kisvegabor and @liamHowatt feedback, right?

AndreCostaaa · 2025-03-22T07:40:23Z

That's right :)

AndreCostaaa · 2025-04-13T20:10:44Z

Hey just pushed some updates:

Updated the docker image. The new version adds python, pcpp and removes a lot of verbosity.
- build: remove pcpp dependency #8090 removes pcpp but it's not a big dependency so it doesn't hurt to have it in the image
- Based on the discussion in test(perf): add perf test framework #8064, i believe I will have to update the image again to point to lvgl io instead of so3
Added a way to easily add new dependencies to the docker image via a shell script
Added a disclaimer to the PR comment that gets generated
Very rarely, the script fails because of a run condition when creating a loop device, see https://github.com/lvgl/lvgl/actions/runs/14432699079/job/40469393600 so to circumvent failing the CI in case that happens I added some retry logic because it should never fail twice in a row

kisvegabor · 2025-04-15T08:09:12Z

Thank you Andre!
Let's get back to this after the release of v9.3.

lvgl-bot · 2025-05-01T02:14:28Z

We need some feedback on this pull request.

Now we mark this as "Abandoned" because there was no activity here for 14 days.

Remove the "Abandoned" label or comment else this will be closed in 7 days.

AndreCostaaa · 2025-06-14T16:02:21Z

smartobjectoriented/so3#166 has now been merged and this PR has been updated and ready for another round of reviews @liamHowatt @kisvegabor .
I modified the image version, removed the retry logic from the CI as it should no longer be necessary and modified the docs so it made more sense after the other PR got merged. You can see all of that in the latest two commits bf77d58 and 34e096e

kisvegabor

I've just checked the high level things. We will see better how it works in the practice.

We have som many tests now, that a bullet point summary at the beginning of teh README would be great too.

kisvegabor · 2025-06-19T10:14:28Z

scripts/perf/tests/benchmark_results_comment/no_mpk/expected.md

@@ -0,0 +1,83 @@
+Hi :wave:, thank you for your PR!


What does no_mpk mean?

MessagePack is the file format used to store the performance history: https://msgpack.org/index.html
There's one mpk file per config so this the expected comment when such file doesn't exist (i.e. we don't have a history file for that specific config)

tests/README.md

AndreCostaaa · 2025-06-19T11:56:14Z

Concerns should be handled in latest 3 commits @kisvegabor

… fix

kisvegabor · 2025-06-23T18:48:28Z

It works well, thank you! I just got these warnings:

Warning: The guest is now late by 0.0 to 1.0 seconds
Warning: The guest is now late by 1.0 to 2.0 seconds
Warning: The guest is now late by 2.0 to 3.0 seconds
Warning: The guest is now late by 1.0 to 2.0 seconds
Warning: The guest is now late by 2.0 to 3.0 seconds
Warning: The guest is now late by 3.0 to 4.0 seconds
Warning: The guest is now late by 4.0 to 5.0 seconds
Warning: The guest is now late by 5.0 to 6.0 seconds

I don't know if it's normal.

AndreCostaaa · 2025-06-23T21:19:03Z

It is! Since QEMU is slowed down it's basically telling you the difference in the emulation vs real world

AndreCostaaa · 2025-06-26T06:09:15Z

Thank you for approving. I'll merge this now as is so we can iterate on it to make it perfect 👌

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch 4 times, most recently from 6cd518c to 4c209be Compare March 18, 2025 21:51

AndreCostaaa marked this pull request as draft March 18, 2025 22:20

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch 4 times, most recently from 1735c55 to b6681d8 Compare March 19, 2025 15:12

AndreCostaaa marked this pull request as ready for review March 19, 2025 15:22

kisvegabor requested review from kisvegabor and liamHowatt March 24, 2025 09:11

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch from b6681d8 to 6782fc8 Compare April 9, 2025 12:32

kisvegabor added the 👀 Review needed Anyone can review a PR label Apr 10, 2025

AndreCostaaa marked this pull request as draft April 13, 2025 16:10

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch 4 times, most recently from e004d50 to 0bd7041 Compare April 13, 2025 19:56

AndreCostaaa marked this pull request as ready for review April 13, 2025 20:01

AndreCostaaa mentioned this pull request Apr 13, 2025

test(perf): add perf test framework #8064

Merged

lvgl-bot added the 💤 Stale label May 1, 2025

AndreCostaaa removed the 💤 Stale label May 1, 2025

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch 2 times, most recently from 29953fe to 34e096e Compare June 14, 2025 15:51

AndreCostaaa requested a review from liamHowatt June 14, 2025 16:02

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch from 34e096e to 67b4698 Compare June 15, 2025 19:27

kisvegabor reviewed Jun 19, 2025

View reviewed changes

AndreCostaaa added 14 commits June 19, 2025 14:00

feat(config): add perf configs

16f8217

feat(scripts): add docker benchmark logs filter

ad59f2b

feat(scripts): add benchmark results comment generator

7f6dad0

feat(scripts): add serialize results script

e78a292

ci: add emulated perf check workflow

b9ff2a1

ci(perf): support adding extra build dependencies via script

eee1b06

ci: add emulated perf results disclaimer to pr comment

c4929d3

ci: add retry logic to avoid rare failures on race conditions

c756fbf

docs: document how to run performance tests locally

16eaacf

ci: update lvperf images with 9.3 support and mounting race condition…

a53b0c2

… fix

docs: update performance test section in tests readme

c9494de

chore: add python script to run emulated benchmarks

cd3dc4b

docs: fix branch name

7414bdb

docs: add quick start to tests readme

209c84c

AndreCostaaa force-pushed the ci-emulated-perf-workflow branch from f2fd419 to 209c84c Compare June 19, 2025 12:00

kisvegabor approved these changes Jun 23, 2025

View reviewed changes

uLipe approved these changes Jun 25, 2025

View reviewed changes

AndreCostaaa merged commit 47977f3 into lvgl:master Jun 26, 2025
42 checks passed

AndreCostaaa deleted the ci-emulated-perf-workflow branch June 26, 2025 06:09

AndreCostaaa added a commit to AndreCostaaa/lvgl that referenced this pull request Aug 25, 2025

ci: emulated perf workflow (lvgl#7949)

60edf65

Uh oh!

ci: emulated perf workflow #7949

ci: emulated perf workflow #7949

Uh oh!

Conversation

AndreCostaaa commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why This?

Uh oh!

AndreCostaaa commented Mar 18, 2025

Uh oh!

AndreCostaaa commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daniel-rossier commented Mar 21, 2025

Uh oh!

AndreCostaaa commented Mar 22, 2025

Uh oh!

AndreCostaaa commented Apr 13, 2025

Uh oh!

kisvegabor commented Apr 15, 2025

Uh oh!

lvgl-bot commented May 1, 2025

Uh oh!

AndreCostaaa commented Jun 14, 2025

Uh oh!

kisvegabor left a comment

Choose a reason for hiding this comment

Uh oh!

kisvegabor Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

AndreCostaaa Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AndreCostaaa commented Jun 19, 2025

Uh oh!

kisvegabor commented Jun 23, 2025

Uh oh!

AndreCostaaa commented Jun 23, 2025

Uh oh!

AndreCostaaa commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

AndreCostaaa commented Mar 18, 2025 •

edited

Loading

AndreCostaaa commented Mar 19, 2025 •

edited

Loading