feat: GPU shared texture offscreen rendering #42953

reitowo · 2024-07-18T03:57:09Z

Description of Change

Reopen #42001

I managed to add ARGB + kGpuMemoryBuffer support for FrameSinkVideoCapturer in recent upstream changes. Thus, we can finally use zero-copy (actually one, there's a CopyRequest of frame texture) GPU shared texture OSR in chromium apps.

This will be the fastest OSR mode than the existing two, and it supports enabling hardware acceleration.

However, this mode will not compose the popup widgets, directly provide both window types' textures to user. It is expected that user themselves to compose these textures at their rendering pipeline.

Details:

Add webPreferences.offscreen.useSharedTexture to enable this feature.
Add texture to paint event of webContent.
Add structure definitions and docs.
Add tests about this feature.

Fix: #41972

Checklist

PR description included and stakeholders cc'd
npm test passes
tests are changed or added
relevant documentation, tutorials, templates and examples are changed or added
PR release notes describe the change in a way relevant to app developers, and are capitalized, punctuated, and past tense.

Release Notes

Notes: feat: GPU accelerated shared texture offscreen rendering.

github-actions

Hello @reitowo! It looks like this pull request touches one of our dependency files, and per our contribution policy we do not accept these types of changes in PRs.

reitowo · 2024-07-18T06:27:10Z

Well, it seems the CI machine failed to create a D3D11 device.

Trying to make this optional.

The change requested was to not have unknown contributors modifying yarn.lock, but the change here is just to add an internal vendorized dependency for specs. It's harmless.

docs/api/web-contents.md

docs/tutorial/offscreen-rendering.md

shell/browser/api/electron_api_web_contents.cc

spec/fixtures/native-addon/osr-gpu/binding_win.cc

samuelmaddock · 2024-07-21T05:56:29Z

patches/chromium/osr_shared_texture_remove_keyed_mutex_on_win_dxgi.patch

+  // once the GMB is returned from CopyRequest. So there will be no race
+  // condition on that texture. We can request a GMB without a keyed mutex to
+  // accelerate and probably prevent some driver deadlock.
+  if (format_ == PIXEL_FORMAT_ARGB || format_ == PIXEL_FORMAT_ABGR) {


Electron's desktopCapturer uses this format. Would it be affected?

electron/shell/browser/api/frame_subscriber.cc

Line 50 in 7a101a1

video_capturer_->SetFormat(media::PIXEL_FORMAT_ARGB);

As long as that code use BufferFormatPreference::kDefault which will only requests a CPU only buffer, it won't be affected by this, this patch made changes to renderable_gpu_memory_buffer_video_frame_pool which is a GPU related buffer code.

Even it use GPU texture later (BufferFormatPreference::kPreferGpuMemoryBuffer), it will not be affected if you won't write to the texture again. The patch just remove the keyed mutex on that texture.

PS: Does it mean desktopCapturer also possible to use GPU for acceleration? Just randomly thinking...

reitowo · 2024-07-25T18:31:40Z

Rebased conflict.

docs/api/structures/web-preferences.md

erickzhao

Thanks for this PR! Some API comments. :)

docs/api/web-contents.md

docs/api/structures/offscreen-shared-texture.md

itsananderson · 2024-08-02T19:14:56Z

If someone enables this feature, but neglects to call texture.release(), it sounds like that could cause a memory leak. Would it be possible to monitor for when the JS texture object is getting GC'd and either release the shared texture automatically (if that is safe to do), or print a warning so that the developer can detect that they're leaking memory?

There is an EmitWarning function that you could use to log this warning. You'd also only want to log it once per process lifetime, to avoid spamming for every paint event.

reitowo · 2024-08-02T19:20:09Z

Definetely ok, but not necessary. People use OSR features are expected to receive continuous frames. And since it will be drained only after 10-11 frames, the user will no longer receive any updates, so they must notice this as a bug.

Relying on GC is not working because the unknown timing. 10 frames are only 160ms at 60fps and GC are not running that fast.

However I think it is best to add that hint when it got GC-ed but not manually released for better understanding whats going on. 👍🏻 (If theres really a way to monitor that)

reitowo · 2024-08-05T06:56:19Z

Implemented it, fun feature. @itsananderson

Note the warning won't show until next GC, so its kinda slow under testing enviroment. You can manually trigger GC to see this.

samuelmaddock · 2024-08-05T15:00:42Z

@reitowo looks like the patch needs to be updated as it's failing to apply.

reitowo · 2024-08-05T15:06:27Z

I noticed that and I already updated the patch.

Terrifying that they are working on opt-out GpuMemoryBuffer......

PS: For future maintaince of this patch please @ me at any time.

reitowo · 2024-08-05T18:51:43Z

Fixed the patch and verified to work! Hope we can merge this in next days 😎

release-clerk · 2024-08-23T00:23:16Z

Release Notes Persisted

feat: GPU accelerated shared texture offscreen rendering.

reitowo · 2024-08-23T00:47:24Z

Great!

Thanks for all your work reviewing this PR and maintaining electron. I'm glad to be able to make this contribution! ❤️

reitowo · 2024-09-11T16:37:09Z

Just to ask, is it possible to backport this feature to previous Electron versions? (32)

erickzhao · 2024-09-11T17:50:22Z

Hi @reitowo,

If you would like to backport the PR to previous branches, a releaser can assist by running Trop automation. Backporting a feature PR will automatically put it up for a vote by the Releases working group, whereafter the PR can be accepted or denied on a case-by-case basis.

See https://www.electronjs.org/docs/latest/tutorial/electron-versioning#backport-request-process for more info.

As an aside (I'm not in the WG), I think the general policy for stable release lines is that PRs with large patches or changes to existing code paths are considered riskier and have a lower chance of being accepted.

VerteDinde · 2024-09-11T22:15:28Z

Thanks @erickzhao! @reitowo, I'll go ahead and run backports for you to 33-x-y and 32-x-y. To Erick's point, I think we're probably unlikely to backport a change this large into an already existing stable release line, but we can vote and determine whether or not it's a good fit 🙂

VerteDinde · 2024-09-11T22:15:39Z

/trop run backport-to 33-x-y

trop · 2024-09-11T22:15:42Z

The backport process for this PR has been manually initiated - sending your PR to 33-x-y!

trop · 2024-09-11T22:16:31Z

I have automatically backported this PR to "33-x-y", please check out #43694

VerteDinde · 2024-09-11T22:16:49Z

/trop run backport-to 32-x-y

trop · 2024-09-11T22:16:52Z

The backport process for this PR has been manually initiated - sending your PR to 32-x-y!

trop · 2024-09-11T22:16:59Z

I was unable to backport this PR to "32-x-y" cleanly;
you will need to perform this backport manually.

reitowo · 2024-09-12T01:28:24Z

Thanks! I got your point about bp into 32, thats reasonable, and it will still be nice if we can bp into 33. 😉

I'll still try manual bp into 32 and wait for your vote.

trop · 2024-11-01T16:22:44Z

@reitowo has manually backported this PR to "33-x-y", please check out #44509

trop · 2024-11-01T16:27:10Z

@reitowo has manually backported this PR to "32-x-y", please check out #44510

trop · 2024-11-01T17:06:07Z

@reitowo has manually backported this PR to "33-x-y", please check out #44511

reitowo requested a review from a team as a code owner July 18, 2024 03:57

electron-cation bot added the new-pr 🌱 PR opened recently label Jul 18, 2024

github-actions bot previously requested changes Jul 18, 2024

View reviewed changes

reitowo mentioned this pull request Jul 18, 2024

feat: GPU shared texture offscreen rendering #42001

Closed

5 tasks

samuelmaddock added the semver/minor backwards-compatible functionality label Jul 18, 2024

electron-cation bot added the api-review/requested 🗳 label Jul 18, 2024

samuelmaddock added the no-backport label Jul 18, 2024

ckerr self-requested a review July 19, 2024 19:47

ckerr requested a review from a team July 19, 2024 19:50

reitowo force-pushed the main branch from caedcd3 to c89800f Compare July 20, 2024 04:14

samuelmaddock reviewed Jul 21, 2024

View reviewed changes

erickzhao self-assigned this Jul 22, 2024

electron-cation bot removed the new-pr 🌱 PR opened recently label Jul 25, 2024

reitowo force-pushed the main branch from ea16943 to f38fbf8 Compare July 25, 2024 18:31

reitowo mentioned this pull request Aug 1, 2024

OSR mode with shared textures: pass out partial update information in "OnAcceleratedPaint" chromiumembedded/cef#3730

Closed

reitowo force-pushed the main branch from f38fbf8 to f0e7fdc Compare August 1, 2024 05:28

samuelmaddock reviewed Aug 2, 2024

View reviewed changes

docs/api/structures/web-preferences.md Outdated Show resolved Hide resolved

erickzhao reviewed Aug 2, 2024

View reviewed changes

docs/api/web-contents.md Outdated Show resolved Hide resolved

docs/api/structures/offscreen-shared-texture.md Show resolved Hide resolved

reitowo force-pushed the main branch from 292f68c to 8eef094 Compare August 5, 2024 07:06

reitowo force-pushed the main branch 2 times, most recently from c72e37c to 6bd174d Compare August 5, 2024 18:50

trop bot mentioned this pull request Sep 11, 2024

feat: GPU shared texture offscreen rendering #43694

Closed

trop bot added the in-flight/33-x-y label Sep 11, 2024

VerteDinde removed the no-backport label Sep 11, 2024

trop bot added the needs-manual-bp/32-x-y label Sep 11, 2024

reitowo mentioned this pull request Nov 1, 2024

feat: GPU shared texture offscreen rendering (33-x-y) #44509

Closed

reitowo mentioned this pull request Nov 1, 2024

feat: GPU shared texture offscreen rendering (32-x-y) #44510

Closed

trop bot added in-flight/32-x-y and removed needs-manual-bp/32-x-y in-flight/33-x-y labels Nov 1, 2024

reitowo mentioned this pull request Nov 1, 2024

feat: GPU shared texture offscreen rendering (33-x-y) #44511

Merged

trop bot added in-flight/33-x-y merged/33-x-y PR was merged to the "33-x-y" branch. and removed in-flight/33-x-y in-flight/32-x-y labels Nov 1, 2024

RenaudRohlinger mentioned this pull request Apr 21, 2025

importExternalTexture: Support importing platform‑shared GPU textures (DXGI / IOSurface / dma‑buf) gpuweb/gpuweb#5167

Closed

feat: GPU shared texture offscreen rendering #42953

feat: GPU shared texture offscreen rendering #42953

Conversation

reitowo commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Change

Checklist

Release Notes

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

reitowo commented Jul 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samuelmaddock Jul 21, 2024

Choose a reason for hiding this comment

Uh oh!

reitowo Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reitowo commented Jul 25, 2024

Uh oh!

Uh oh!

erickzhao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

itsananderson commented Aug 2, 2024

Uh oh!

reitowo commented Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reitowo commented Aug 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelmaddock commented Aug 5, 2024

Uh oh!

reitowo commented Aug 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reitowo commented Aug 5, 2024

Uh oh!

release-clerk bot commented Aug 23, 2024

Uh oh!

reitowo commented Aug 23, 2024

Uh oh!

reitowo commented Sep 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erickzhao commented Sep 11, 2024

Uh oh!

VerteDinde commented Sep 11, 2024

Uh oh!

VerteDinde commented Sep 11, 2024

Uh oh!

trop bot commented Sep 11, 2024

Uh oh!

trop bot commented Sep 11, 2024

Uh oh!

VerteDinde commented Sep 11, 2024

Uh oh!

trop bot commented Sep 11, 2024

Uh oh!

trop bot commented Sep 11, 2024

Uh oh!

reitowo commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trop bot commented Nov 1, 2024

Uh oh!

trop bot commented Nov 1, 2024

Uh oh!

reitowo commented Jul 18, 2024 •

edited

Loading

reitowo commented Jul 18, 2024 •

edited

Loading

reitowo Jul 22, 2024 •

edited

Loading

reitowo commented Aug 2, 2024 •

edited

Loading

reitowo commented Aug 5, 2024 •

edited

Loading

reitowo commented Aug 5, 2024 •

edited

Loading

reitowo commented Sep 11, 2024 •

edited

Loading

reitowo commented Sep 12, 2024 •

edited

Loading