Fix handling of external texture transforms on Android #24888
Conversation
|
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat. If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix? Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. |
There was a problem hiding this comment.
I think the change sounds reasonable. I had a few questions that I added. You're going to have to add tests. It's difficult to run the math through your head and know that it draws correctly. I think the best thing would be to add a golden image test that shows the texture getting drawn correctly. You can see an example golden image test here:
|
|
||
| SkPaint paint; | ||
| paint.setShader(shader); | ||
| canvas.drawRect(SkRect::MakeWH(1, 1), paint); |
There was a problem hiding this comment.
Will this work for images that aren't square? It seems like the transform may not include the scalar for the aspect ratio considering how it is used in the else branch.
There was a problem hiding this comment.
The call to canvas.scale above applies the aspect ratio needed to present the layer. After scaling, canvas coordinates from [0, 1] map directly to [bounds.x/y, +bounds.width/height] in their respective dimensions.
|
|
||
| SkMatrix inverted; | ||
| if (!transform.invert(&inverted)) { | ||
| FML_LOG(FATAL) << "Invalid SurfaceTexture transformation matrix"; | ||
| } | ||
| transform = inverted; | ||
|
|
||
| transform.preScale(1.0, -1.0); | ||
| transform.postTranslate(0.0, -1.0); |
There was a problem hiding this comment.
Would it be more appropriate to fix the math in the java layer? It seems like keeping the lower level more dumb would be preferable, unless that is breaking user's code.
There was a problem hiding this comment.
I'm not entirely sure if this should live in the Java layer. The SurfaceTexture transform matrix is defined as transformations on texture coordinate lookup.
The only reason we need to invert, scale, and translate, is that we aren't able to supply a matrix that transforms texture lookup to Skia, but we are able to supply a matrix that transforms the image inside of a fixed size box (which is the inverse). It's inherently tied to how we present the surface inside the engine.
Also, Android only has APIs exposed for working on 4x4 matrices, and lacks some pieces that Skia provides us (like preScale).
| // Skia (Row Major): | ||
| // | m[0] m[1] m[2] | | ||
| // | m[3] m[4] m[5] | | ||
| // | m[6] m[7] m[8] | |
There was a problem hiding this comment.
Why is the skia format described here? I'm not seeing a translation of the matrix to convert between column major and row major. The matrix3 format looks like like the column major format you describe.
There was a problem hiding this comment.
If this were a row-major -> row-major conversion, the matrix accesses below would be in the form:
m[0], m[1], m[2],
...
since this would produce an identical matrix.
The formats described above are the formats when written in standard matrix notation, which is row-major when read from left to right.
|
cc @chinmaygarde, I gave this PR a review but I think you might be the person that knows the most about this code, so heads up. |
|
It hard for me to verify the math here. I'll take a closer look in a but @ds84182, are you able to setup a test for this like Aaron suggested? |
|
I haven't had a chance to setup the test yet, I'll try to get to that later this week @chinmaygarde |
|
@ds84182 Any updates? |
|
I've been a little swamped the past week so I haven't had a chance to write the screenshot test. Preferably we should also compare against what SurfaceView gives for the same surface, right? |
|
I am not sure if that strict expectation can be satisfied. But it is certainly worth a shot. |
|
So I've got a test written that compares a SurfaceView with Flutter, which helped me find a bug with this PR. I'll split the test from this change since it's a bit larger than the PR itself. Getting Android to produce Surfaces with correct transforms and crops has been an.. experience to say the least.
|
|
cc @gaaclarke for followups and @blasten for routing. |
|
@ds84182 @gaaclarke , is there any action item left? |
|
This is waiting on #26109, then I'll add the new goldens to this PR. |
|
The linked PR was just closed as stale. Closing this dependency. Please reopen if progress can be made. |
|
This pull request was opened against a branch other than main. Since Flutter pull requests should not normally be opened against branches other than main, I have changed the base to main. If this was intended, you may modify the base back to master. See the Release Process for information about how other branches get updated. Reviewers: Use caution before merging pull requests to branches other than main, unless this is an intentional hotfix/cherrypick. |
|
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!). If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix? Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. |
1 similar comment
|
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!). If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix? Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. |
|
Reopening this now that external textures have tests on Android. May need to add an ASSERT if the incoming SkPaint already has an SkShader, which is less than ideal. I'll investigate if there's a way to avoid using SkShader for this. |
|
It works fine on Android with Who want to test it: Flutter 3.1.0-0.0.pre.2225 • channel master • https://github.com/flutter/flutter.git pubspec:
Thank a lot for this fix |
|
|
|
works like charm, thanks!. Now I have to solve video queue issues in my app |
|
awesome! when would this be on stable channel? |
|
I am using flutter 3.3.9 and this problem still exists. flutter 3.3.9 has this fix? |
|
it is on master branch, not on stable releases
El mié, 30 nov 2022 a la(s) 08:48, Cool Apps ***@***.***)
escribió:
… I am using flutter 3.3.9 and this problem still exists. flutter 3.3.9 has
this fix?
—
Reply to this email directly, view it on GitHub
<#24888 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARW46ESKX23PWY7D4T2F5RTWK45HRANCNFSM4Y24OIZA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
*Rafael Sandoval*
Desarrollador - F: 228953717
www.carpetres.cl
|
|
When it will be available on stable releases? Should I switch to master branch until it will be on stable? |
|
yeah, you should switch instead.
The fix has been done for a long time now, but the test are not ended
El mié, 30 nov 2022 a la(s) 12:14, Cool Apps ***@***.***)
escribió:
… When it will be available on stable releases? Should I switch to master
branch until it will be on stable?
—
Reply to this email directly, view it on GitHub
<#24888 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARW46ETOT4W33REMTSW742LWK5VNBANCNFSM4Y24OIZA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
*Rafael Sandoval*
Desarrollador - F: 228953717
www.carpetres.cl
|
|
Switched to master channel and problem has gone. I hope it will be merged to stable channel as soon as possible. |
|
How do we know that its on stable? It will show on flutter releases, for example flutter release changelog 3.3.0? |
Android provides a transformation matrix for SurfaceTextures to allow decoders, cameras, etc. to have the flexibility with the buffers they output, while allowing applications to sample them regardless of layout. However, outside of identity matrices, Flutter was not using this correctly:
Fixes: flutter/flutter#58138 (And possibly more issues with stranger device buffer layouts)
Pre-launch Checklist
writing and running engine tests.
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.