[video_player_avfoundation] enable more than 30 fps #7466

misos1 · 2024-08-21T16:18:33Z

In the player there was hardcoded 30 fps when setting up video composition. Now it uses timing from source track and also fallback minFrameDuration as seems frameDuration must be always set to something and it takes over in some situations as is mentioned in documentation about sourceTrackIDForFrameTiming. Also video composition setup is skipped when it is not needed when preferredTransform is identity.

Function updatePlayingState is called often right after setupEventSinkIfReadyToPlay but seems it was forgotten in onListenWithArguments and also it cannot be called in that way because setupEventSinkIfReadyToPlay may finish asynchronously when called again from line [self performSelector:_cmd onThread:NSThread.mainThread withObject:self waitUntilDone:NO] so now is updatePlayingState called right after _isInitialized = YES which is what it needs to even do something.

There was one more obstacle for playing 60 fps videos on 60 hz screen. Seems there are at least two "display links" at play when playing video, one calls function displayLinkFired and other one from flutter engine calls copyPixelBuffer but only when textureFrameAvailable was called previously. But the order in which those two are called is undefined so 16 ms after displayLinkFired may be called copyPixelBuffer and right after that displayLinkFired and so on. But copyPixelBuffer steals the newest pixel buffer from video player output and in displayLinkFired hasNewPixelBufferForItemTime will not report another pixel buffer for time close to that. Then the next frame is not called copyPixelBuffer because textureFrameAvailable was not called and in this way it skips every second frame so it plays video at 30 fps. There was also a synchronization problem with lastKnownAvailableTime. Now pixel buffers are produced and reported just on a single place in displayLinkFired and received with correct synchronization in copyPixelBuffer. Ideally there would be just a single "display link" from flutter engine if it supported also pulling frames instead of only pushing (which is good enough for a camera where the system is pushing frames to us, but from player video output is needed to pull frames). Calling textureFrameAvailable every frame could accomplish that but it looks like this line in flutter engine is calling even when copyPixelBuffer returns NULL and it may be expensive (although there is no need to call it in such case):

sk_sp<flutter::DlImage> image = [self wrapExternalPixelBuffer:_lastPixelBuffer context:context];

Seems there is some bug with the video player using this flutter engine on macos. Looks like the video is playing normally but then it starts "tearing", it looks like it is displaying frames normally but once in a while it shows some frame from the past like some previously cached frame. This is happening on the main branch but rendering on 60 fps exaggerates it (it is not caused by this PR).

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

stuartmorgan-g · 2024-08-23T00:28:57Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

@@ -21,27 +21,38 @@ @interface FVPFrameUpdater : NSObject
 @property(nonatomic, weak, readonly) NSObject<FlutterTextureRegistry> *registry;
 // The output that this updater is managing.
 @property(nonatomic, weak) AVPlayerItemVideoOutput *videoOutput;
-// The last time that has been validated as avaliable according to hasNewPixelBufferForItemTime:.
-@property(nonatomic, assign) CMTime lastKnownAvailableTime;
+@property(nonatomic) CVPixelBufferRef latestPixelBuffer;


All properties need comments, per the style guide. Please explain in comments what the purpose of these two new properties is.

stuartmorgan-g · 2024-08-23T00:45:43Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

+      if (self.latestPixelBuffer) {
+        CFRelease(self.latestPixelBuffer);
+      }
+      self.latestPixelBuffer = [self.videoOutput copyPixelBufferForItemTime:outputItemTime


This is essentially doubling the memory usage for video output, isn't it? Why doesn't the previous approach of only storing the timestamp work? The PR description discusses the early-consume problem, but it seems like that could be addressed simply by changing copyPixelBuffer to prefer the last time instead of the current time.

This is essentially doubling the memory usage for video output, isn't it?

Can you please explain how? Maybe memory usage with the original approach would be one frame less if flutter engine deleted its previous pixel buffer right before calling copyPixelBuffer but it would not want to do that because copyPixelBuffer can also return NULL and then it needs something latest to show.

Can you please explain how?

Aren't you keeping an extra copy of the frame besides the one kept by the player and the one kept by the engine?

I guess not doubled, but increasing by one frame relative to the current implementation.

(Also looking again, the current PR code appears to be leaking every frame it consumes.)

(Also looking again, the current PR code appears to be leaking every frame it consumes.)

I cannot see it.

Oh, I see now. The memory flow is pretty hard to follow here on the copied buffer as currently written, with the buffer sometimes freed by the frame updater, and sometimes handed off to to the engine.

Which brings us back to the initial question: why do we need to copy the buffer proactively when the display link fires instead of storing the timestamp?

Aren't you keeping an extra copy of the frame besides the one kept by the player and the one kept by the engine?

Both versions have a worst case number of pixel buffers 2+N where N is the number held by the player. This case is between copyPixelBufferForItemTime and until the engine replaces its own after copyPixelBuffer. Current version just can have 2+N for a little longer, especially when displayLinkFired is called after copyPixelBuffer at each frame. Btw if the player kept only a single frame then storing timestamp instead of buffer would not work.

Which brings us back to the initial question: why do we need to copy the buffer proactively when the display link fires instead of storing the timestamp?

That would mean fetching pixel buffers from the past, especially when displayLinkFired is called after copyPixelBuffer at each frame. But this is based on undocumented and possibly wrong assumption that the player always leaves at least one past frame ready for us. What if this is not the case? Actually it is not. Seems the player periodically flushes all past pixel buffers. After there are some 3 pixel buffers in the past then the player flushes them all.

I tried an implementation which was sending timestamps instead of pixel buffers and copyPixelBufferForItemTime often returned NULL with timestamp for which hasNewPixelBufferForItemTime returned true before. It had 4x more frame drops in average during my tests compared to little modified current implementation (with copyPixelBufferForItemTime moved outside of pixelBufferSynchronizationQueue).

There are several causes of frame drops. This modified implementation minimises cases where copyPixelBufferForItemTime returns NULL and accidentally also frame drops caused by artefacts of using two "display links". They are caused by displayLinkFired and copyPixelBuffer changing order. Because things running on pixelBufferSynchronizationQueue are short (in time) and new pixel buffer is generated after some time then even when is copyPixelBuffer called after displayLinkFired at some frame (while before it was conversely) it has chance to obtain pixel buffer from displayLinkFired from previous frame and to not clear _textureFrameAvailable from this latest displayLinkFired. But this is of course not ideal, a more proper way would be to wait until the middle of frame and then send a new pixel buffer and call textureFrameAvailable but even more proper solution would be to not use two "display links".

Another cause of frame drops is when hasNewPixelBufferForItemTime returns false which is now prevalent. Seems this is caused by irregularities at which is called displayLinkFired which causes irregular timestamps returned by CACurrentMediaTime. I tested to use CADisplayLink::timestamp instead and frame drops dropped almost to zero, below 0.1%, around 20x less than current implementation (modified) on average during my tests. But this would need access to CADisplayLink through FVPDisplayLink and some other implementation for macos.

I also tested an implementation without a display link where textureFrameAvailable and copyPixelBuffer were called for each frame and it had around 3x less frame drops than the current implementation (modified). Unfortunately I could not use CADisplayLink::timestamp here so there were still frame drops due to irregularities. Flutter engine would need to provide something similar, or maybe it can be simply obtained by rounding CACurrentMediaTime down to refresh duration of display but that would then require update frequency of display from engine (or something what is doing FVPDisplayLink for macos).

That would mean fetching pixel buffers from the past, especially when displayLinkFired is called after copyPixelBuffer at each frame. But this is based on undocumented and possibly wrong assumption that the player always leaves at least one past frame ready for us. What if this is not the case? Actually it is not. Seems the player periodically flushes all past pixel buffers. After there are some 3 pixel buffers in the past then the player flushes them all.

I tried an implementation which was sending timestamps instead of pixel buffers and copyPixelBufferForItemTime often returned NULL with timestamp for which hasNewPixelBufferForItemTime returned true before. It had 4x more frame drops in average during my tests compared to little modified current implementation

Very interesting, thanks for the details! That definitely seems worth the slight memory hit.

Another cause of frame drops is when hasNewPixelBufferForItemTime returns false which is now prevalent.

Could you file an issue with the details of what you've found here for us to follow up on in the future? It sounds like you've done a lot of great investigation here that we should be sure to capture and track.

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

stuartmorgan-g · 2024-09-17T11:04:01Z

@misos1 Are you still planning on addressing the remaining comments?

misos1 · 2024-09-17T18:46:34Z

@stuartmorgan Yes, I was waiting for your input as I thought you wanted to handle when minFrameDuration is kCMTimeInvalid differently. I thought of setting frameDuration to some constant in such a case but also emit some warning log message like I saw sometimes a flutter engine does but I did not find any way how that is normally done in plugins.

stuartmorgan-g · 2024-09-17T20:17:42Z

Sorry, I didn't realize that was waiting for my input. I'll respond there.

stuartmorgan-g · 2024-09-17T20:27:24Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

 @end

 @implementation FVPFrameUpdater
 - (FVPFrameUpdater *)initWithRegistry:(NSObject<FlutterTextureRegistry> *)registry {
  NSAssert(self, @"super init cannot be nil");
  if (self == nil) return nil;
  _registry = registry;
-  _lastKnownAvailableTime = kCMTimeInvalid;
  return self;
 }

 - (void)displayLinkFired {
  // Only report a new frame if one is actually available.
  CMTime outputItemTime = [self.videoOutput itemTimeForHostTime:CACurrentMediaTime()];
  if ([self.videoOutput hasNewPixelBufferForItemTime:outputItemTime]) {


Hm, shouldn't these two lines be inside the dispatch_async? AVPlayerItemVideoOutput doesn't seem to be marked as threadsafe.

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

misos1 · 2024-09-26T18:13:57Z

Seems there is some bug with the video player using this flutter engine on macos. Looks like the video is playing normally but then it starts "tearing", it looks like it is displaying frames normally but once in a while it shows some frame from the past like some previously cached frame. This is happening on the main branch but rendering on 60 fps exaggerates it (it is not caused by this PR).

Seems this is caused by a bug in the flutter engine. There is this "You need to maintain a strong reference to textureOut until the GPU finishes execution of commands accessing the texture, because the system doesn’t automatically retain it.". But here is textureOut released right after CVMetalTextureCacheCreateTextureFromImage:

https://github.com/flutter/engine/blob/6f802b39ab0669eb6ba3272dff1d34e85febeb77/shell/platform/darwin/graphics/FlutterDarwinExternalTextureMetal.mm#L233-L249
https://github.com/flutter/engine/blob/6f802b39ab0669eb6ba3272dff1d34e85febeb77/shell/platform/darwin/graphics/FlutterDarwinExternalTextureMetal.mm#L177-L197

I thought that it flashes video frames from the past but I looked closely at video recorded from the screen and for example at one moment for 1/60 of a second it shows a frame from video 12 frames into the future. Maybe when the underlying textureOut is released this memory may be reused and overwritten by AVPlayer for decoding following video frames.

This does not explain so well another thing which I noticed. If copyPixelBuffer returns for some frame NULL then the engine shows a transparent image. But I will assume this is also caused by that use after free bug.

There is also this "Note that Core Video doesn’t explicitly declare any pixel format types as Metal compatible. Specify true for the kCVPixelBufferMetalCompatibilityKey option to create Metal-compatible buffers when creating or requesting Core Video pixel buffers.". Maybe the player should also specify this in pixBuffAttributes when creating AVPlayerItemVideoOutput?

stuartmorgan-g · 2024-09-26T18:33:21Z

Seems this is caused by a bug in the flutter engine.

Please definitely file an issue with details if you haven't already!

Edited to add: You can cross-reference the engine issue with flutter/flutter#135999, which sounds like the macOS playback issue you are describing here if I'm understanding correctly.

stuartmorgan-g · 2024-09-26T18:35:28Z

There is also this "Note that Core Video doesn’t explicitly declare any pixel format types as Metal compatible. Specify true for the kCVPixelBufferMetalCompatibilityKey option to create Metal-compatible buffers when creating or requesting Core Video pixel buffers.". Maybe the player should also specify this in pixBuffAttributes when creating AVPlayerItemVideoOutput?

This would probably be a question for the #hackers-engine channel; I'm not familiar with the current end-to-end pipeline requirements for the engine texture rendering path.

stuartmorgan-g · 2024-09-26T18:44:55Z

In terms of moving this forward, from my perspective:

Structurally everything here makes sense to me at this point; thanks again for the clear explanations!
I don't consider the macOS issue blocking as it's an existing issue. If we get feedback that the experience is substantially worse we can always do a follow-up to disable just >30fps for macOS, while leaving all the other improvements (e.g., threading model fixes) in place.

So once the smaller feedback items still open are addressed in an an updated version of the PR, I can re-review, and I expect we'll be on track for getting this landed.

Does that sound right?

misos1 · 2024-09-26T19:19:35Z

If we get feedback that the experience is substantially worse we can always do a follow-up to disable just >30fps for macOS, while leaving all the other improvements (e.g., threading model fixes) in place.

Yes, video composition can be set every time even with affinity and fps forced to 30 but it still happens. I wonder why no one reported this. And there are other things which may seemingly help little with it like retaining pixel buffer by frame updater and creating fresh copy of pixel buffer (I first though that AVPlayer at some point rewrites pixel buffers already returned by copyPixelBufferForItemTime but that is not the case). At least these worked on my specific system, but there is nothing for sure if there is UB, someone may experience it even worse with a 30 fps version.

stuartmorgan-g · 2024-09-26T19:31:21Z

I wonder why no one reported this.

I think they have, per my edit to this comment above. Unless that's not the same behavior you're describing?

And there are other things which may seemingly help little with it like retaining pixel buffer by frame updater and creating fresh copy of pixel buffer

If we have lifetime issues within the engine pipeline itself, that's definitely something we should fix in the engine rather than try to hack around at the video_player level.

misos1 · 2024-09-26T19:38:59Z

I think they have, per my edit to this comment above. Unless that's not the same behavior you're describing?

Maybe partially, as I wrote, if copyPixelBuffer returns for some frame NULL then the engine shows a transparent image meaning it will flicker into background. I did not experience this on ios although it seems they share the same code so UB should be also on ios.

misos1 · 2024-09-29T19:07:39Z

Would it be doable to add new capability to the flutter engine and depend on it in this package or it needs to also work with the engine prior to such change?

stuartmorgan-g · 2024-09-30T16:54:00Z

Whether we try to work around engine bugs at the plugin level is something we decide on a case-by-cases basis. If the macOS engine texture pipeline is buggy, I would not attempt to work around that at the plugin level without a very compelling reason to do so.

misos1 · 2024-09-30T19:03:32Z

No I do not mean this, rather for the engine being able to pull pixel buffers rather than needing textureFrameAvailable. Something like a flag when the engine will call copyPixelBuffer for every frame, if it returns NULL it will do nothing, if it returns pixel buffer it will update internal buffers.

stuartmorgan-g · 2024-09-30T19:11:42Z

I'm not really sure what you mean by "for every frame", but if you want to propose a new engine API the place to start would be a design document. Details of exactly when video_player would adopt a new proposed API would be figured out much later in the process.

misos1 · 2024-09-30T19:20:35Z

By "for every frame" I mean that it would be called as if _textureFrameAvailable was always true. _textureFrameAvailable is set to true by calling textureFrameAvailable and set to false soon after is copyPixelBuffer called.

misos1 · 2024-10-10T19:36:45Z

Regarding #7466 (comment), it seems that macOS actually uses some different code path than ios but here it is the same, cvMetalTexture is released right after CVMetalTextureCacheCreateTextureFromImage:

https://github.com/flutter/engine/blob/6f802b39ab0669eb6ba3272dff1d34e85febeb77/shell/platform/darwin/macos/framework/Source/FlutterExternalTexture.mm#L119-L135
https://github.com/flutter/engine/blob/6f802b39ab0669eb6ba3272dff1d34e85febeb77/shell/platform/darwin/macos/framework/Source/FlutterExternalTexture.mm#L77-L97

For both it seems like a change here was some 4 years ago so I am not sure why such similar things were done twice.

And this is eventually called from embedder_external_texture_metal.mm which handles things differently, it does not use flag like _textureFrameAvailable like is used in FlutterDarwinExternalTextureMetal.mm but instead it nullifies last stored image when is called textureFrameAvailable so if copyPixelBuffer returns NULL it will not show anything and will periodically call it until it returns non-NULL and this explains what I observed.

I am not sure if this is intended or not but FlutterTexture does not tell anything about whether copyPixelBuffer can or cannot return NULL or what should happen in such a case:

/**
 * Copy the contents of the texture into a `CVPixelBuffer`.
 *
 * The type of the pixel buffer is one of the following:
 * - `kCVPixelFormatType_32BGRA`
 * - `kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange`
 * - `kCVPixelFormatType_420YpCbCr8BiPlanarFullRange`
 */
- (CVPixelBufferRef _Nullable)copyPixelBuffer;

So I will assume that returning NULL from copyPixelBuffer after previously returning non-NULL is undefined (it is practically unavoidable to return NULL at least once at start because the engine will call it until it returns non-NULL even without calling textureFrameAvailable). Then both old and current implementation are not entirely correct because they can return NULL from copyPixelBuffer because copyPixelBufferForItemTime can return NULL anytime so player needs to remember last returned pixel buffer and if it does not have any new then return this remembered buffer again until there is some new.

hellohuanlin · 2024-11-08T18:26:00Z

packages/video_player/video_player_avfoundation/darwin/RunnerTests/VideoPlayerTests.m

@@ -125,6 +125,7 @@ @interface StubFVPDisplayLinkFactory : NSObject <FVPDisplayLinkFactory>

 /** This display link to return. */
 @property(nonatomic, strong) FVPDisplayLink *displayLink;
+@property(nonatomic) void (^fireDisplayLink)(void);


nit: (nonatomic, copy)

hellohuanlin · 2024-11-08T18:29:15Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

-@property(nonatomic, assign) CMTime lastKnownAvailableTime;
+// The display link that drives frameUpdater.
+@property(nonatomic) FVPDisplayLink *displayLink;
+// The time interval between screen refresh updates.


Nit: Can you call it something like frameDuration or frameDelta or displayLinkDuration

hellohuanlin · 2024-11-08T18:46:27Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

@@ -543,16 +562,25 @@ - (void)setPlaybackSpeed:(double)speed {
 }

 - (CVPixelBufferRef)copyPixelBuffer {
+  // Ensure video sampling at regular intervals. This function is not called at exact time intervals
+  // so CACurrentMediaTime returns irregular timestamps which causes missed video frames. The range
+  // outside of which targetTime is reset should be narrow enough to make possible lag as small as


nit: make lags (due to skipping frames?) as less frequent as possible.

"Lag" here means that targetTime is lagging behind current time or conversely.

hellohuanlin · 2024-11-08T18:49:58Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

+  // so CACurrentMediaTime returns irregular timestamps which causes missed video frames. The range
+  // outside of which targetTime is reset should be narrow enough to make possible lag as small as
+  // possible and at the same time wide enough to avoid too frequent resets which would lead to
+  // irregular sampling. Ideally there would be a targetTimestamp of display link used by flutter


Can you add more explanation on how it leads to irregular sampling?

Can we do an experiment to always reset the time (remove the if check below) and see how it performs on a sample video?

Can you add more explanation on how it leads to irregular sampling?

Because each "reset" (meaning if is true) changes targetTime to a different value than it had. Adding the same number (self.frameUpdater.duration) results in regular (enough) time intervals in targetTime but suddenly changing it to value "outside" of that breaks that regularity. In the worst case when it changes always it is like directly using CACurrentMediaTime.

Can we do an experiment to always reset the time (remove the if check below) and see how it performs on a sample video?

Yes I did that of course. I also calculated standard deviation and tried to compute the probability of frame drop, result was about 10x higher than I observed but I probably did something wrong in this calculation (it does not really matter). And also distribution here is not normal so it may not match calculations assuming normal distribution. Here black is the distribution of differences between results of CACurrentMediaTime in consecutive copyPixelBuffer calls (on my ios device) and green is normal distribution (with the same standard deviation). Horizontal axis goes from around 8 to 24 ms with 1/60 s in middle and vertical axis is relative number of occurrences (this graph shows that there should be less drops than with normal distribution so I am satisfied enough with it as explanation why my calculated number was higher):

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

cbracken · 2024-11-20T22:19:16Z

@misos1 when you get a chance, please review @stuartmorgan's latest round of comments. Thanks for your contribution.

stuartmorgan-g · 2024-11-25T14:40:37Z

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m

+  // some other alternative, instead of on demand by calling textureFrameAvailable.
+  if (self.displayLink.running) {
+    dispatch_async(dispatch_get_main_queue(), ^{
+      [self.frameUpdater.registry textureFrameAvailable:self.frameUpdater.textureId];


I missed this change in the last review. So this version just constantly calls textureFrameAvailable:, as fast as possible, unconditionally, while the video is playing? Doesn't that just completely defeat the purpose of having the display link at all?

Display link is still usable for obtaining actual duration and for starting after pause. I also tried to call it conditionally using hasNewPixelBufferForItemTime with targetTime + duration (another +duration into future) but it did not work well for below 60 fps videos, there is that race problem even when just some video frames depend on textureFrameAvailable from display link.

Display link is still usable for obtaining actual duration and for starting after pause.

Neither of those things require a regularly firing display link; one is just a getter, and the other is a one-time call. This version appears to have two sets of regular calls: the display link callback, on a set cadence that we determine based on the video, and then this, which is as fast as possible no matter what the refresh rate, frame rate, or elapsed time are.

I understand the goals of the previous iterations of this PR, but I don't understand unconditionally driving buffer copies as fast as possible no matter what, or constantly telling the engine that there are new frames available regardless of whether or not that's true.

on a set cadence that we determine based on the video

It is actually based on display refresh rate.

which is as fast as possible no matter what the refresh rate, frame rate, or elapsed time are

This too, it is not faster than display refresh rate, it is at display refresh rate.

It is actually based on display refresh rate.

Right, sorry; I was misremembering the logic used to set videoComposition.frameDuration logic as feeding into the display link.

This too, it is not faster than display refresh rate, it is at display refresh rate.

How is an unconditional and immediate call every time a frame is provided the same as the refresh rate?

I did not observe such high deviations. But much smaller deviations (than screen refresh period) are enough to give different frames by using CACurrentMediaTime vs something more regular. All calls to CACurrentMediaTime may return timestamp strictly between timestamps of current and next screen refresh but time points of frames in video are shifted randomly to this (and this shift may change in time). Here is example of irregular sampling (like with CACurrentMediaTime) where video frame 2 was shown twice while frame 3 was dropped in contrast with regular sampling (like with targetTime in this PR):

screen refresh number: -1---|---2---|---3---|---4---|--- irregular sampling: v v v v video frame number: |---1---|---2---|---3---|---4---| regular sampling: ^ ^ ^ ^

Thanks, that's extremely helpful! I understand the issue much more clearly now. It would be a great idea to put that ASCII diagram into the comment in the code.

I think the next step is to file an issue to explore with the engine team whether making the copyPixelBuffer/textureFrameAvailable more explicit about call patterns (e.g., saying that calls will not be more frequent than ~the screen refresh rate) is something the engine team is comfortable with, so we know whether or not we need to build limiting logic at the plugin level.

So from flutter/flutter#160520 (comment) "We could make the behavior more documented/explicit but we may still need to change things in the future." should I take that as yes or do I need to implement that frame limiting? What is the next step?

The "may still need to change things in the future" means we shouldn't rely on the current behavior, so we should do plugin-level frame limiting.

Oh ok, I understood that as "ok, but in future switch to platform views".

misos1 · 2025-02-02T20:20:53Z

I suppose I have to merge the latest upstream for these failing checks to pass? "warning - lib/src/common/package_looping_command.dart:343:30 - The receiver can't be null"

hellohuanlin

LGTM! The review comments are very helpful for me to understand!

jmagman · 2025-02-12T22:24:35Z

@misos1 the analyzer warnings look unrelated to your change (warnings are related to turning on the analyzer for another part of the code). Could you rebase/merge onto master, and resolve the merge conflicts?

stuartmorgan-g · 2025-03-25T17:25:22Z

In case this was waiting on me, the changes here LGTM. I was just waiting until the merge to pass CI has happened for final review+approval, since that merge will need a review.

jmagman · 2025-03-31T20:25:28Z

@misos1 the analyzer warnings look unrelated to your change (warnings are related to turning on the analyzer for another part of the code). Could you rebase/merge onto master, and resolve the merge conflicts?

Hi @misos1, friendly ping that this would still need a rebase to land (to get past the analyzer issues), and also re-update the CHANGELOG and pubspec. It's so close to landing! 🙂

stuartmorgan-g

LGTM. Sorry for the delay on the final approval! I didn't notice it was ready since I get a ton of notification emails that are just code pushes on PRs, and they usually aren't relevant.

Thanks again for all the iteration and deep investigation on this issue to get to something that is as robust as possible given the current engine constraints

…r/packages#7466)

flutter/packages@2fcc403...ac21f53 2025-04-20 [email protected] Roll Flutter from 3ed38e2 to cfb887c (17 revisions) (flutter/packages#9118) 2025-04-19 [email protected] [various] Scrubs pre-SDK-21 Android code (flutter/packages#9112) 2025-04-18 [email protected] Roll Flutter from ecabb1a to 3ed38e2 (23 revisions) (flutter/packages#9114) 2025-04-18 [email protected] [flutter_svg] feat: Expose the `colorMapper` property in `SvgPicture` (flutter/packages#9043) 2025-04-18 [email protected] [tool] Add initial file-based command skipping (flutter/packages#8928) 2025-04-18 [email protected] [pigeon] Convert test plugins to SPM (flutter/packages#9105) 2025-04-18 [email protected] [webview_flutter] Adds support to control overscrolling (flutter/packages#8451) 2025-04-17 [email protected] [in_app_purchase] add Storefront.countryCode() and AppStore.sync() (flutter/packages#8900) 2025-04-17 [email protected] [webview_flutter_wkwebview] Expose the allowsLinkPreview property in WKWebView for iOS (flutter/packages#5029) 2025-04-17 [email protected] [webview_flutter_android][webview_flutter_wkwebview] Adds platform implementations to set over-scroll mode (flutter/packages#9101) 2025-04-17 [email protected] [shared_preferences] Update AGP to 8.9.1 (flutter/packages#9106) 2025-04-17 [email protected] [pigeon] Adds Kotlin lint tests to example code and fix lints (flutter/packages#9034) 2025-04-17 [email protected] [video_player_avfoundation] enable more than 30 fps (flutter/packages#7466) 2025-04-17 [email protected] Roll Flutter from aef4718 to ecabb1a (25 revisions) (flutter/packages#9104) 2025-04-16 [email protected] [pigeon] Unify iOS and macOS test plugins (flutter/packages#9100) 2025-04-16 [email protected] Roll Flutter from db68c95 to aef4718 (7 revisions) (flutter/packages#9098) 2025-04-16 [email protected] [webview_flutter_platform_interface] Adds method to set overscroll mode (flutter/packages#9099) 2025-04-16 [email protected] Update `CODEOWNERS` (flutter/packages#8984) 2025-04-16 [email protected] [google_sign_is] Update iOS SDK to 8.0 (flutter/packages#9081) 2025-04-16 [email protected] [camera_avfoundation] Implementation swift migration (flutter/packages#8988) 2025-04-16 [email protected] [go_router] Adds `caseSensitive` to `GoRoute` (flutter/packages#8992) 2025-04-16 [email protected] Manual roll Flutter from 30e53b0 to db68c95 (98 revisions) (flutter/packages#9092) 2025-04-15 [email protected] [tool] Run config-only build for iOS/macOS native-test (flutter/packages#9080) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

flutter/packages@2fcc403...ac21f53 2025-04-20 [email protected] Roll Flutter from 409a8ac to cd51fa3 (17 revisions) (flutter/packages#9118) 2025-04-19 [email protected] [various] Scrubs pre-SDK-21 Android code (flutter/packages#9112) 2025-04-18 [email protected] Roll Flutter from d0741df to 409a8ac (23 revisions) (flutter/packages#9114) 2025-04-18 [email protected] [flutter_svg] feat: Expose the `colorMapper` property in `SvgPicture` (flutter/packages#9043) 2025-04-18 [email protected] [tool] Add initial file-based command skipping (flutter/packages#8928) 2025-04-18 [email protected] [pigeon] Convert test plugins to SPM (flutter/packages#9105) 2025-04-18 [email protected] [webview_flutter] Adds support to control overscrolling (flutter/packages#8451) 2025-04-17 [email protected] [in_app_purchase] add Storefront.countryCode() and AppStore.sync() (flutter/packages#8900) 2025-04-17 [email protected] [webview_flutter_wkwebview] Expose the allowsLinkPreview property in WKWebView for iOS (flutter/packages#5029) 2025-04-17 [email protected] [webview_flutter_android][webview_flutter_wkwebview] Adds platform implementations to set over-scroll mode (flutter/packages#9101) 2025-04-17 [email protected] [shared_preferences] Update AGP to 8.9.1 (flutter/packages#9106) 2025-04-17 [email protected] [pigeon] Adds Kotlin lint tests to example code and fix lints (flutter/packages#9034) 2025-04-17 [email protected] [video_player_avfoundation] enable more than 30 fps (flutter/packages#7466) 2025-04-17 [email protected] Roll Flutter from 9616f9c to d0741df (25 revisions) (flutter/packages#9104) 2025-04-16 [email protected] [pigeon] Unify iOS and macOS test plugins (flutter/packages#9100) 2025-04-16 [email protected] Roll Flutter from a7ce7ff to 9616f9c (7 revisions) (flutter/packages#9098) 2025-04-16 [email protected] [webview_flutter_platform_interface] Adds method to set overscroll mode (flutter/packages#9099) 2025-04-16 [email protected] Update `CODEOWNERS` (flutter/packages#8984) 2025-04-16 [email protected] [google_sign_is] Update iOS SDK to 8.0 (flutter/packages#9081) 2025-04-16 [email protected] [camera_avfoundation] Implementation swift migration (flutter/packages#8988) 2025-04-16 [email protected] [go_router] Adds `caseSensitive` to `GoRoute` (flutter/packages#8992) 2025-04-16 [email protected] Manual roll Flutter from f2d54fd to a7ce7ff (98 revisions) (flutter/packages#9092) 2025-04-15 [email protected] [tool] Run config-only build for iOS/macOS native-test (flutter/packages#9080) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

In the player there was hardcoded 30 fps when setting up video composition. Now it uses timing from source track and also fallback `minFrameDuration` as seems `frameDuration` must be always set to something and it takes over in some situations as is mentioned in documentation about `sourceTrackIDForFrameTiming`. Also video composition setup is skipped when it is not needed when `preferredTransform` is identity. Function `updatePlayingState` is called often right after `setupEventSinkIfReadyToPlay` but seems it was forgotten in `onListenWithArguments` and also it cannot be called in that way because `setupEventSinkIfReadyToPlay` may finish asynchronously when called again from line `[self performSelector:_cmd onThread:NSThread.mainThread withObject:self waitUntilDone:NO]` so now is `updatePlayingState` called right after `_isInitialized = YES` which is what it needs to even do something. There was one more obstacle for playing 60 fps videos on 60 hz screen. Seems there are at least two "display links" at play when playing video, one calls function `displayLinkFired` and other one from flutter engine calls `copyPixelBuffer` but only when `textureFrameAvailable` was called previously. But the order in which those two are called is undefined so 16 ms after `displayLinkFired` may be called `copyPixelBuffer` and right after that `displayLinkFired` and so on. But `copyPixelBuffer` steals the newest pixel buffer from video player output and in `displayLinkFired` `hasNewPixelBufferForItemTime` will not report another pixel buffer for time close to that. Then the next frame is not called `copyPixelBuffer` because `textureFrameAvailable` was not called and in this way it skips every second frame so it plays video at 30 fps. There was also a synchronization problem with `lastKnownAvailableTime`. Now pixel buffers are produced and reported just on a single place in `displayLinkFired` and received with correct synchronization in `copyPixelBuffer`. Ideally there would be just a single "display link" from flutter engine if it supported also pulling frames instead of only pushing (which is good enough for a camera where the system is pushing frames to us, but from player video output is needed to pull frames). Calling `textureFrameAvailable` every frame could accomplish that but it looks like this line in flutter engine is calling even when `copyPixelBuffer` returns NULL and it may be expensive (although there is no need to call it in such case): ``` sk_sp<flutter::DlImage> image = [self wrapExternalPixelBuffer:_lastPixelBuffer context:context]; ``` Seems there is some bug with the video player using this flutter engine on macos. Looks like the video is playing normally but then it starts "tearing", it looks like it is displaying frames normally but once in a while it shows some frame from the past like some previously cached frame. This is happening on the main branch but rendering on 60 fps exaggerates it (it is not caused by this PR).

split video_player changes from more_fps

0bade59

misos1 requested a review from hellohuanlin as a code owner August 21, 2024 16:18

github-actions bot added platform-ios platform-macos p: video_player labels Aug 21, 2024

stuartmorgan-g requested changes Aug 23, 2024

View reviewed changes

stuartmorgan-g mentioned this pull request Aug 29, 2024

[video_player_avfoundation] Video FPS is SLOW from version 2.5.7+ flutter/flutter#150891

Closed

stuartmorgan-g reviewed Sep 17, 2024

View reviewed changes

...on/darwin/video_player_avfoundation/Sources/video_player_avfoundation/FVPVideoPlayerPlugin.m Outdated Show resolved Hide resolved

rework few details

ad0ba7d

misos1 added 4 commits October 11, 2024 18:06

avoid race between displayLinkFired and copyPixelBuffer

02a1241

update warning message and format

e0f85ce

Merge branch 'main' into more_fps_player

014df97

fix bad merge and format

89b0517

stuartmorgan-g added the triage-ios Should be looked at in iOS triage label Nov 8, 2024

hellohuanlin reviewed Nov 8, 2024

View reviewed changes

This was referenced Nov 18, 2024

FlutterTexture should provide timestamp and duration or targetTimestamp flutter/flutter#159087

Open

Better implementation of FlutterTexture flutter/flutter#159162

Open

stuartmorgan-g reviewed Nov 25, 2024

View reviewed changes

misos1 mentioned this pull request Dec 18, 2024

Question for the engine team about making copyPixelBuffer/textureFrameAvailable more explicit about call patterns flutter/flutter#160520

Open

misos1 added 3 commits February 2, 2025 20:07

add logic to turn off self refresh

016f799

fix format

8fb95cb

fix format

836733c

misos1 requested review from hellohuanlin and stuartmorgan-g February 2, 2025 20:21

hellohuanlin approved these changes Feb 3, 2025

View reviewed changes

change order of consts

e304b16

cbenhagen mentioned this pull request Mar 25, 2025

[video_player_avfoundation] Flickering on macOS flutter/flutter#135999

Open

2 tasks

Merge branch 'main' into more_fps_player

af56816

stuartmorgan-g approved these changes Apr 17, 2025

View reviewed changes

stuartmorgan-g added the autosubmit Merge PR when tree becomes green via auto submit App label Apr 17, 2025

auto-submit bot merged commit 04de46e into flutter:main Apr 17, 2025
82 checks passed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Apr 21, 2025

04de46e0b [video_player_avfoundation] enable more than 30 fps (flutte…

50800b9

…r/packages#7466)

engine-flutter-autoroll mentioned this pull request Apr 21, 2025

Roll Packages from 2fcc4032dd8b to ac21f531a0cc (23 revisions) flutter/flutter#167479

Merged

stuartmorgan-g mentioned this pull request Jun 4, 2025

[video_player] Hardcoded to 30 FPS on iOS flutter/flutter#78210

Closed

[video_player_avfoundation] enable more than 30 fps #7466

[video_player_avfoundation] enable more than 30 fps #7466

Uh oh!

Conversation

misos1 commented Aug 21, 2024

Pre-launch Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

misos1 Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuartmorgan-g commented Sep 17, 2024

Uh oh!

misos1 commented Sep 17, 2024

Uh oh!

stuartmorgan-g commented Sep 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

misos1 commented Sep 26, 2024

Uh oh!

stuartmorgan-g commented Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stuartmorgan-g commented Sep 26, 2024

Uh oh!

stuartmorgan-g commented Sep 26, 2024

Uh oh!

misos1 commented Sep 26, 2024

Uh oh!

stuartmorgan-g commented Sep 26, 2024

Uh oh!

misos1 commented Sep 26, 2024

Uh oh!

misos1 commented Sep 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stuartmorgan-g commented Sep 30, 2024

Uh oh!

misos1 commented Sep 30, 2024

Uh oh!

stuartmorgan-g commented Sep 30, 2024

Uh oh!

misos1 commented Sep 30, 2024

Uh oh!

misos1 commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

misos1 Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

misos1 Aug 23, 2024 •

edited

Loading

stuartmorgan-g commented Sep 26, 2024 •

edited

Loading

misos1 commented Sep 29, 2024 •

edited

Loading

misos1 commented Oct 10, 2024 •

edited

Loading

misos1 Feb 2, 2025 •

edited

Loading

misos1 Nov 18, 2024 •

edited

Loading

stuartmorgan-g Nov 25, 2024 •

edited

Loading

misos1 Dec 5, 2024 •

edited

Loading