@dnfield I managed to wire it up with Integration Tests here. But when I tried reusing the collected SkSL, (following the repro from #76180), it didn't seem to help eliminate jank, while the manually collected ones through flutter run --cache-sksl did for me locally. Do you know if it's possible that with the IntegrationTestWidgetsFlutterBinding bindings, the collected SkSL is somehow different from the ones expected in a normal app?

If you compare the JSON output are they different?

Yes, they are different (uploaded here). I've checked the usual things - running in profile mode on the same iOS physical device, software rendering is disabled, not sure if you have any ideas.

Could it be the extra repaint boundary we added?

I found this draft PR looking for something similar. Sorry to to bother here, but I indeed stumbled with disappointment that compiled shaders extracted from integration test don't reduce jank at all. Why is this?

Sorry, I haven't had time to follow up and investigate further. As Dan mentioned, it could be due to the extra repaint boundary that we use to wrap all widgets, or some subtle difference in how the IntegrationTestWidgetsFlutterBinding differs from the default WidgetsFlutterBinding.

This is a huge bummer. In the documentation https://flutter.dev/docs/perf/rendering/shader it talks how integration tests could make the task of compiling shader cache easier and it does not specifies that the new integration_test will run useless the exported shaders. I have a bunch of tests written the new way that you are telling me that I can't use to compile shaders?

The jank issues have been (since I started with flutter) the worst part of all the delightful experience flutter is. I accepted and realised that compiling shaders is a temporary solution that solves a problem with good results. But been not able to make this automatically brings us way back again to not solving the jank issues at all. Manual shader compilation by playing with the app is not acceptable for large scale applications.

Sorry to repeat this but is a huge disappointment.

I strongly suspect it's this code here that's causing trouble:

@jiahaog - are you set up to easily verify if deleting that code produces the expected shaders or not?

Thanks Dan for the pointer, I'll try that out and update here

I made a test my self commenting the code you @dnfield suggested and it does not seem to fix the issue of janks on first run. I commented the lines in the file in the dart package and then run the integration test. I don't know if that is enough or I have to do an extra step.

Oh I have to say that the compiled shaders JSON file size was reduced on this run with the commented lines. So it seems that something is doing but not enough.

I'm suspicious of that code because it changes what gets drawn and thus might change shaders. We have to see if there are other places too.

It definitively change shaders because size was reduced. My question is, do you consider that this issue will get fixed and integration_test package will be a valid way to generate shaders? Or is safer if I re write them the "old way"? I want to know your opinion on this because now I have some feature tests but if in a couple of months the app grows it would be kind of annoying to re write them the old way.

Thanks!

This is a prolem we want to fix, but I don't have an estimate on when it will be fixed right now.

What do you mean the size is reduced? I thought integration_test was using the native device resolution.

Okey so I will continue doing tests this way and making cache shaders by playing around with the app.

By reduced size I mean that the json file first was around 217kb and after commenting the code you suggested the json was 191kb.

Ah! yes, a change in size would be expected since you're not compiling the same shaders anymore.

It would be helpful to compare the files against a regular flutter run --profile, or a flutter_driver based test.

Also, I'm assuming you're running in profile mode on a real device.

Yes, real device and profile.

flutter drive --profile --cache-sksl --write-sksl-on-exit flutter_01.sksl.json --driver=test/integration_test_driver.dart --target=test/${TARGET}

I'll try to get some time to compare the .json files.

I have one doubt that got me thinking. Running multiple group tests in a single file, initialising apps main method in each group could lead to wrong shaders?

For example:

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  group('smoke test', () {
    testWidgets('test 1', (WidgetTester tester) async {
        app.main();
        await tester.pumpAndSettle();
        await testOne(tester);
    });

    testWidgets('test 2', (WidgetTester tester) async {
        app.main();
        await tester.pumpAndSettle();
        await testTwo(tester);
    });
  });

  group('smoke test logged', () {
    testWidgets('test 3', (WidgetTester tester) async {
        app.main();
        await tester.pumpAndSettle();
        await testThree(tester);
    });
  });
}

I don't think so, but it would be worth isolating

The changes mentioned in #85118 (comment) did not help. I was able to get the collected shaders with this to work with a few modifications to the integration test:

Approach 1 - setting the frame policy, using timeouts

https://github.com/jiahaog/hello_flutter_integration_test/blob/2a423a4ea157cec90d395ab27ef7e0663977d283/integration_test/collect_sksl_test.dart#L9-L34

Approach 2 - Large time dilation value

https://github.com/jiahaog/hello_flutter_integration_test/blob/20d77e8842aa055345ef3a3151b8dda9ecb64469/integration_test/collect_sksl_test.dart#L8-L26

The value of timeDilation required for pumpAndSettle is fairly large. I suspect that using pumpAndSettle is potentially expensive, which relates to #84994 where frames are missed, preventing the shaders from being cached properly, so we need to set the time dilation to a large value.

I'm less favourable to the first approach since that requires the use of timeouts, and changing the frame policy may not be compatible with existing tests, which does not make them portable for both regular testing and collecting of SkSL.

@dnfield any thoughts about the above? For approach 2 we could potentially send a signal from the Tool to the test to increase the time dilation, though it may not be easy to choose a good value. Or is there a way to disable frame skipping entirely (as mentioned in #84994)? Then existing tests can be reused without code changes.

Workspace for reference: https://github.com/jiahaog/hello_flutter_integration_test/tree/cache-sksl

The changes mentioned in #85118 (comment) did not help. I was able to get the collected shaders with this to work with a few modifications to the integration test:

Approach 1 - setting the frame policy, using timeouts

https://github.com/jiahaog/hello_flutter_integration_test/blob/2a423a4ea157cec90d395ab27ef7e0663977d283/integration_test/collect_sksl_test.dart#L9-L34

Approach 2 - Large time dilation value

https://github.com/jiahaog/hello_flutter_integration_test/blob/20d77e8842aa055345ef3a3151b8dda9ecb64469/integration_test/collect_sksl_test.dart#L8-L26

The value of timeDilation required for pumpAndSettle is fairly large. I suspect that using pumpAndSettle is potentially expensive, which relates to #84994 where frames are missed, preventing the shaders from being cached properly, so we need to set the time dilation to a large value.

---

I'm less favourable to the first approach since that requires the use of timeouts, and changing the frame policy may not be compatible with existing tests, which does not make them portable for both regular testing and collecting of SkSL.

@dnfield any thoughts about the above? For approach 2 we could potentially send a signal from the Tool to the test to increase the time dilation, though it may not be easy to choose a good value. Then existing tests can be reused without code changes.

Workspace for reference: https://github.com/jiahaog/hello_flutter_integration_test/tree/cache-sksl

This means that tests will be longer with a time dilation so big? In my test I had to use several pumpAndSettle consecutively as it was the only way to advance some frames. My whole test skips frames abruptly but I did not think that this would affect the generated shaders.

The time delation for some complex animations may be needed to be longer?

Yes, you may be able to find some success by playing around with the time dilation value.

I think this makes sense - if your test skips frames (perhaps because of shader compilation), it will likely not have shaders.

I agree that time dilation is probably the way to go. I think as long as it's set low enough it should just work right? It sounds like what we basically want is something where the framework gets to render every possible frame of every animation. Time dilation is more or less a timeout in this sense, since we're saying we'll slow things down and that should be enough for the rasterizer to finish its work on every frame. That should work in most cases, but it would be even better if we just had some kind of new frame policy like neverSkip or something.

I'm not sure if frame skipping happens within the framework or engine, would you have a pointer to the code that controls that?

You'd have to do some tracing through the scheduler binding - I think if you look at the way the time dilation logic is handled, it may be possible to instead just adjust it so that it appears that we've always passed a single frame's worth of time (e.g. 16ms).

Thanks, I'll try that out 🙂

flutter test runs the tests in an 800x600 window with dpr=3x, which is extremely different from most phones. Could that be an issue?

(Should we close this PR? It hasn't been touched in a while.)

integration_test overrides those settings to match the size/dpr of the host device.

I think we should close this. The guidance around using SkSL is about to change anyway.