2h is probably too much but a huge increase in execution|queue time is expected after #113539 where we are explicitly cleaning the local xcode cache

Setting as P1 seems like mac capacity is at 100% utilization from 6:00AM PST to 5:00PM PST.

Next step we need to identify if this is normal usage or something strange is happening.

From the data it seems like we've always had peeks of 30M 100% utilization at ~9:00AM PST but we started to have 3+ peeks per day on 10/17/2022. Things got really bad on 10/24/2022 where we started with 100% utilization from 9:00 to 5:00

The local cache cleanup happened on 10/19 (#113729), the queue time (90th%) was <1min for 10/19-10/24.
The queue time started increasing from 10/25
10/25-10/26: 6.06 min
10/26-10/27: 60.5 min
10/17-10/28: 58.68 min
10/28-11/04: 85.27 min

One correlated change per the timing is the quota increase (cl/483782210, which was landed 0n 10/25). @godofredoc Are you comfortable with a revert?

One correlated change per the timing is the quota increase (cl/483782210). @godofredoc Are you comfortable with a revert?
That change is supposed to help. Can we take a look if the repo sync time has increased? if cold checkouts are >7m then please do not revert.

Ths is

0/17-10/28: 58.68 min

The local cache cleanup happened on 10/19 (#113729), the queue time (90th%) was <1min for 10/19-10/24.
The queue time started increasing from 10/25
10/25-10/26: 6.06 min
10/26-10/27: 60.5 min
10/17-10/28: 58.68 min
10/28-11/04: 85.27 min

Can you please post daily data for: 10/25, 10/27, 11/03?

The engine source checkout was not improved with this quota increase CL. It has been taking 5-7 mins before and after. See
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Host%20clang-tidy?cursor=id%3E8799256324311980481&limit=200
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Unopt?cursor=id%3E8799256222795131969&limit=200
https://ci.chromium.org/p/flutter/builders/prod/Mac%20Web%20Engine?cursor=id%3E8799256222304648017&limit=200
https://ci.chromium.org/p/flutter/builders/prod/Mac%20iOS%20clang-tidy?cursor=id%3E8799263756679076865&limit=200

That is expected, cl/483782210 fixes peek quota whith checkouts taking 40mins in try.

5-7 mins is expected in cold checkouts if we are getting consistent 5-7 mins checkouts then that is a bug that need to be fixed.

Can you please post daily data for: 10/25, 10/27, 11/03?

10/25: 6 min
10/27: 58 min
11/03: 100 min

Are these 90th percentiles? can we also get the 50th?

Here are the comparison between 90th and 50th, plus the build number:
date. 90th 50th No. of builds
10/24. 19 6 1605
10/25. 6 0 1192
10/26. 60 15 1983
10/27. 58 21 2046
11/03. 100 7 2493

The queue time correlates with the No. of builds

When did you enable the Mac engine v2 builders?

Nice, in this case the 50th %tile is giving us a better signal.

• The 10/26 increase in the 90th %tile could be caused by Move 2 mac ios engine v2 builders to prod. engine#37051
• The 11/03 increase in the 90th %tile could be caused by Move the two remaining engine v2 builds to prod. engine#37274

Higher queue times are expected in engine_v2 under the current conditions because they require multiple bots to complete.

The decrease in the 50th percentile which I believe most of the legacy builds will fit into, except for engine_v2, web engine and fuchsia builders was caused by the following PRs:

• Build ios with goma. engine#37246
• https://flutter.googlesource.com/infra/+/f36c28ca89027e3d80d18e68c6e8c5e82fea4e7e
• cl/483782210

Probably we are making thing a bit worse for a short period of time ~4h while we re-allocate mac machines to different pools.

The plan to address the queue time is as follows:

• Fix checkout caches in mac builders
  • Do not clean source code caches by default in Web Engine builds
  • Do not clean source code caches by default engine_arm builds
• Ensure https://flutter.googlesource.com/recipes/+/e02f2b41ba98db6d230fee81689e4a6e89526453 is CP to stable and beta versions of recipes
• Remove the flag to always clean the xcode caches. The caches of all the machines should be clean by now and some - logic has been added to cleanup the cache only when using simulator runtimes.
• Use linux machines for orchestrators in engine_v2
• Move some machines from staging to try now that engine_v2 builders have moved to prod

@khyati82 this is the impact we were expecting during the transition of legacy to engine_v2.

CPs of https://flutter.googlesource.com/recipes/+/e02f2b41ba98db6d230fee81689e4a6e89526453 to stable and beta:

• https://flutter-review.googlesource.com/c/recipes/+/35601
• https://flutter-review.googlesource.com/c/recipes/+/35600

https://chrome-internal-review.googlesource.com/c/infradata/config/+/5074772 to move half (8) bots from staging to try.

Use linux machines for orchestrators in engine_v2

Do you mean to use linux VMs to host Mac engine_v2 builders? @godofredoc

Use linux machines for orchestrators in engine_v2

Do you mean to use linux VMs to host Mac engine_v2 builders? @godofredoc

The current implementation will make this look weird. Basically I meant to change https://github.com/flutter/engine/blob/main/.ci.yaml#L327 to use a linux machine rather than mac. Which I believe requires to change Mac mac_android_aot_engine to Linux mac_android_aot_engine

CL to default to not cleaning source code caches in Web Engine builds: https://flutter-review.googlesource.com/c/recipes/+/35661

I'm not sure how you are distributing mac resources but fwiw https://flutter-review.git.corp.google.com/c/recipes/+/35760 should lower mac clang-tidy bot execution 5-10m. The number of linted files is going from 800->600. Hopefully that helps a bit.

Queue time drops to 20mins for past two weeks. Moving to TD for further optimization track.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.