Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Run check task on CI #3808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Sep 30, 2024
Merged

Run check task on CI #3808

merged 5 commits into from
Sep 30, 2024

Conversation

@whyoleg
Copy link
Collaborator

@whyoleg whyoleg commented Sep 18, 2024

fixes #3805

Introduce a Gradle property to skip integration tests, which could be set on CI/locally to be able to run check task without running all integration tests.

If we will enable running integration tests unconditionally we will increase GA tests time from 10-15 minutes to more than an hour (example).
Note: all tests are red in the above example because most of them failed with OOM or "not enough space" when downloading K/N toolchain for different version of Kotlin.
So enabling integration tests unconditionally on GA is not an option also for reasons other than "slow tests".

Note: running check on CI exposed issue with Windows and DGPv2 function tests. This should be fixed before the merging of this PR.
Additionally after the PR is merged we could change TC setup for Dokka Tests to also use check with integration_test.skip=true flag. Or even drop it, as it doesn't cover other OS like we have on GA.

@whyoleg whyoleg self-assigned this Sep 18, 2024
@adam-enko adam-enko added the infrastructure Everything related to builds tools, CI configurations and project tooling label Sep 18, 2024
@whyoleg
Copy link
Collaborator Author

whyoleg commented Sep 18, 2024

Hm, running check causes tests with K1 runs twice, because we have both test and testDescriptors do run tests with K1 :(

And looks like there is no good way to tell both Gradle and IDEA that we want to have test to just run both testDescriptors and testSymbols but not run anything by itself...

@Tapchicoma
Copy link
Contributor

@whyoleg maybe you should structure your tests using Gradle test-suite plugin and then via Gradle properties just disable some suites

@whyoleg whyoleg mentioned this pull request Sep 20, 2024
Merged
adam-enko
adam-enko approved these changes Sep 23, 2024
View reviewed changes
Copy link
Member

@adam-enko adam-enko left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this approach is fine, although I have some suggestions:

  1. Instead of disabling the entire task, what about adding JUnit tags to the integration test and excluding them via a filter? This would help with test reporting, as the tests would be reported as 'skipped' in places like Build Scan and TeamCity.

  2. It's strange that test and testDescriptors are identical tests.

    dokka/build-logic/src/main/kotlin/dokkabuild.test-k2.gradle.kts

    Lines 62 to 63 in 7733051

    // Create a new target for _only_ running test compatible with descriptor-analysis (K1).
    // Note: this target is the same as the regular test target, it is only created to provide a more descriptive name.

    1. Instead of another custom workaround, maybe just remove testDescriptors?

    2. Alternatively, try removing the testDescriptors test suite target, creating a lifecycle task named testDescriptors that depends on test, and add idea.internal.test=true to trick IntelliJ into thinking testDescriptors is a test task.

  3. Rather than dynamically enabling/disabling tasks, I think it'd be more clear to explicitly list the tasks to be executed in GitHub CI.

    For example:

          - name: Run tests
        run: >
          ./gradlew test testSymbols --stacktrace --continue

@whyoleg
Copy link
Collaborator Author

whyoleg commented Sep 24, 2024

This would help with test reporting, as the tests would be reported as 'skipped' in places like Build Scan and TeamCity.

just curious: this is needed to understand if the test is skipped or just not run?
I can add this tag, but adding it to every test in integration tests feels not right :)

Instead of another custom workaround, maybe just remove testDescriptors?

I've decided to just skip test task running and make it aggregate. In the end, looks like even now running test from IDEA works bad: as no filters are applied to testSymbols and so it runs all tests even if only one test selected.

Rather than dynamically enabling/disabling tasks, I think it'd be more clear to explicitly list the tasks to be executed in GitHub CI.

Let's run check on CI so that if some new tests (or other checks) are added (like with DGPv2 testFunctional), they are run by default. If we will make test dependsOn testFunctional, we will have the same issue as with current test task in other subprojects in IDEA I described above.
Also, we will not solve initial issue in this case.

@adam-enko
Copy link
Member

This would help with test reporting, as the tests would be reported as 'skipped' in places like Build Scan and TeamCity.

just curious: this is needed to understand if the test is skipped or just not run? I can add this tag, but adding it to every test in integration tests feels not right :)

Yeah, fair enough. I think adding a @SlowTest JUnit tag would be useful, but maybe later. I was more thinking-out-loud than actually suggesting.

I'm working on fixing the failing Windows tests in #3811

whyoleg
whyoleg commented Sep 27, 2024
View reviewed changes
dokka-integration-tests/gradle.properties

# !!! ATTENTION: do not commit this !!!
# Uncomment next line to skip running Dokka integration tests
#org.jetbrains.dokka.integration_test.skip=true
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: @atyrin / @vmishenev after merging this PR you can uncomment this line locally to not run integration tests when running build/check tasks

@whyoleg
Copy link
Collaborator Author

whyoleg commented Sep 27, 2024

@adam-enko There is still one more test failure on Windows. Do you think may be it will be better to merge this PR and then fix the test, so that the fix will be properly tested on CI?

@adam-enko
Copy link
Member

@adam-enko There is still one more test failure on Windows. Do you think may be it will be better to merge this PR and then fix the test, so that the fix will be properly tested on CI?

It's a strange failure, I'm really not sure what's wrong.

Let's suppress the test in this PR and fix it later.

@whyoleg whyoleg merged commit c4a33d3 into master Sep 30, 2024
14 checks passed
@whyoleg whyoleg deleted the tests-ci branch September 30, 2024 12:23
@yuriioliinyk4 yuriioliinyk4 mentioned this pull request Jul 10, 2025
Closed
@strangesource strangesource mentioned this pull request Sep 17, 2025
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adam-enko adam-enko adam-enko approved these changes

@Tapchicoma Tapchicoma Awaiting requested review from Tapchicoma

@vmishenev vmishenev Awaiting requested review from vmishenev

Assignees

@whyoleg whyoleg

Labels

infrastructure Everything related to builds tools, CI configurations and project tooling

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

CI does not run all DGP tests

4 participants

@whyoleg @Tapchicoma @adam-enko