Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.webauthn.WebAuthnIdlessTest#testWebAuthnIDLessLogin

Keycloak CI - WebAuthn IT (chrome)

java.lang.AssertionError: Event expected
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertNotNull(Assert.java:713)
	at org.keycloak.testsuite.AssertEvents.poll(AssertEvents.java:87)
...

Report flaky test

org.keycloak.testsuite.forms.LoginTest#loginRememberMeExpiredMaxLifespan

Keycloak CI - Forms IT (chrome)

java.lang.AssertionError: Expected LoginPage but was AUTH_RESPONSE (https://localhost:8543/auth/realms/master/app/auth?state=b921ca98-1c70-4d15-aa85-f920c8027d65&session_state=bbb1ae0b-ca25-4241-a6ab-bc3f5ce8f2ea&iss=https%3A%2F%2Flocalhost%3A8543%2Fauth%2Frealms%2Ftest&code=2135d6fa-02dc-499d-950a-e12f7cf6887b.bbb1ae0b-ca25-4241-a6ab-bc3f5ce8f2ea.e296fc35-904f-475a-aba4-5150e6324064)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.keycloak.testsuite.pages.AbstractPage.assertCurrent(AbstractPage.java:47)
	at org.keycloak.testsuite.pages.LoginPage.open(LoginPage.java:253)
...

Report flaky test

org.keycloak.testsuite.forms.LoginTest#loginMissingUsername

Keycloak CI - Forms IT (chrome)

java.lang.AssertionError: expected:<Invalid username or password.> but was:<null>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
...

Report flaky test

org.keycloak.testsuite.forms.ResetPasswordTest#resetPasswordBeforeUserIsDisabled

Keycloak CI - Forms IT (chrome)

java.lang.AssertionError: Expected LoginPasswordResetPage but was Sign in to test (https://localhost:8543/auth/realms/test/protocol/openid-connect/auth?response_type=code&client_id=test-app&redirect_uri=https%3A%2F%2Flocalhost%3A8543%2Fauth%2Frealms%2Fmaster%2Fapp%2Fauth&state=a5cd0b6c-9986-4765-9b3e-3fe69e57689b&scope=openid)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.keycloak.testsuite.pages.AbstractPage.assertCurrent(AbstractPage.java:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
...

Report flaky test

@shawkins Thank you for creating the follow-up for the Operator.

While I'd love to take this aggressive approach and just remove the no-longer-needed fields (as it's the last opportunity to do that for a long time as we can't do breaking changes in minor versions and 27 is more than a year away by the new release schema), but I'm afraid we can't do it since CRDs are a cluster-wide resource. That would mean e.g. Keycloak 25 and 26 could not exist in the same cluster, AFAIK.

That would mean e.g. Keycloak 25 and 26 could not exist in the same cluster, AFAIK.

I'd say it's more nuanced than that. They can't upgrade until all Keycloaks no longer use the deprecated field. The case you might be worried about is that they won't be able to re-introduce those deprecated fields after the upgrade. I see this as somewhat similar to the problem we introduce for older versions once new CRD fields are available, but that aren't supported.

They can't upgrade until all Keycloaks no longer use the deprecated field.

This is the main issue I see. The typical scenario (for RHBK but upstream possibly as well) would be that user is using 24 in prod but want to try 26 in a different namespace.

@shawkins I'd like to resurrect the discussion here.

If we remove the fields, users can't install KC 24/25 alongside 26 in the same cluster, correct? If that's the case, I'm afraid we can't do it. :(

If we remove the fields, users can't install KC 24/25 alongside 26 in the same cluster, correct? If that's the case, I'm afraid we can't do it. :(

It's probably important to deliniate between 24 and 25 wrt hostname. In 25 v1 was deprecated, so running 25 and 26 with the additional requirement that the deprecated fields are no longer used does not seem like a problem.

However running 24 and 26 together is problematic because these options aren't deprecated in v24. This brings up a larger issue - when we moved up the removal of hostname v1 from the server to v26 from v27 that makes sense from a community perspective. However does seem problematic from a product perspective because the options had yet to have a product release where they were deprecated.

Based upon our offline messages, we'll stick with the "do nothing" option for KC 26. We can revisit this once KC 24 is not expected to be in production use.