Codestin Search App

miquelsi · 2024-07-05T10:21:29Z

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2024-07-05T10:55:16Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.federation.ldap.LDAPReadOnlyTest#testReadOnlyUserGetsPermanentlyLocked

Keycloak CI - Base IT (5)

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertFalse(Assert.java:65)
	at org.junit.Assert.assertFalse(Assert.java:75)
...

Report flaky test

vaceksimon · 2024-07-08T09:44:31Z

...c/framework/src/main/java/org/keycloak/test/framework/server/KeycloakTestServerSupplier.java

    @Override
    public InstanceWrapper<KeycloakTestServer, KeycloakIntegrationTest> getValue(Registry registry, KeycloakIntegrationTest annotation) {
        KeycloakTestServerConfig serverConfig = SupplierHelpers.getInstance(annotation.config());
+        DatabaseConfig databaseConfig = annotation.dbconfig() != null ? SupplierHelpers.getInstance(annotation.dbconfig()) : new DefaultDatabaseConfig();


I think it can be left like this because the dbconfig() returns the default value (DefaultDatabaseConfig.class) if it is not entered.

Suggested change

DatabaseConfig databaseConfig = annotation.dbconfig() != null ? SupplierHelpers.getInstance(annotation.dbconfig()) : new DefaultDatabaseConfig();

DatabaseConfig databaseConfig = SupplierHelpers.getInstance(annotation.dbconfig());

stianst

What I had in mind is close to this, but not quite.

My idea was that we would have multiple database suppliers:

PostgresDatabaseSupplier
MySQLDatabaseSupplier
DevFileDatabaseSupplier
DevMemDatabaseSupplier

Each supplier would supply a TestDatabase with start/stop methods, and a getConfig() method.

You'd select which DB supplier to use with #30609 in the same way as we would select a different server run-mode or browser.

TestDatabase would be responsible for starting/stopping the DB container using test containers.

For now it's fine that DBs are not configurable (we just let it auto configure itself to some sensible defaults), and that we don't support connecting to external DBs, we can do that as a follow-up.

stianst · 2024-07-11T12:24:20Z

...poc/framework/src/main/java/org/keycloak/test/framework/database/ExternalDatabaseConfig.java

What's the purpose of this class?

stianst · 2024-07-11T12:26:39Z

...ork/src/main/java/org/keycloak/test/framework/server/AbstractKeycloakTestServerSupplier.java

    @Override
    public InstanceWrapper<KeycloakTestServer, KeycloakIntegrationTest> getValue(Registry registry, KeycloakIntegrationTest annotation) {
        KeycloakTestServerConfig serverConfig = SupplierHelpers.getInstance(annotation.config());
+        DatabaseSupplier databaseSupplier = new DevMemDatabaseSupplier();


This should not create the database supplier, but it should just use the registry to look it up:

TestDatabase database = registry.getDependency(TestDatabase.class, wrapper);

See

keycloak/test-poc/framework/src/main/java/org/keycloak/test/framework/realm/RealmSupplier.java

Lines 29 to 50 in 9247029

InstanceWrapper<RealmResource, TestRealm> wrapper = new InstanceWrapper<>(this, annotation);

LifeCycle lifecycle = annotation.lifecycle();

Keycloak adminClient = registry.getDependency(Keycloak.class, wrapper);

RealmConfig config = SupplierHelpers.getInstance(annotation.config());

RealmRepresentation realmRepresentation = config.getRepresentation();

if (realmRepresentation.getRealm() == null) {

String realmName = lifecycle.equals(LifeCycle.GLOBAL) ? config.getClass().getSimpleName() : registry.getCurrentContext().getRequiredTestClass().getSimpleName();

realmRepresentation.setRealm(realmName);

}

String realmName = realmRepresentation.getRealm();

wrapper.addNote(REALM_NAME_KEY, realmName);

adminClient.realms().create(realmRepresentation);

RealmResource realmResource = adminClient.realm(realmRepresentation.getRealm());

wrapper.setValue(realmResource, lifecycle);

return wrapper;

as an example on how to get a dependency in a supplier.

Rather then getting supplier I am thinking of fully manged DB called within the KeycloakIntegrationTest annotation as a prerequisite, do you think @stianst it makes sense?

lhanusov · 2024-07-11T12:28:55Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/DevMemDatabaseConfig.java

@@ -0,0 +1,4 @@
+package org.keycloak.test.framework.database;
+
+public class DevMemDatabaseConfig implements DatabaseConfig {


Don't you want to introduce rather DefaultDatabaseConfig class?

lhanusov · 2024-07-11T12:31:32Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/DatabaseSupplier.java

+
+import java.lang.annotation.Annotation;
+
+public abstract class DatabaseSupplier implements Supplier<TestDatabase, Annotation> {


Did you consider some TestManagedDatabase annotation which will be reused within KeycloakIntegrationTest annotation for ServiceLoader injection?

e.g. https://github.com/keycloak/keycloak/blob/main/testsuite/integration-arquillian/tests/base/src/main/java/org/keycloak/testsuite/util/JavascriptBrowser.java#L32

stianst · 2024-07-11T12:27:57Z

...amework/src/main/java/org/keycloak/test/framework/server/DistributionKeycloakTestServer.java


    @Override
-    public void start(KeycloakTestServerConfig serverConfig) {
+    public void start(KeycloakTestServerConfig serverConfig, TestDatabase testDatabase) {


Maybe it would be cleaner to just pass the DatabaseConfig? Makes it clear that it shouldn't be stopping/starting the db or anything like that ;)

...ork/src/main/java/org/keycloak/test/framework/server/AbstractKeycloakTestServerSupplier.java

...c/framework/src/main/java/org/keycloak/test/framework/server/EmbeddedKeycloakTestServer.java

lhanusov · 2024-07-11T12:49:00Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/DatabaseSupplier.java

+
+public abstract class DatabaseSupplier implements Supplier<TestDatabase, Annotation> {
+
+    protected static TestDatabase testDatabase = new TestDatabase(new DevMemDatabaseConfig());


Since the annotation introduction this could be then omitted and used like this: https://github.com/keycloak/keycloak/blob/main/test-poc/framework/src/main/java/org/keycloak/test/framework/KeycloakIntegrationTest.java#L15

jonkoops · 2024-07-15T09:44:38Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/DatabaseConfig.java

+import java.util.Collections;
+import java.util.Map;
+
+public interface DatabaseConfig {


Do we want to provide default implementations here? Or would all database providers have to implement this? Perhaps we can provide something else than an empty string such as null or Optional.empty()?

jonkoops · 2024-07-15T10:31:13Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/DatabaseConfig.java

+    default String vendor() {
+        return "";
+    }
+
+    default String containerImage() {
+        return "";
+    }
+


Both vendor and containerImage are not used anywhere, perhaps remove these until we need them to support a functionality?

Suggested change

default String vendor() {

return "";

}

default String containerImage() {

return "";

}

jonkoops · 2024-07-15T10:32:31Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/DatabaseSupplier.java

+    public TestDatabase getTestDatabase() {
+        return testDatabase;
+    }


This method is never used anywhere.

Suggested change

public TestDatabase getTestDatabase() {

return testDatabase;

}

...-poc/framework/src/main/java/org/keycloak/test/framework/database/DefaultDatabaseConfig.java

...oc/framework/src/main/java/org/keycloak/test/framework/database/DefaultDatabaseSupplier.java

jonkoops · 2024-07-15T10:44:53Z

test-poc/framework/src/main/java/org/keycloak/test/framework/database/TestDatabase.java

+    public void start() {
+
+    }
+
+    public void stop() {
+
+    }


None of these methods currently do anything? Is there a place where we actually need to start a database and tear it down? Also it seems that stop() is never called.

lhanusov · 2024-07-15T13:53:44Z

...c/framework/src/main/java/org/keycloak/test/framework/server/EmbeddedKeycloakTestServer.java

-        serverConfig.adminUserName().ifPresent(username -> rawOptions.add("--bootstrap-admin-username=" + username));
-        serverConfig.adminUserPassword().ifPresent(password -> rawOptions.add("--bootstrap-admin-password=" + password));
+
+        serverConfig.adminUserName().ifPresent(username -> System.setProperty("keycloakAdmin", username));


@miquelsi dont't forget to rebase to main, because this change was added few days ago: 4970a9b

Signed-off-by: Miquel Simon <[email protected]>

* Closes keycloak#30616. Added database suppliers. Signed-off-by: Miquel Simon <[email protected]> Co-authored-by: stianst <[email protected]>

keycloak-github-bot bot added the flaky-test label Jul 5, 2024

keycloak-github-bot bot reviewed Jul 5, 2024

View reviewed changes

vaceksimon suggested changes Jul 8, 2024

View reviewed changes

miquelsi force-pushed the main_db branch 2 times, most recently from 9ca00b2 to 93f632e Compare July 8, 2024 15:57

vaceksimon approved these changes Jul 9, 2024

View reviewed changes

stianst reviewed Jul 9, 2024

View reviewed changes

miquelsi force-pushed the main_db branch 3 times, most recently from 18f01af to cf6eae4 Compare July 10, 2024 08:38

miquelsi changed the title ~~DRAFT. Adding database configuration.~~ DRAFT. Adding database suppliers. Jul 10, 2024

keycloak-github-bot bot mentioned this pull request Jul 10, 2024

Flaky test: org.keycloak.testsuite.broker.KcOidcBrokerTest#testPostBrokerLoginFlowWithOTP_bruteForceEnabled #30188

Closed

miquelsi force-pushed the main_db branch 2 times, most recently from 20c5a3f to a7f61f5 Compare July 10, 2024 11:15

miquelsi changed the title ~~DRAFT. Adding database suppliers.~~ Adding database suppliers Jul 11, 2024

miquelsi force-pushed the main_db branch 2 times, most recently from db709fe to 9869df8 Compare July 11, 2024 10:15

miquelsi marked this pull request as ready for review July 11, 2024 11:05

miquelsi requested a review from a team as a code owner July 11, 2024 11:05

stianst reviewed Jul 11, 2024

View reviewed changes

lhanusov reviewed Jul 11, 2024

View reviewed changes

stianst reviewed Jul 11, 2024

View reviewed changes

lhanusov reviewed Jul 11, 2024

View reviewed changes

miquelsi force-pushed the main_db branch 2 times, most recently from 173da6f to 053a1a2 Compare July 12, 2024 14:10

keycloak-github-bot bot mentioned this pull request Jul 12, 2024

Flaky test: org.keycloak.testsuite.forms.RecoveryAuthnCodesAuthenticatorTest#test03AuthenticateRecoveryAuthnCodes #26176

Closed

jonkoops reviewed Jul 15, 2024

View reviewed changes

stianst force-pushed the main_db branch from 053a1a2 to 3f6b918 Compare July 15, 2024 10:53

keycloak-github-bot bot mentioned this pull request Jul 15, 2024

Flaky test: org.keycloak.testsuite.broker.KcSamlBrokerTest#testPostBrokerLoginFlowWithOTP_bruteForceEnabled #30641

Closed

lhanusov suggested changes Jul 15, 2024

View reviewed changes

miquelsi and others added 2 commits July 15, 2024 17:09

Closes keycloak#30616. Added database suppliers.

f8f61b1

Signed-off-by: Miquel Simon <[email protected]>

WIP

2b6fda4

miquelsi force-pushed the main_db branch from 3f6b918 to 2b6fda4 Compare July 15, 2024 15:09

stianst approved these changes Jul 16, 2024

View reviewed changes

stianst merged commit 73247f5 into keycloak:main Jul 16, 2024

miquelsi deleted the main_db branch July 16, 2024 07:40

stianst added a commit to stianst/keycloak that referenced this pull request Jul 25, 2024

Adding database suppliers (keycloak#31073)

c633984

* Closes keycloak#30616. Added database suppliers. Signed-off-by: Miquel Simon <[email protected]> Co-authored-by: stianst <[email protected]>

stianst mentioned this pull request Jul 25, 2024

testsuite poc managed realm wrapper #31618

Closed

	DatabaseConfig databaseConfig = annotation.dbconfig() != null ? SupplierHelpers.getInstance(annotation.dbconfig()) : new DefaultDatabaseConfig();
	DatabaseConfig databaseConfig = SupplierHelpers.getInstance(annotation.dbconfig());

	InstanceWrapper<RealmResource, TestRealm> wrapper = new InstanceWrapper<>(this, annotation);
	LifeCycle lifecycle = annotation.lifecycle();

	Keycloak adminClient = registry.getDependency(Keycloak.class, wrapper);

	RealmConfig config = SupplierHelpers.getInstance(annotation.config());
	RealmRepresentation realmRepresentation = config.getRepresentation();

	if (realmRepresentation.getRealm() == null) {
	String realmName = lifecycle.equals(LifeCycle.GLOBAL) ? config.getClass().getSimpleName() : registry.getCurrentContext().getRequiredTestClass().getSimpleName();
	realmRepresentation.setRealm(realmName);
	}

	String realmName = realmRepresentation.getRealm();
	wrapper.addNote(REALM_NAME_KEY, realmName);

	adminClient.realms().create(realmRepresentation);

	RealmResource realmResource = adminClient.realm(realmRepresentation.getRealm());
	wrapper.setValue(realmResource, lifecycle);

	return wrapper;

		@@ -0,0 +1,4 @@
		package org.keycloak.test.framework.database;

		public class DevMemDatabaseConfig implements DatabaseConfig {


		import java.lang.annotation.Annotation;

		public abstract class DatabaseSupplier implements Supplier<TestDatabase, Annotation> {


		public abstract class DatabaseSupplier implements Supplier<TestDatabase, Annotation> {

		protected static TestDatabase testDatabase = new TestDatabase(new DevMemDatabaseConfig());

	public TestDatabase getTestDatabase() {
	return testDatabase;
	}

Conversation

miquelsi commented Jul 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

keycloak-github-bot bot commented Jul 5, 2024

Unreported flaky test detected

org.keycloak.testsuite.federation.ldap.LDAPReadOnlyTest#testReadOnlyUserGetsPermanentlyLocked

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stianst left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stianst Jul 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

miquelsi commented Jul 5, 2024 •

edited

Loading

stianst left a comment •

edited

Loading

stianst Jul 11, 2024 •

edited

Loading