Add refresh-token support to OAuth2 #12664

s2lomon · 2022-06-02T15:22:41Z

Description

It adds refresh tokens support to OAuth2. Whenever refresh-token is issued
it's beeing wrapped along with access-token into self issued JWT token
and send to the client. Whenever token needs to be refreshed, refresh-token
is extracted and used to refresh access-token.

Is this change a fix, improvement, new feature, refactoring, or other?

new feature

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

it's a change to OAuth2Authenticator

How would you describe this change to a non-technical end user or system administrator?

Allows to automatically refresh tokens without opening new browser every time a token times out

Related issues, pull requests, and links

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Security
* Add refresh-token support to OAuth2

wendigo · 2022-06-02T16:00:24Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+        public static TokenPair fromOAuth2Response(Response tokens)
+        {
+            requireNonNull(tokens, "tokens is null");
+            return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());


Suggested change

return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());

return accessAndRefreshTokens(tokens.getAccessToken(), tokens.getRefreshToken());

Ok so I will leave it as it is. accessAndRefreshTokens suits me better with allowing null on the refreshToken parameter. It just simplifies usage in the authenticator and I don't want to repackage Optionals just to use this method here.

wendigo · 2022-06-02T16:00:51Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)
+        {
+            return new TokenPair(accessToken, Optional.of(requireNonNull(refreshToken, "refreshToken is null")));


Suggested change

return new TokenPair(accessToken, Optional.of(requireNonNull(refreshToken, "refreshToken is null")));

return new TokenPair(accessToken, Optional.ofNullable(refreshToken));

wendigo · 2022-06-02T16:01:21Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+            return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());
+        }
+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)


Suggested change

public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)

public static TokenPair accessAndRefreshTokens(String accessToken, @Nullable String refreshToken)

Will it be Nullable ?

Obviously not. I will remove it and make it so the caller of this factory method needs to make sure that there is an actual refreshToken to be passed. My mistake, thanks for noticing.

Ok I take it back, based on the 11xor6 comments it actually makes sense to have it Nullable and allow such TokenPair to exists, even with refresh_tokens enabled.

wendigo · 2022-06-02T16:02:22Z

core/trino-main/src/main/java/io/trino/server/security/AbstractBearerAuthenticator.java

@@ -75,4 +75,9 @@ protected abstract Optional<Identity> createIdentity(String token)
            throws UserMappingException;

    protected abstract AuthenticationException needAuthentication(ContainerRequestContext request, String message);
+
+    protected AuthenticationException needAuthentication(ContainerRequestContext request, String token, String message)


Suggested change

protected AuthenticationException needAuthentication(ContainerRequestContext request, String token, String message)

protected AuthenticationException needAuthentication(ContainerRequestContext request, String currentToken, String message)

wendigo · 2022-06-02T16:04:32Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+                    claims.get(REFRESH_TOKEN_KEY, String.class));
+        }
+        catch (ExpiredJwtException ex) {
+            throw new IllegalArgumentException("Token has timed out", ex);


Suggested change

throw new IllegalArgumentException("Token has timed out", ex);

throw new IllegalArgumentException("Token has expired", ex);

...ests/src/main/java/io/trino/tests/product/jdbc/TestExternalAuthorizerOAuth2RefreshToken.java

wendigo · 2022-06-02T16:20:02Z

...ests/src/main/java/io/trino/tests/product/jdbc/TestExternalAuthorizerOAuth2RefreshToken.java

+    private void disableHandler()
+    {
+        TestingRedirectHandlerInjector.setRedirectHandler(uri -> {
+            throw new IllegalStateException("Redirects has been disabled are not allowed");


Suggested change

throw new IllegalStateException("Redirects has been disabled are not allowed");

throw new UnsupportedOperationException("Redirect has been disabled as it's no longer allowed");

wendigo · 2022-06-02T16:20:27Z

...launcher/src/main/java/io/trino/tests/product/launcher/env/common/HydraIdentityProvider.java

@@ -72,7 +72,7 @@ public void extendEnvironment(Environment.Builder builder)
        binder.exposePort(hydraConsent, 3000);

        DockerContainer hydra = new DockerContainer(HYDRA_IMAGE, "hydra")
-                .withEnv("LOG_LEVEL", "debug")
+                .withEnv("LOG_LEVEL", "trace")


my bad, a leftover from the testing

wendigo · 2022-06-02T16:20:51Z

core/trino-main/src/test/java/io/trino/server/security/oauth2/TestJWTTokenSerializer.java

+        }
+    }
+
+    private static class ManualClock


TestingClock

wendigo · 2022-06-02T16:21:03Z

core/trino-main/src/test/java/io/trino/server/security/oauth2/TestJWTTokenSerializer.java

+            return currentTime;
+        }
+
+        public void moveCurrentTime(Duration currentTimeDelta)


advanceBy()

11xor6

This is only a review of the first commit (excluding tests), I will continue with a follow up review, however there are things that must be addressed for security purposes before allowing this to be merged.

11xor6 · 2022-06-02T22:05:06Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/NimbusOAuth2Client.java

+            if (nonce.isEmpty()) {
+                throw new ChallengeFailedException("Missing nonce");
+            }


There is no guarantee that we have a nonce hence why the parameter is Optional. There are existing code paths which do not provide the OAuth flow with a nonce, by requiring it here those code paths can only result in an error.

So this is a small refactoring of the existing code (required to share code for the refresh token, that doesn't have a nonce). All the paths that are not requiring nonce cookies are already going through some other paths (or are not working). I think that some time ago we've decided to have this initialization url, that sets up the nonce cookie etc. so it should be covered everywhere.

If this function can never be called without a nonce, then please change the signature. However I'm pretty sure not all paths generate a nonce.

Right so it's always required for OIDC flows, but never present in Oauth2.0 flows. Still, since the flow is an interface, we have this Optional in a parameter list. Maybe it could be refactored , but I think it would take much time to hide this one parameter and I wouldn't like to do it in this pr, as it would include too many changes.

I see what I was missing here. This is specifically the OIDC AuthorizationCodeFlow implementation. I am still worried that there is a way for us to end up here without a nonce (perhaps through the UI?), but if it exists I can't find it. This check now makes sense.

I will leave it as it is then.

11xor6 · 2022-06-02T22:37:34Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializerModule.java

+        install(conditionalModule(RefreshTokensConfig.class, config -> config.getSigningAlgorithm().isEllipticCurve() || config.getSigningAlgorithm().isRsa(),
+                keyPairBinder -> {
+                    configBinder(keyPairBinder).bindConfig(AsymmetricKeyPairConfig.class);


Why are we supporting asymmetric encryption for signing tokens we issue? Is there some need to have a second or third party decode our tokens? It seems to me we issue these tokens to the client and we are the only ones that need to verify the signature.

It was just easy to be done. The whole purpose of this is to not allow someone to tamper with our Tokens, so being the only one that can sign this token is exactly what we need. We can allow users to setup their own symetric or asymetric keys, but then it's a user option and I feel ok with letting people do this. I don't know, maybe I'm too paranoid with this - would like to know your thoughts.

Asymmetric keys buy us nothing in this case. The only reason to even consider them is to allow a third party or another service to introspect and/or trust the tokens. Given that's the case I see no reason to support them. Additionally this combined with the need to encrypt values within the tokens is problematic as it's computationally expensive and easy to misunderstand how to do the encryption properly.

Ok, I've removed all the assymetric keys setup, leaving with just symetric one to encrypt the token. It's much easier this way and I guess you are right that it doesn't give us anything that we would actually need.

11xor6 · 2022-06-02T22:44:48Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+                .signWith(signingKey, signatureAlgorithm)
+                .compressWith(COMPRESSION_CODEC);
+        return jwt.compact();


Signing and compressing a token containing plain text access and refresh tokens is incredibly insecure. Both access tokens and refresh tokens are secrets that have been entrusted to us (Trino) and as such if we're returning them to the client in any form they MUST be encrypted.

The options available are to either encrypt the specific values included in the token or encrypt the token as a whole (which JJWT supports).

I agree, I think using asymetric encryption here pose a security risk without any benefits as Trino is the only recipient of the token. JWE should be a better fit here: https://datatracker.ietf.org/doc/html/rfc7516

Ok, I'm not sure however how this differs from keeping these tokens in the WebBrowser Cookies, as in order to use the RefreshToken you need to have valid client credentials, so it's not that you can just steal it. Our initial idea was that we should secure our token from any kind of tampering with it and that sharing these tokens is "OKish", as it already happens in the WebBrowser. Still - I'm fine with encrypting these claims and then signing and compressing the whole thing.

This is a matter of trust, access and refresh tokens are entrusted to Trino, they are not entrusted to the client. Thus giving the user access to the tokens (even if they're their own) breaks our contract with the token issuer (IdP). We don't control the client environment and there's no way for us to ensure or validate that no third-party has access to any storage mechanism the client supports and/or uses, and that is in addition to the possible malicious action of the client/credential holder themselves. Access tokens (and by extension refresh tokens) can grant access to resources which are not directly entrusted to the credential holder. In a way the ability to act on behalf of another user can be more dangerous than what that user could do themselves.

Ok I agree, I've not been thinking about the other end of the equation when wanted to protect ourselves from token tampering, so that preventing someone to steal the tokens is as valid as preventing someone to swap these tokens when in communication with us. I'm about to add the WebUI part as well, do you think that we should encrypt these tokens in Cookies there?

In addition, we are not encrypting any token for the accessTokens in WebUI, so maybe we should do this as a followup pr? What's more, maybe it actually make sense to issue these encrypted tokens, even if refreshTokens are not available? - again as a follow up.

If we're sending access tokens unencrypted to the UI then we definitely need to fix that ASAP. If a token (access or refresh) ever leaves our control then it should be encrypted. Also if it's possible we shouldn't even hold on to tokens unless we need them.

11xor6 · 2022-06-02T22:58:09Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        catch (ExpiredJwtException ex) {
+            throw new IllegalArgumentException("Token has timed out", ex);


We need proper controls around the expiration of these JWT tokens. The expiration of the contained access token is not good enough as when it expires our token would expire and that invalidates the viability of the refresh token. I propose a configurable maximum token lifetime (as the token expiration) coupled with a secondary expiration related to the access token (stored in a separate field and validated as part of authentication).

Ok so this scenario won't happen, as right now we are keeping our token with the expiration of an accessToken and I'm providing long expirationTimeExtensionSeconds (it's in seconds because the api accepts only seconds).
This means that the expiration time of token issued by us is accessToken.expiration + expirationTimeExtension.
Right now refresh will happen only after the accessToken has expired, so the expirationTimeExtension is actually only here to allow refresh to happen.

I understand that we implicitly capture when the access token expires, however I'm asking here that it be explicit instead. The current value for the token expiration is good, having it expire some time after the access token itself should expire is good. However I would also like to see the token expiration encoded into the token and used to force a refresh of the token rather than us having to find out the hard way by retrieving the claims associated with the token.

I see and I like that this gives us easy possibility to refresh tokens before they timeout.

11xor6 · 2022-06-02T23:00:45Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        this.parser = newJwtParserBuilder()
+                .setSigningKey(verificationKey)
+                .setClock(this.clock)
+                .requireIssuer(this.issuer)


The issuer should most certainly be required and should have a value that uniquely identifies this Trino instance. Otherwise if two Trino instances have the same signing/encryption keys it would be allowed to use a token issued to one on the other.

11xor6 · 2022-06-02T23:18:16Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        Optional<Map<String, Object>> claims = client.getClaims(tokenPair.getAccessToken());
+        JwtBuilder jwt = newJwtBuilder()
+                .addClaims(claims.orElse(Map.of()))


There is no need to, nor should we, include the claims from the access token. These could contain sensitive values which should not be present in our token. Additionally we will validate the access token on authentication and retrieve these claims then. Thus there's no need to do the extra work here; it makes the token smaller and prevents the possible leakage of sensitive data.

Ok the reason behind it was to make it available for some network proxies or load balancers that could potentially tap into claims of the accessToken. I would gladly remove it if we think that this is not necessary.

No network proxy or load balancer would even be able to access these tokens as it's all behind TLS/SSL.

So for example in AWS you can have a Gateway, that modifies the requests through some AWS Lambdas. It can also redirect these calls, based on the requests etc.. It's all happening in the DMZ, so no TLS/SSL is involved (at least during the processing itself). It's a little bit farfetched, but just so you know, that this is possible in principle (I've actually seen systems that were doing first layer of authorization in such places). Still - I'm happy to remove these claims.

Yeah, I don't think we'll need any introspection, if it does come down to that we can reexamine at that time, but I don't think it'll be an issue.

11xor6 · 2022-06-02T23:28:48Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Authenticator.java

@@ -77,4 +83,14 @@ protected AuthenticationException needAuthentication(ContainerRequestContext req
        URI tokenUri = request.getUriInfo().getBaseUri().resolve(getTokenUri(authId));
        return new AuthenticationException(message, format("Bearer x_redirect_server=\"%s\", x_token_server=\"%s\"", initiateUri, tokenUri));
    }
+
+    @Override
+    protected AuthenticationException needAuthentication(ContainerRequestContext request, String token, String message)


Given the new signature of this method is the other needAuthentication implementation necessary? It seems we would always use this version.

11xor6 · 2022-06-02T23:39:06Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Authenticator.java

+        Optional<UUID> refreshProcess = tokenRefresher.refreshToken(tokenPair);
+        return refreshProcess.map(refreshId -> request.getUriInfo().getBaseUri().resolve(getTokenUri(refreshId)))
+                .map(tokenUri -> new AuthenticationException(message, format("Bearer x_token_server=\"%s\"", tokenUri)))


This seems unnecessary, we already have the refresh token and no user interaction is required in order acquire a new access (and possible refresh) token. We could easily perform the refresh token flow and return a new JWT token instead of going through all the work of returning this response and making the client do more work.

Perhaps we need to add an additional step to the AbstractBearerAuthenticator to support this behavior?

Ok so we can't do that I think. I don't believe that any of our client (jdbc/cli/python etc.) is taking the auth response headers and replacing it for the next request call. We do however allowed this Bearer x_token_server header to contain only x_token_server, so that client would only start polling for a new token to replace the old one, but would not try to open web-browser etc. This should be supported by every client that claims to support OAuth2/ExternalAuthentication protocol and it's the only way we can replace the existing token in these clients. In a way, this is exactly what you are talking about, but executed in the norms of the already existing flow.

My bad, I thought this worked differently and after digging into the code I realize that this is the right thing to do. It would be better if the client protocol was cleaner, but this is what we have right now.

11xor6 · 2022-06-02T23:45:28Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)
+        {
+            return new TokenPair(accessToken, Optional.of(requireNonNull(refreshToken, "refreshToken is null")));


There is no guarantee that we will ever be issued a refresh token, it is always at the discretion of the IdP if they issue one, making the assumption that we will always receive one is a bug waiting to happen. It is even such that different users or different circumstances completely out of our control could cause the IdP to not issue a refresh token.

Additionally a successful response will always contain an access token and that should be verified to be non-null.

Sure, I would just rather want that to be verified by the caller, so that the caller would use another method like TokenPair.accessToken(). I've added however method to handle Response from OAuth2Client, so this what you are saying should be handled there - I will double check that this is the case.

If the calling code is choosing a method based on an optional field then there should be a single method to handle both cases, there's no real need for the calling code to care if the refresh token is present or not.

Ok you are right, It is a bug waiting to happen.

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenRefresher.java

dain

The comments from @11xor6 should be addressed

Praveen2112 · 2022-06-03T11:54:24Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/AsymmetricKeyPairConfig.java

+    @ConfigSecuritySensitive
+    public AsymmetricKeyPairConfig setPrivateSigningKeyPath(String keyPath)
+    {
+        if (isNullOrEmpty(keyPath)) {


nit: We could throw some configuration error - if it is sent to empty ?

Any issues on using File instead of String ?

We might also need to add assertion if the file exists.

Ok, we can distinguish null from empty. File should work as well.

Praveen2112 · 2022-06-03T13:01:56Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+            return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());
+        }
+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)


Will it be Nullable ?

Praveen2112 · 2022-06-03T13:06:34Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/RefreshTokensConfig.java

+
+    @Config("http-server.authentication.oauth2.refresh-tokens.signing.algorithm")
+    @ConfigDescription("Signature algorithm that will be used to secure a token. If no keys will be provided, it will be used as a base to generate adequate keys")
+    public RefreshTokensConfig setSigningAlgorithm(String algorithm)


Since it is an enum , airlift would handle the conversion right ?

Yes, but one of the value of the SignatureAlgorithm is NONE. I think that I would like to prevent that from beeing used. (I've forget to add that here) WIll airlift print out all possible values if we use enum here? If not then I can use enum and make sure that it's never NONE.

Praveen2112 · 2022-06-03T13:07:51Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/RefreshTokensConfig.java

+
+    @Config("http-server.authentication.oauth2.refresh-tokens.expiration.seconds")
+    @ConfigDescription("Signature algorithm that will be used to secure a token. If no keys will be provided, it will be used as a base to generate adequate keys")
+    public RefreshTokensConfig setRefreshTokenExpirationExtensionSeconds(long refreshTokenExpirationExtensionSeconds)


Can we use Duration here - it would be more configurable ?

I was thinking about the Duration, but the thing is that I couldn't think of a good way of validating it. This value is later on used as a seconds, so anything lower that that shouldn't be allowed in the config as well. On the other hand, having an option to disable it might be handy as well. Right now, it would require us to proactively refreshing tokens, before they expire, in order for such an option to have sense, which is not yet implemented, but seems to be a natural evolution of such mechanisms.
On the other hand, we could have Duration here with @Min("1s") and say that it's not possible to disable it completely - even with the proactive refreshing.

s2lomon

Thanks for quick review guys. Please take a look at my responses.

s2lomon · 2022-06-03T14:49:20Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/AsymmetricKeyPairConfig.java

+    @ConfigSecuritySensitive
+    public AsymmetricKeyPairConfig setPrivateSigningKeyPath(String keyPath)
+    {
+        if (isNullOrEmpty(keyPath)) {


Ok, we can distinguish null from empty. File should work as well.

s2lomon · 2022-06-03T14:51:36Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        this.parser = newJwtParserBuilder()
+                .setSigningKey(verificationKey)
+                .setClock(this.clock)
+                .requireIssuer(this.issuer)


Ok so I guess, we should have version + something elese. The question is what should it be, as I think tokens should survive coordinator restart (assuming that the signing keys are provided and not generated)

s2lomon · 2022-06-03T14:52:34Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+                    claims.get(ACCESS_TOKEN_KEY, String.class),
+                    claims.get(REFRESH_TOKEN_KEY, String.class));
+        }
+        catch (ExpiredJwtException ex) {


Yes you are right, I will add proper error handling here.

s2lomon · 2022-06-03T14:56:16Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        catch (ExpiredJwtException ex) {
+            throw new IllegalArgumentException("Token has timed out", ex);


Ok so this scenario won't happen, as right now we are keeping our token with the expiration of an accessToken and I'm providing long expirationTimeExtensionSeconds (it's in seconds because the api accepts only seconds).
This means that the expiration time of token issued by us is accessToken.expiration + expirationTimeExtension.
Right now refresh will happen only after the accessToken has expired, so the expirationTimeExtension is actually only here to allow refresh to happen.

s2lomon · 2022-06-03T14:56:49Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+
+        Optional<Map<String, Object>> claims = client.getClaims(tokenPair.getAccessToken());
+        JwtBuilder jwt = newJwtBuilder()
+                .addClaims(claims.orElse(Map.of()))


s2lomon · 2022-06-03T15:33:54Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+            return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());
+        }
+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)


Obviously not. I will remove it and make it so the caller of this factory method needs to make sure that there is an actual refreshToken to be passed. My mistake, thanks for noticing.

s2lomon · 2022-06-03T15:35:32Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)
+        {
+            return new TokenPair(accessToken, Optional.of(requireNonNull(refreshToken, "refreshToken is null")));


Sure, I would just rather want that to be verified by the caller, so that the caller would use another method like TokenPair.accessToken(). I've added however method to handle Response from OAuth2Client, so this what you are saying should be handled there - I will double check that this is the case.

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenRefresher.java

core/trino-main/src/test/java/io/trino/server/ui/TestWebUi.java

s2lomon · 2022-06-03T15:39:26Z

...launcher/src/main/java/io/trino/tests/product/launcher/env/common/HydraIdentityProvider.java

@@ -72,7 +72,7 @@ public void extendEnvironment(Environment.Builder builder)
        binder.exposePort(hydraConsent, 3000);

        DockerContainer hydra = new DockerContainer(HYDRA_IMAGE, "hydra")
-                .withEnv("LOG_LEVEL", "debug")
+                .withEnv("LOG_LEVEL", "trace")


my bad, a leftover from the testing

lukasz-walkiewicz

I took a quick look and it looks really promissing!
A few initial comments.

lukasz-walkiewicz · 2022-06-02T21:03:49Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/AsymmetricKeyPairConfig.java

+
+public class AsymmetricKeyPairConfig
+{
+    private String privateKeyPath;


Woudn't storing both keys in the same file be easier? IIRC it's already the case for SSL in Trino, you can use a file with both.

lukasz-walkiewicz · 2022-06-02T21:14:07Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/AsymmetricKeyPairConfig.java

+    @PostConstruct
+    public void validate()
+    {
+        verify((publicKeyPath != null && privateKeyPath != null) || (publicKeyPath == null && privateKeyPath == null),


nit

verify(!(publicKeyPath == null ^ privateKeyPath == null));

lukasz-walkiewicz · 2022-06-03T12:36:55Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializerModule.java

+    @Singleton
+    @Provides
+    @Inject
+    @ForTokenRefresher


Wouldn't it be easier to just use RefreshTokensConfig instead of this annotated type?

lukasz-walkiewicz · 2022-06-03T13:00:16Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Config.java

@@ -220,6 +221,19 @@ public OAuth2Config setUserMappingFile(File userMappingFile)
        return this;
    }

+    public boolean isEnableRefreshTokens()


Shouldn't it be inferred from scopes content? If offline_access scope is present then refreshing tokens is enabled.

I was thinking about that, but I can't find any clear indication that scopes that enables refresh_token are always offline, or offline_access. I don't find it in the rfc at least. Due to that, I'm thinking it's best to keep it explicitly required to enable refresh tokens and to require additional scopes to be present.

If you know any place that makes it a part of the standard then please share. Without this I wouldn't like to remove this explicit flag.

The IdP can issue a refresh token independently of the offline_access scope. The only thing the offline_access scope is there for is to allow requesting a refresh token that can be used when the user is no longer present for the session (i.e. to do background or offline processing). It's also worth noting that it's up to the IdP's discretion to issue a refresh token and as such you should never expect to receive one. See:

https://datatracker.ietf.org/doc/html/rfc6749#section-1.5

https://datatracker.ietf.org/doc/html/rfc6749#section-5.1

https://datatracker.ietf.org/doc/html/rfc6749#section-10.4

https://openid.net/specs/openid-connect-core-1_0.html#OfflineAccess

lukasz-walkiewicz · 2022-06-03T13:03:32Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/NimbusOAuth2Client.java

    private interface AuthorizationCodeFlow
    {
        Request createAuthorizationRequest(String state, URI callbackUri);

        Response getOAuth2Response(String code, URI callbackUri, Optional<String> nonce)
                throws ChallengeFailedException;
+
+        Response refreshTokens(String refreshToken)


Response getTokens(String code, URI callbackUri, Optional<String> nonce); Response getTokens(String refreshToken);

?

In general I like this, conceptually they are doing the same thing (calling the token endpoint on the IdP), just with different parameters. However I'm fine with just keeping and overloading the existing name getOAuth2Response which I think is clearer.

I would go with resolveTokens and refreshTokens as you can't actually call Response getTokens(String code, URI callbackUri, Optional<String> nonce); multiple times in a row, so having a method name that actually gives us a hint that a possibly non repeatable action is taking place is a better thing to have. Still, let's have it in a subsequent pr.

lukasz-walkiewicz · 2022-06-03T13:07:41Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/RefreshTokensConfig.java

+    }
+
+    @Config("http-server.authentication.oauth2.refresh-tokens.signing.algorithm")
+    @ConfigDescription("Signature algorithm that will be used to secure a token. If no keys will be provided, it will be used as a base to generate adequate keys")


So either keys or signature algorithm? If yes then maybe move to the same file and add validation

lukasz-walkiewicz · 2022-06-03T13:17:12Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+                .signWith(signingKey, signatureAlgorithm)
+                .compressWith(COMPRESSION_CODEC);
+        return jwt.compact();


I agree, I think using asymetric encryption here pose a security risk without any benefits as Trino is the only recipient of the token. JWE should be a better fit here: https://datatracker.ietf.org/doc/html/rfc7516

s2lomon

Sorry for a bit longer pause, I've had to think over few things here to fit your comments right in. Please take a look again.

s2lomon · 2022-06-08T11:38:21Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        catch (ExpiredJwtException ex) {
+            throw new IllegalArgumentException("Token has timed out", ex);


I see and I like that this gives us easy possibility to refresh tokens before they timeout.

s2lomon · 2022-06-08T11:47:55Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        Optional<Map<String, Object>> claims = client.getClaims(tokenPair.getAccessToken());
+        JwtBuilder jwt = newJwtBuilder()
+                .addClaims(claims.orElse(Map.of()))


So for example in AWS you can have a Gateway, that modifies the requests through some AWS Lambdas. It can also redirect these calls, based on the requests etc.. It's all happening in the DMZ, so no TLS/SSL is involved (at least during the processing itself). It's a little bit farfetched, but just so you know, that this is possible in principle (I've actually seen systems that were doing first layer of authorization in such places). Still - I'm happy to remove these claims.

s2lomon · 2022-06-08T11:56:12Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+                .signWith(signingKey, signatureAlgorithm)
+                .compressWith(COMPRESSION_CODEC);
+        return jwt.compact();


Ok I agree, I've not been thinking about the other end of the equation when wanted to protect ourselves from token tampering, so that preventing someone to steal the tokens is as valid as preventing someone to swap these tokens when in communication with us. I'm about to add the WebUI part as well, do you think that we should encrypt these tokens in Cookies there?

In addition, we are not encrypting any token for the accessTokens in WebUI, so maybe we should do this as a followup pr? What's more, maybe it actually make sense to issue these encrypted tokens, even if refreshTokens are not available? - again as a follow up.

s2lomon · 2022-06-13T14:30:09Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializerModule.java

+        install(conditionalModule(RefreshTokensConfig.class, config -> config.getSigningAlgorithm().isEllipticCurve() || config.getSigningAlgorithm().isRsa(),
+                keyPairBinder -> {
+                    configBinder(keyPairBinder).bindConfig(AsymmetricKeyPairConfig.class);


Ok, I've removed all the assymetric keys setup, leaving with just symetric one to encrypt the token. It's much easier this way and I guess you are right that it doesn't give us anything that we would actually need.

s2lomon · 2022-06-13T14:36:45Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/NimbusOAuth2Client.java

+            if (nonce.isEmpty()) {
+                throw new ChallengeFailedException("Missing nonce");
+            }


Right so it's always required for OIDC flows, but never present in Oauth2.0 flows. Still, since the flow is an interface, we have this Optional in a parameter list. Maybe it could be refactored , but I think it would take much time to hide this one parameter and I wouldn't like to do it in this pr, as it would include too many changes.

s2lomon · 2022-06-13T14:48:14Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Config.java

@@ -220,6 +221,19 @@ public OAuth2Config setUserMappingFile(File userMappingFile)
        return this;
    }

+    public boolean isEnableRefreshTokens()


I was thinking about that, but I can't find any clear indication that scopes that enables refresh_token are always offline, or offline_access. I don't find it in the rfc at least. Due to that, I'm thinking it's best to keep it explicitly required to enable refresh tokens and to require additional scopes to be present.

If you know any place that makes it a part of the standard then please share. Without this I wouldn't like to remove this explicit flag.

s2lomon · 2022-06-13T15:00:37Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+            return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());
+        }
+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)


Ok I take it back, based on the 11xor6 comments it actually makes sense to have it Nullable and allow such TokenPair to exists, even with refresh_tokens enabled.

s2lomon · 2022-06-13T15:01:41Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+
+        public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)
+        {
+            return new TokenPair(accessToken, Optional.of(requireNonNull(refreshToken, "refreshToken is null")));


Ok you are right, It is a bug waiting to happen.

11xor6

Generally looks better. A few additional comments, but most of the major stuff has been addressed.

11xor6 · 2022-06-15T07:51:05Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JweSecretKeysProvider.java

+    private static SecretKey createKey(SecretKeyConfig config)
+            throws NoSuchAlgorithmException
+    {
+        SecretKey signingKey = config.getSigningKey();


Nit: this isn't a signing key any more, please rename.

Correct, that's a leftover. Renaming it.

11xor6 · 2022-06-15T07:55:55Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Config.java

@@ -220,6 +221,19 @@ public OAuth2Config setUserMappingFile(File userMappingFile)
        return this;
    }

+    public boolean isEnableRefreshTokens()


The IdP can issue a refresh token independently of the offline_access scope. The only thing the offline_access scope is there for is to allow requesting a refresh token that can be used when the user is no longer present for the session (i.e. to do background or offline processing). It's also worth noting that it's up to the IdP's discretion to issue a refresh token and as such you should never expect to receive one. See:

https://datatracker.ietf.org/doc/html/rfc6749#section-1.5

https://datatracker.ietf.org/doc/html/rfc6749#section-5.1

https://datatracker.ietf.org/doc/html/rfc6749#section-10.4

https://openid.net/specs/openid-connect-core-1_0.html#OfflineAccess

11xor6 · 2022-06-15T10:17:46Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializer.java

+        Optional<Map<String, Object>> accessTokenClaims = client.getClaims(tokenPair.getAccessToken());
+        JwtBuilder jwt = newJwtBuilder()
+                .setExpiration(Date.from(clock.instant().plusMillis(accessTokenExpiration.toMillis())))
+                .setSubject(accessTokenClaims.map(claims -> claims.get(SUBJECT).toString()).orElse("Unknown"))


This should use the principalField from OAuth2Config rather than SUBJECT as that is not necessarily the correct claim; see OAuth2Authenticator.

good point.

11xor6 · 2022-06-15T10:18:45Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializer.java

+
+        this.parser = newJwtParserBuilder()
+                .setClock(() -> Date.from(clock.instant()))
+                .requireIssuer(this.issuer)


Also require the proper audience.

My bad, fixing.

11xor6 · 2022-06-15T10:22:54Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializer.java

+        return new IllegalStateException("Expected refresh token to be present. Please check your identity provider setup, or disable refresh tokens");
+    }
+
+    static class ZstdCodec


Let's move this to its own class.

11xor6 · 2022-06-15T10:39:31Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/NimbusOAuth2Client.java

+            if (nonce.isEmpty()) {
+                throw new ChallengeFailedException("Missing nonce");
+            }


I see what I was missing here. This is specifically the OIDC AuthorizationCodeFlow implementation. I am still worried that there is a way for us to end up here without a nonce (perhaps through the UI?), but if it exists I can't find it. This check now makes sense.

11xor6 · 2022-06-15T10:41:33Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/NimbusOAuth2Client.java

    private interface AuthorizationCodeFlow
    {
        Request createAuthorizationRequest(String state, URI callbackUri);

        Response getOAuth2Response(String code, URI callbackUri, Optional<String> nonce)
                throws ChallengeFailedException;
+
+        Response refreshTokens(String refreshToken)


In general I like this, conceptually they are doing the same thing (calling the token endpoint on the IdP), just with different parameters. However I'm fine with just keeping and overloading the existing name getOAuth2Response which I think is clearer.

11xor6 · 2022-06-15T10:48:52Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/NimbusOAuth2Client.java

+            if (nonce.isEmpty()) {
+                throw new ChallengeFailedException("Missing nonce");
+            }


No, I believe this should be the checked ChallengeFailedException a missing nonce here does mean that the challenge failed, this isn't a precondition, it's a requirement of the protocol.

s2lomon

I've fixed extending issues and I've added a WebUI implementation for refresh tokens.
I've decided to replace existing Cookie with encrypted one, as this implementation seems much easier and just fitting in exiting design, that creating separate Cookie for refresh token etc. (not to mention lack of encryption for the existing accessToken)

Please take a look again.

s2lomon · 2022-06-16T07:17:25Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenPairSerializer.java

+        public static TokenPair fromOAuth2Response(Response tokens)
+        {
+            requireNonNull(tokens, "tokens is null");
+            return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());


Ok so I will leave it as it is. accessAndRefreshTokens suits me better with allowing null on the refreshToken parameter. It just simplifies usage in the authenticator and I don't want to repackage Optionals just to use this method here.

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Service.java

11xor6 · 2022-06-21T13:48:43Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/SecretKeyConfig.java

+
+import static com.google.common.base.Strings.isNullOrEmpty;
+
+public class SecretKeyConfig


Merge this config with RefreshTokensConfig now that there is only one key mechanism.

11xor6 · 2022-06-21T13:50:06Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/SecretKeyConfig.java

+
+public class SecretKeyConfig
+{
+    private SecretKey secretKey;


We could default this to a random value so that it's configuration would be optional. Really the only reason you might want to set this value is if you wanted tokens to persist across restarts. I can go either way on this one, but I think it might be convenient.

JweSecretKeysProvider implements this. I'm not sure if implementing it as a part of config object.

11xor6 · 2022-06-21T13:57:05Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializer.java

+        Optional<Map<String, Object>> accessTokenClaims = client.getClaims(tokenPair.getAccessToken());
+        JwtBuilder jwt = newJwtBuilder()
+                .setExpiration(Date.from(clock.instant().plusMillis(accessTokenExpiration.toMillis())))
+                .claim(principalField, accessTokenClaims.map(claims -> claims.get(principalField).toString()).orElse("Unknown"))


Not having the proper principal field is an error.

11xor6 · 2022-06-21T13:58:16Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JweSecretKeysProvider.java

+
+import static java.util.Objects.requireNonNull;
+
+public class JweSecretKeysProvider


I'm not sure we need this class, can it be inlined?

Seems like this is only used in JWETokenSerializer and we can just move this there.

11xor6 · 2022-06-21T13:59:35Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWTTokenSerializer.java

+        Optional<Map<String, Object>> claims = client.getClaims(tokenPair.getAccessToken());
+        JwtBuilder jwt = newJwtBuilder()
+                .addClaims(claims.orElse(Map.of()))
+                .setIssuer(issuer)


11xor6

Generally looks pretty good, there's still this comment to address, and I've added a few other minor things. I'm happy to walk someone through how to accomplish the change request from the linked comment.

Nik's concerns have been addressed

Praveen2112 · 2022-06-22T11:58:04Z

@11xor6 , @wendigo Addressed the comments.

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Authenticator.java

11xor6 · 2022-06-22T17:56:43Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializer.java

+            throw new IllegalArgumentException("Claims are missing");
+        }
+        Map<String, Object> claims = accessTokenClaims.get();
+        if (claims.containsKey(principalField)) {


Missing !

11xor6 · 2022-06-22T18:00:16Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializer.java

+    private final AESDecrypter jweDecrypter;
+    private final String principalField;
+
+    public JWETokenSerializer(


nit: should be JweTokenSerializer

11xor6 · 2022-06-22T18:01:38Z

core/trino-main/src/test/java/io/trino/server/security/oauth2/TestJWETokenSerializer.java

+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+public class TestJWETokenSerializer


nit: TestJweTokenSerializer

11xor6 · 2022-06-22T18:03:15Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/JWETokenSerializerModule.java

+
+import static io.airlift.configuration.ConfigBinder.configBinder;
+
+public class JWETokenSerializerModule


nit: JweTokenSerializerModule

11xor6

I few nits left over, but overall I'm happy with this.

core/trino-main/src/main/java/io/trino/server/security/oauth2/JweTokenSerializer.java

...rc/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeOauth2Refresh.java

...ests/src/main/java/io/trino/tests/product/jdbc/TestExternalAuthorizerOAuth2RefreshToken.java

core/trino-main/src/main/java/io/trino/server/security/oauth2/TokenRefresher.java

lukasz-walkiewicz · 2022-06-23T14:24:07Z

core/trino-main/src/main/java/io/trino/server/security/oauth2/RefreshTokensConfig.java

+        return refreshTokenExpiration;
+    }
+
+    @Config("http-server.authentication.oauth2.refresh-tokens.refresh.timeout")


I'm not sure about the description and overall naming convention. Using refresh and refresh-tokens for different things might be confusing. Expiration time for refresh token it's not really expiration time of the refresh token but the token issued by Trino.

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Service.java

It adds refresh tokens support to OAuth2. Whenever refresh-token is issued it's beeing wrapped along with access-token into self issued JWT token and send to the client. Whenever token needs to be refreshed, refresh-token is extracted and used to refresh access-token.

Co-authored-by: aczajkowski <[email protected]>

wendigo · 2022-06-27T14:17:57Z

🎉

colebow · 2022-06-27T20:42:46Z

Would we like to mention this change in release notes, and if so, could you propose a release note? Also, does this require documentation on the OAuth2 page?

cc @Praveen2112

kokosing · 2022-06-27T20:56:51Z

Yay!! 🎉

Praveen2112 · 2022-06-28T09:43:26Z

@colebow I have updated the release notes entry in the PR description and have also linked the documentation PR.

cla-bot bot added the cla-signed label Jun 2, 2022

s2lomon requested review from lukasz-walkiewicz and 11xor6 June 2, 2022 15:22

github-actions bot added the tests:hive label Jun 2, 2022

s2lomon requested review from kokosing, Praveen2112 and wendigo June 2, 2022 15:22

wendigo reviewed Jun 2, 2022

View reviewed changes

11xor6 suggested changes Jun 3, 2022

View reviewed changes

dain previously requested changes Jun 3, 2022

View reviewed changes

Praveen2112 reviewed Jun 3, 2022

View reviewed changes

s2lomon commented Jun 3, 2022

View reviewed changes

lukasz-walkiewicz reviewed Jun 6, 2022

View reviewed changes

s2lomon force-pushed the refresh-token branch from cbf7da5 to 5ed2326 Compare June 13, 2022 15:07

s2lomon commented Jun 13, 2022

View reviewed changes

11xor6 reviewed Jun 15, 2022

View reviewed changes

s2lomon force-pushed the refresh-token branch from 5ed2326 to c561a4c Compare June 16, 2022 10:50

s2lomon commented Jun 16, 2022

View reviewed changes

Praveen2112 force-pushed the refresh-token branch 2 times, most recently from 54f0beb to 049deff Compare June 20, 2022 17:23

wendigo reviewed Jun 21, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Service.java Outdated Show resolved Hide resolved

wendigo reviewed Jun 21, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Service.java Outdated Show resolved Hide resolved

11xor6 reviewed Jun 21, 2022

View reviewed changes

Praveen2112 force-pushed the refresh-token branch from 049deff to 824de55 Compare June 22, 2022 11:48

Praveen2112 force-pushed the refresh-token branch 2 times, most recently from e2ee31e to 1a8d72d Compare June 22, 2022 13:48

11xor6 reviewed Jun 22, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/server/security/oauth2/OAuth2Authenticator.java Show resolved Hide resolved

Praveen2112 force-pushed the refresh-token branch from 1a8d72d to 03798ef Compare June 22, 2022 17:53

11xor6 reviewed Jun 22, 2022

View reviewed changes

11xor6 approved these changes Jun 22, 2022

View reviewed changes

Praveen2112 force-pushed the refresh-token branch 3 times, most recently from a63d897 to a5446b9 Compare June 23, 2022 09:54

lukasz-walkiewicz approved these changes Jun 23, 2022

View reviewed changes

Praveen2112 force-pushed the refresh-token branch 6 times, most recently from 62c8c0a to 4c07b9f Compare June 24, 2022 14:28

s2lomon and others added 3 commits June 27, 2022 16:15

Add product test for refresh-token flow

977756b

Add refresh token to Web-UI

56ad6c5

Co-authored-by: aczajkowski <[email protected]>

Praveen2112 force-pushed the refresh-token branch from 4c07b9f to 56ad6c5 Compare June 27, 2022 10:46

Praveen2112 approved these changes Jun 27, 2022

View reviewed changes

Praveen2112 merged commit b3d03cd into trinodb:master Jun 27, 2022

github-actions bot added this to the 388 milestone Jun 27, 2022

colebow mentioned this pull request Jun 27, 2022

Add Trino 388 release notes #13004

Merged

Praveen2112 mentioned this pull request Jun 28, 2022

Add docs for refresh-token support to OAuth2 #13009

Merged

	return new TokenPair(tokens.getAccessToken(), tokens.getRefreshToken());
	return accessAndRefreshTokens(tokens.getAccessToken(), tokens.getRefreshToken());

	return new TokenPair(accessToken, Optional.of(requireNonNull(refreshToken, "refreshToken is null")));
	return new TokenPair(accessToken, Optional.ofNullable(refreshToken));

	public static TokenPair accessAndRefreshTokens(String accessToken, String refreshToken)
	public static TokenPair accessAndRefreshTokens(String accessToken, @Nullable String refreshToken)

	protected AuthenticationException needAuthentication(ContainerRequestContext request, String token, String message)
	protected AuthenticationException needAuthentication(ContainerRequestContext request, String currentToken, String message)

	throw new IllegalArgumentException("Token has timed out", ex);
	throw new IllegalArgumentException("Token has expired", ex);

	throw new IllegalStateException("Redirects has been disabled are not allowed");
	throw new UnsupportedOperationException("Redirect has been disabled as it's no longer allowed");

		catch (ExpiredJwtException ex) {
		throw new IllegalArgumentException("Token has timed out", ex);


		import static com.google.common.base.Strings.isNullOrEmpty;

		public class SecretKeyConfig


		import static java.util.Objects.requireNonNull;

		public class JweSecretKeysProvider


		import static io.airlift.configuration.ConfigBinder.configBinder;

		public class JWETokenSerializerModule

Add refresh-token support to OAuth2 #12664

Add refresh-token support to OAuth2 #12664

Conversation

s2lomon commented Jun 2, 2022 • edited by Praveen2112 Loading

Description

Related issues, pull requests, and links

Documentation

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

11xor6 left a comment

Choose a reason for hiding this comment

11xor6 Jun 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

11xor6 Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s2lomon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s2lomon commented Jun 2, 2022 •

edited by Praveen2112

Loading

11xor6 Jun 2, 2022 •

edited

Loading

11xor6 Jun 7, 2022 •

edited

Loading