Update Fence to Python 3.13 + Run as gen3 user #1312

nss10 · 2025-12-12T21:24:30Z

Link to JIRA ticket if there is one: PD-77

New Features

Breaking Changes

Bug Fixes

Improvements

Replace raw strings in session.execute() with sqlalchemy.text
Update deprecated datetime.utcnow() and datetime.utcfromtimestamp() with 3.13 compatible equivalents
Replace deprecated request attributes with request.payload
Replace deprecated SqlAlchemy's query.get() with Session.get()
Eliminate all boto usage in favor of boto3
Add a local implementation of strtobool since distutils is no longer available since Python 3.12
Add NoAsyncMagicMock since MagicMock returns AsyncMocks even if the source object is partially async
Replace assert <mocked_method>.not_called with <mocked_method>.assert_not_called

Dependency updates

Update Fence to Python 3.13

Deployment changes

Change Docker image to run with gen3 user instead of root

…ocal_settings.py`

…dependency_updates

…update_fence_3_13

coveralls · 2025-12-12T21:29:30Z

Pull Request Test Coverage Report for Build 20358027677

Details

0 of 0 changed or added relevant lines in 0 files are covered.
423 unchanged lines in 17 files lost coverage.
Overall coverage increased (+0.02%) to 75.002%

Files with Coverage Reduction	New Missed Lines	%
blueprints/misc.py	1	88.24%
ea7e1b843f82_optional_client_redirect_uri.py	1	81.82%
resources/audit/client.py	1	77.31%
scripting/google_monitor.py	1	40.17%
a04a70296688_non_unique_client_name.py	2	89.47%
blueprints/oauth2.py	2	76.03%
blueprints/login/ras.py	6	75.47%
blueprints/data/blueprint.py	13	86.9%
resources/ga4gh/passports.py	15	89.13%
9b3a5a7145d7_authlib_update_1_2_1.py	17	60.22%

Totals
Change from base Build 20076693184:	0.02%
Covered Lines:	8440
Relevant Lines:	11253

💛 - Coveralls

github-actions · 2025-12-12T22:46:29Z

Failed to Prepare CI environment

Please find the Github Action logs here

github-actions · 2025-12-12T23:43:29Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	2	2	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_user_token.py	5	0	0	5
tests/test_register_user.py	2	0	0	2
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	1	1	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	65	3	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-15T15:30:14Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	2	2	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_user_token.py	5	0	0	5
tests/test_register_user.py	2	0	0	2
tests/test_oidc_client.py	2	0	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	66	2	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-16T03:40:23Z

filepath	passed	skipped	SUBTOTAL
tests/test_dbgap.py	4	1	5
TOTAL	4	1	5

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-16T04:29:18Z

filepath	passed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	15
tests/test_centralized_auth.py	16	0	16
tests/test_audit_service.py	3	3	6
tests/test_data_upload.py	8	1	9
tests/test_presigned_url.py	7	0	7
tests/test_dbgap.py	4	1	5
tests/test_drs_endpoint.py	4	0	4
tests/test_user_token.py	5	0	5
tests/test_register_user.py	2	0	2
tests/test_oidc_client.py	2	0	2
tests/test_google_data_access.py	1	0	1
tests/test_client_credentials.py	1	0	1
tests/test_ras_authn.py	0	3	3
TOTAL	68	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-16T16:33:35Z

filepath	passed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	15
tests/test_centralized_auth.py	16	0	16
tests/test_audit_service.py	3	3	6
tests/test_data_upload.py	8	1	9
tests/test_presigned_url.py	7	0	7
tests/test_dbgap.py	4	1	5
tests/test_drs_endpoint.py	4	0	4
tests/test_user_token.py	5	0	5
tests/test_register_user.py	2	0	2
tests/test_client_credentials.py	1	0	1
tests/test_oidc_client.py	2	0	2
tests/test_google_data_access.py	1	0	1
tests/test_ras_authn.py	0	3	3
TOTAL	68	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T14:08:41Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_user_token.py	5	0	0	5
tests/test_dbgap.py	2	2	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_register_user.py	2	0	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_ras_authn.py	0	0	3	3
TOTAL	66	2	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T15:20:00Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_user_token.py	5	0	0	5
tests/test_dbgap.py	4	0	1	5
tests/test_register_user.py	1	1	0	2
tests/test_oidc_client.py	2	0	0	2
tests/test_drs_endpoint.py	4	0	0	4
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	67	1	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T18:13:13Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	5	3	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	4	0	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_register_user.py	2	0	0	2
tests/test_user_token.py	5	0	0	5
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_ras_authn.py	0	0	3	3
TOTAL	65	3	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T19:28:28Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	7	1	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	4	0	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_user_token.py	5	0	0	5
tests/test_register_user.py	2	0	0	2
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	67	1	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T21:02:21Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	2	2	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_user_token.py	5	0	0	5
tests/test_register_user.py	2	0	0	2
tests/test_oidc_client.py	2	0	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	66	2	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T21:54:28Z

filepath	passed	failed	error	skipped	SUBTOTAL
tests/test_pfb_export.py	0	0	1	0	1
tests/test_study_viewer.py	0	0	1	0	1
tests/test_dbgap.py	2	2	0	1	5
TOTAL	2	2	2	1	7

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-17T22:28:35Z

filepath	passed	skipped	SUBTOTAL
tests/test_dbgap.py	4	1	5
TOTAL	4	1	5

Please find the detailed integration test report here

Please find the Github Action logs here

…ng change

nss10 · 2025-12-18T14:25:26Z

fence/resources/openid/idp_oauth2.py

            refresh_token=refresh_token,
            expires=expires,
        )
-        current_db_session = db_session.object_session(upstream_refresh_token)


object_session returns None in python 3.13, since the UpstreamRefreshToken was not implicitly attached to any session

github-actions · 2025-12-18T14:30:40Z

Failed to Prepare CI environment

Please find the Github Action logs here

nss10 · 2025-12-18T17:08:02Z

tests/test_app_config.py

        patchers.append(patcher)

+    # create a fresh local app
+    from flask import Flask


This did not throw an error in older versions of fence due to the order in which tests ran.

nss10 · 2025-12-18T17:40:53Z

tests/data/test_data.py

        headers = {"Authorization": "Bearer " + encoded_creds_jwt.jwt}
        response = client.delete("/data/{}".format(did), headers=headers)
        assert response.status_code == 204
-        assert mock_boto_delete.called_once()


Since IndexedFile.delete_files is patched with a MagicMock, we never really hit the core boto delete functionality in the past, since we were patching IndexedFile.delete_files before we reached the core functionality we wanted to test

The tests were passing in the past, because assert .called_once() returned a new MagicMock which is non-falsy in nature, making the assert to pass

nss10 · 2025-12-18T18:01:26Z

tests/data/test_indexed_file.py

            """
            raise Exception("url not available")

-    with patch("fence.blueprints.data.indexd.flask.current_app", return_value=app):


Since flask is coming from a fixture, flask.current_app is already set to app.
Setting this patch, was making everything internally to MagicMock. Removing this patch altogether tests the logic better

nss10 · 2025-12-18T18:29:57Z

tests/ras/test_ras.py

    assert username_to_log_in == username
    iss_sub_pair_to_user_records = db_session.query(IssSubPairToUser).all()
    assert len(iss_sub_pair_to_user_records) == 1
-    iss_sub_pair_to_user = db_session.query(IssSubPairToUser).get((iss, sub))


# LegacyAPIWarning: The Query.get() method is considered legacy as of the 1.x series of SQLAlchemy and becomes a legacy construct in 2.0. The method is now available as Session.get() (deprecated since: 2.0) (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

github-actions · 2025-12-18T18:59:55Z

Failed to Prepare CI environment

Please find the Github Action logs here

github-actions · 2025-12-19T03:21:33Z

filepath	passed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	15
tests/test_centralized_auth.py	16	0	16
tests/test_audit_service.py	3	3	6
tests/test_data_upload.py	8	1	9
tests/test_presigned_url.py	7	0	7
tests/test_user_token.py	5	0	5
tests/test_dbgap.py	4	1	5
tests/test_drs_endpoint.py	4	0	4
tests/test_oidc_client.py	2	0	2
tests/test_client_credentials.py	1	0	1
tests/test_google_data_access.py	1	0	1
tests/test_register_user.py	2	0	2
tests/test_ras_authn.py	0	3	3
TOTAL	68	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

nss10 added 26 commits October 10, 2025 14:54

Update dependencies in fence

4feccd7

Remove Deprecated local_settings.py

44a6eea

Delete references of local_settings and files that are linked to `l…

71102ee

…ocal_settings.py`

Merge branch 'master' of https://github.com/uc-cdis/fence into chore/…

c04e664

…dependency_updates

Merge branch 'master' of https://github.com/uc-cdis/fence into chore/…

ce9c522

…dependency_updates

Update poetry lock

62909d0

Update poetry lock to resolve snyk errors

bb89c64

Add a forced dependency on idna, to overcome a snyk dependency

fc45c35

Initial commit with 3.13 -- failing unit tests due to boto

9b61c79

Update boto to boto3 in cleversafe. Failing dbgap unit tests

e288114

Fix unit tests to work with Python 3.13

ca80e1b

Fix python version in ci

0b326af

Update deprecated distutils

58134c7

Custom implementation of strtobool., instead of using setuptools

674bbbf

Update raw sql statement from str to sq.text

08db93c

Update Dockerfile with 3.13 image

093d288

Revert cov report to include coveralls

1b49239

Merge branch 'master' of https://github.com/uc-cdis/fence into chore/…

c8e62ca

…update_fence_3_13

Update Arborist MagicMock to NoAsyncMagicMock

a59830d

Make changes to remove warnings

7664b57

Run dockerfile with root user

c4ab7b9

Set the final user for fence as gen3 instead of root

9a76456

Add empty line for clarity

01147ee

Set the user back to root

1d8e3a8

parallel image with gen3 user

6c7ea5d

Use custom helm chart to work with gen3 user

61f80d2

nss10 removed the TestDbgap label Dec 16, 2025

nss10 marked this pull request as ready for review December 16, 2025 03:47

nss10 added the TestDbgap label Dec 17, 2025

nss10 removed the TestDbgap label Dec 17, 2025

Update Fence project version -- since running as non-root is a breaki…

ad7e107

…ng change

nss10 changed the title ~~Temp/python 313 gen3 user~~ Update Fence to Python 3.13 + Run as gen3 user Dec 18, 2025

nss10 commented Dec 18, 2025

View reviewed changes

Final updates before making PR ready for review

55c2e3b

Run with HELM master branch

0317dc7

nss10 requested review from Avantol13 and k-burt-uch and removed request for k-burt-uch December 19, 2025 03:24

Update Fence to Python 3.13 + Run as gen3 user #1312

Are you sure you want to change the base?

Update Fence to Python 3.13 + Run as gen3 user #1312

Uh oh!

Conversation

nss10 commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

Uh oh!

coveralls commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 20358027677

Details

💛 - Coveralls

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

github-actions bot commented Dec 16, 2025

Uh oh!

github-actions bot commented Dec 16, 2025

Uh oh!

github-actions bot commented Dec 16, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

nss10 Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nss10 commented Dec 12, 2025 •

edited

Loading

coveralls commented Dec 12, 2025 •

edited

Loading

nss10 Dec 18, 2025 •

edited

Loading

nss10 Dec 18, 2025 •

edited

Loading