Codestin Search App

cruseakshay · 2026-01-16T16:40:10Z

Fix unclosed aiohttp.ClientSession and TCPConnector warnings when using AzureDataFactoryRunPipelineOperator with deferrable=True.

The AzureDataFactoryAsyncHook creates an AsyncDataFactoryManagementClient but never closes it, causing resource leak errors in logs regardless of pipeline success/failure.

Changes

Add __aenter__, __aexit__, and close() methods to AzureDataFactoryAsyncHook
Update triggers to use async with hook: for proper cleanup
Add unit tests for context manager behavior

Follows the same pattern as #52119 (Databricks fix).

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude following the guidelines

SameerMesiah97

Looks good to me. Turning the async hook into a context manager and updating the trigger call sites fixes the aiohttp client leak cleanly. Tests look solid as well.

Not a blocker but I would suggest double checking if there any places outside this file where this hook is being called so that the leak does not persist after your fix.

cruseakshay · 2026-01-19T13:09:35Z

@SameerMesiah97 Double checked other places, looks good.

dabla

2 small nits but looking good!

dabla

Looking good to me.

dabla · 2026-01-22T16:01:35Z

@cruseakshay Did you actually tests this changes and can you confirm this works?

dabla · 2026-01-22T16:04:13Z

@cruseakshay Also, the changes made in the data_factory trigger, like the added _build_trigger_event method like I asked, would be nice to see a test for it as well. Is the trigger data_factory tests also testing the context manger?

cruseakshay · 2026-01-23T05:06:16Z

@cruseakshay Did you actually tests this changes and can you confirm this works?

Yes, here are trail logs from latest run

[2026-01-23T04:53:13.235944Z] INFO - Sleeping for 60. The pipeline state is Queued.
[2026-01-23T04:54:13.245821Z] INFO - Request URL: 'https://management.azure.com/subscriptions/REDACTED/resourceGroups/airflow-adf-test-rg/providers/Microsoft.DataFactory/factories/airflow-adf-test/pipelineruns/REDACTED?api-version=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '9001a594-f817-11f0-b40d-b64b850643a2'
    'User-Agent': 'azsdk-python-mgmt-datafactory/9.2.0 Python/3.12.12 (Linux-6.12.54-linuxkit-aarch64-with-glibc2.36)'
    'Authorization': 'REDACTED'
No body was attached to the request
[2026-01-23T04:54:13.973572Z] INFO - Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '816'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-operation-identifier': 'REDACTED'
    'x-ms-ratelimit-remaining-subscription-reads': '249'
    'x-ms-ratelimit-remaining-subscription-global-reads': 'REDACTED'
    'x-ms-request-id': '59f3995c-910c-4abe-952a-5e3e8fa3bff3'
    'x-ms-routing-request-id': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'X-Cache': 'REDACTED'
    'X-MSEdge-Ref': 'Ref A: 1C93C26BC61B478A8D2F48098F9A042C Ref B: MAA201060513029 Ref C: 2026-01-23T04:54:13Z'
    'Date': 'Fri, 23 Jan 2026 04:54:13 GMT'
[2026-01-23T04:54:13.974417Z] INFO - Trigger fired eventname=test_adf_trigger_fix/manual__2026-01-23T04:53:02.479036+00:00/run_pipeline_deferrable/-1/1 (ID 1) result=TriggerEvent<{'status': 'success', 'message': 'The pipeline run 66bb30ba-f817-11f0-bec5-16db0266592f has Succeeded.', 'run_id': '66bb30ba-f817-11f0-bec5-16db0266592f'}> 
[2026-01-23T04:54:13.975233Z] INFO - trigger completedname=test_adf_trigger_fix/manual__2026-01-23T04:53:02.479036+00:00/run_pipeline_deferrable/-1/1 (ID 1) 
[2026-01-23T04:54:15.380187Z] INFO - DAG bundles loaded: dags-folder
[2026-01-23T04:54:15.380487Z] INFO - Filling up the DagBag from /usr/local/airflow/dags/test_adf_trigger_fix.py
[2026-01-23T04:54:15.488568Z] INFO - TaskInstance Details: dag_id=test_adf_trigger_fix task_id=run_pipeline_deferrable dagrun_id=manual__2026-01-23T04:53:02.479036+00:00 map_index=-1 run_start_date=2026-01-23T04:54:15.366708Z try_number=1 op_classpath=["airflow.providers.microsoft.azure.operators.data_factory.AzureDataFactoryRunPipelineOperator"] 
[2026-01-23T04:54:15.488765Z] INFO - The pipeline run 66bb30ba-f817-11f0-bec5-16db0266592f has Succeeded.
[2026-01-23T04:54:15.508872Z] WARNING - Connection schemes (type: azure_data_factory) shall not contain '_' according to RFC3986.

cruseakshay · 2026-01-23T05:38:01Z

@cruseakshay Also, the changes made in the data_factory trigger, like the added _build_trigger_event method like I asked, would be nice to see a test for it as well. Is the trigger data_factory tests also testing the context manger?

hook-level test that covers the context manager behavior: AzureDataFactoryAsyncHook has test_context_manager_calls_close that explicitly verifies async with hook: calls close(). Haven't seen any trigger tests explicitly assert __aenter__/__aexit__ calls.

dabla · 2026-01-27T06:58:48Z

@cruseakshay Thanks for the PR and your patience, good job!

boring-cyborg · 2026-01-27T06:58:58Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…he#60650) * Fix unclosed aiohttp ClientSession in AzureDataFactoryAsyncHook * If cancel_pipeline_run() creates a new connection * Address review comments * review: early return * add tests: build_trigger_event --------- Co-authored-by: Akshay <[email protected]>

Fix unclosed aiohttp ClientSession in AzureDataFactoryAsyncHook

7fd3d09

boring-cyborg Bot added area:providers provider:microsoft-azure Azure-related issues labels Jan 16, 2026

Merge branch 'main' into fix/adf-unclosed-aiohttp-session

c0f86c2

cruseakshay marked this pull request as ready for review January 16, 2026 16:42

cruseakshay requested a review from dabla as a code owner January 16, 2026 16:42

cruseakshay mentioned this pull request Jan 16, 2026

Unclosed aiohttp ClientSession and TCPConnector after AzureDataFactoryRunPipelineOperator execution #55724

Closed

2 tasks

SameerMesiah97 reviewed Jan 16, 2026

View reviewed changes

cruseakshay and others added 2 commits January 17, 2026 12:46

If cancel_pipeline_run() creates a new connection

ceb3634

Merge branch 'main' into fix/adf-unclosed-aiohttp-session

508ef44

dabla reviewed Jan 19, 2026

View reviewed changes

Address review comments

fc9ec16

cruseakshay requested a review from dabla January 20, 2026 04:05

Merge branch 'main' into fix/adf-unclosed-aiohttp-session

f4fe40e

dabla reviewed Jan 20, 2026

View reviewed changes

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Outdated

Comment thread providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py Outdated

cruseakshay and others added 2 commits January 22, 2026 09:44

Merge branch 'main' into fix/adf-unclosed-aiohttp-session

936224e

review: early return

054d2d3

dabla approved these changes Jan 22, 2026

View reviewed changes

add tests: build_trigger_event

73ea68a

dabla merged commit 033d185 into apache:main Jan 27, 2026
89 checks passed

vincbeck mentioned this pull request Jan 28, 2026

Status of testing Providers that were prepared on January 28, 2026 #61165

Closed

85 tasks

Conversation

cruseakshay commented Jan 16, 2026

Changes

Was generative AI tooling used to co-author this PR?

Uh oh!

SameerMesiah97 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cruseakshay commented Jan 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dabla left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dabla left a comment

Choose a reason for hiding this comment

Uh oh!

dabla commented Jan 22, 2026

Uh oh!

dabla commented Jan 22, 2026

Uh oh!

cruseakshay commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cruseakshay commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dabla commented Jan 27, 2026

Uh oh!

Uh oh!

boring-cyborg Bot commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SameerMesiah97 left a comment •

edited

Loading

cruseakshay commented Jan 23, 2026 •

edited

Loading

cruseakshay commented Jan 23, 2026 •

edited

Loading