Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Reject async callables in @task.virtualenv and @task.external_python - #73081

Draft
FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-virtualenv-operators-reject-async-callable
Draft

FrankYang0529 wants to merge 1 commit into
apache:mainfrom
FrankYang0529:airflow-virtualenv-operators-reject-async-callable

Conversation

@FrankYang0529

Copy link
Copy Markdown
Member

Why

  • PythonVirtualenvOperator uses a virtualenv built from requirements, and ExternalPythonOperator uses the interpreter given in python. An async def callable never reaches execute_callable(), so requirements, python, and venv_cache_path are ignored.
  • Either the callable runs with the worker's own packages and the task still succeeds, or it fails with an ImportError that looks like a broken virtualenv.

How

  • _BasePythonVirtualenvOperator.__init__ raises ValueError for async callables. The Dag now fails at parse time.

Verification

  • Unit test: uv run --frozen --project providers/standard pytest providers/standard/tests/unit/standard/operators/test_python.py
  • Integration test:
  1. Setup
export DEMO_DIR="${TMPDIR:-/tmp}/airflow-async-venv-demo"
export AIRFLOW_HOME="$DEMO_DIR/airflow-home"
export AIRFLOW__CORE__DAGS_FOLDER="$DEMO_DIR/dags"
export AIRFLOW__CORE__LOAD_EXAMPLES=False
mkdir -p "$AIRFLOW__CORE__DAGS_FOLDER"
uv run --frozen --no-dev --project providers/standard --with "pandas>=3,<4" airflow db migrate
uv venv --allow-existing "$DEMO_DIR/pandas2-venv"
uv pip install --python "$DEMO_DIR/pandas2-venv/bin/python" "pandas>=2.2,<3"
  1. Create Dags
cat > "$AIRFLOW__CORE__DAGS_FOLDER/virtualenv_example.py" <<'EOF'
from __future__ import annotations

from airflow.sdk import dag, task


@dag(schedule=None)
def virtualenv_example():
    @task.virtualenv(requirements=["pandas>=2.2,<3"], system_site_packages=False)
    def virtualenv_sync():
        import sys

        import pandas as pd

        return {"python": sys.executable, "pandas": pd.__version__}

    @task.virtualenv(requirements=["pandas>=2.2,<3"], system_site_packages=False)
    async def virtualenv_async():
        import sys

        import pandas as pd

        return {"python": sys.executable, "pandas": pd.__version__}

    virtualenv_sync()
    virtualenv_async()


example_dag = virtualenv_example()

if __name__ == "__main__":
    dag_run = example_dag.test()
    for ti in sorted(dag_run.get_task_instances(), key=lambda ti: ti.task_id):
        print(f"EXAMPLE STATE {ti.task_id} {ti.state}")
EOF

cat > "$AIRFLOW__CORE__DAGS_FOLDER/external_python_example.py" <<'EOF'
from __future__ import annotations

import os

from airflow.sdk import dag, task

PANDAS2_PYTHON = os.path.join(os.environ["DEMO_DIR"], "pandas2-venv", "bin", "python")


@dag(schedule=None)
def external_python_example():
    @task.external_python(python=PANDAS2_PYTHON)
    def external_python_sync():
        import sys

        import pandas as pd

        return {"python": sys.executable, "pandas": pd.__version__}

    @task.external_python(python=PANDAS2_PYTHON)
    async def external_python_async():
        import sys

        import pandas as pd

        return {"python": sys.executable, "pandas": pd.__version__}

    external_python_sync()
    external_python_async()


example_dag = external_python_example()

if __name__ == "__main__":
    dag_run = example_dag.test()
    for ti in sorted(dag_run.get_task_instances(), key=lambda ti: ti.task_id):
        print(f"EXAMPLE STATE {ti.task_id} {ti.state}")
EOF
  1. Run dags to check result

On main branch:

uv run --frozen --no-dev --project providers/standard --with "pandas>=3,<4" python "$AIRFLOW__CORE__DAGS_FOLDER/virtualenv_example.py" 2>&1 | grep -oE "running task <TaskInstance: [^ ]+|Returned value was: \{[^}]*\}|EXAMPLE STATE .*"
uv run --frozen --no-dev --project providers/standard --with "pandas>=3,<4" python "$AIRFLOW__CORE__DAGS_FOLDER/external_python_example.py" 2>&1 | grep -oE "running task <TaskInstance: [^ ]+|Returned value was: \{[^}]*\}|EXAMPLE STATE .*"

All four tasks succeed. The sync tasks report pandas 2 from the virtualenv or the external environment. The async tasks report pandas 3 from the uv run environment, which shows they ignored requirements and python:

running task <TaskInstance: virtualenv_example.virtualenv_sync
Returned value was: {'python': '.../venvymv4y504/bin/python', 'pandas': '2.3.3'}
running task <TaskInstance: virtualenv_example.virtualenv_async
Returned value was: {'python': '.../builds-v0/.tmpDC6uaL/bin/python', 'pandas': '3.0.5'}
EXAMPLE STATE virtualenv_sync success
EXAMPLE STATE virtualenv_async success
running task <TaskInstance: external_python_example.external_python_sync
Returned value was: {'python': '.../airflow-async-venv-demo/pandas2-venv/bin/python', 'pandas': '2.3.3'}
running task <TaskInstance: external_python_example.external_python_async
Returned value was: {'python': '.../builds-v0/.tmp90SjJP/bin/python', 'pandas': '3.0.5'}
EXAMPLE STATE external_python_sync success
EXAMPLE STATE external_python_async success

On this branch, both Dags now fail at parse time.

uv run --frozen --no-dev --project providers/standard --with "pandas>=3,<4" python "$AIRFLOW__CORE__DAGS_FOLDER/virtualenv_example.py" 2>&1 | grep ValueError
uv run --frozen --no-dev --project providers/standard --with "pandas>=3,<4" python "$AIRFLOW__CORE__DAGS_FOLDER/external_python_example.py" 2>&1 | grep ValueError
    raise ValueError(
ValueError: _PythonVirtualenvDecoratedOperator does not support async functions as python_callable. Call asyncio.run() inside a regular function instead.
    raise ValueError(
ValueError: _PythonExternalDecoratedOperator does not support async functions as python_callable. Call asyncio.run() inside a regular function instead.

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@Vamsi-klu Vamsi-klu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants