Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fill DagBag from dag_folder setting for upgrade rules#14588

Merged
kaxil merged 9 commits intoapache:v1-10-stablefrom
kenfinster:rules
Mar 9, 2021
Merged

Fill DagBag from dag_folder setting for upgrade rules#14588
kaxil merged 9 commits intoapache:v1-10-stablefrom
kenfinster:rules

Conversation

@kenfinster
Copy link

For some reason:
BaseOperatorMetaclassRule
NoAdditionalArgsInOperatorsRule
Don't consider the dag folder set by the user. This causes issues when the user needs to set the dag folder with environment variable.

This fix makes the rules to take the path from the settings.

@kaxil following the chat on slack.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg
Copy link

boring-cyborg bot commented Mar 3, 2021

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@kenfinster kenfinster changed the title Rules Fill DagBag from dag_folder setting for upgrade rules Mar 3, 2021
@ephraimbuddy ephraimbuddy added the upgrade-check upgrade-check CLI label Mar 4, 2021
@github-actions
Copy link

github-actions bot commented Mar 4, 2021

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Mar 4, 2021
@kenfinster
Copy link
Author

@kaxil @ephraimbuddy @XD-DENG can you help with the tests? The failures are on rules that I did not change.

tests/upgrade/rules/test_chain_between_dag_and_operator_not_allowed_rule.py:135: IndexError
tests/upgrade/rules/test_import_changes.py:96: IndexError

Also I don't understand why it passes for postgres 13 + py 3.7 but not for postgres 13 + py 2.7

@XD-DENG
Copy link
Member

XD-DENG commented Mar 5, 2021

Hi @kenfinster , most likely the error has nothing to do with your changes, because the same errors already exist in the latest commit in branch v1-10-stable (https://github.com/apache/airflow/commits/v1-10-stable), which is #14493

But on the other hand, the most recent change you made in 4b7f4a7 is not making sense to me. I guess you were making a random try? Otherwise may you please clarify it? Thanks.

@kenfinster
Copy link
Author

@XD-DENG There was a failure with:

________________ TestNoAdditionalArgsInOperatorsRule.test_check ________________

self = <tests.upgrade.rules.test_no_additional_args_operators.TestNoAdditionalArgsInOperatorsRule testMethod=test_check>

    @pytest.mark.filterwarnings('always')
    def test_check(self):
        rule = NoAdditionalArgsInOperatorsRule()
    
        assert isinstance(rule.title, str)
        assert isinstance(rule.description, str)
    
        with NamedTemporaryFile(mode='w', suffix='.py') as dag_file:
            dag_file.write(
                textwrap.dedent('''
            from airflow import DAG
            from airflow.utils.dates import days_ago
            from airflow.operators.bash_operator import BashOperator
    
            with DAG(dag_id="test_no_additional_args_operators", start_date=days_ago(0)):
                BashOperator(task_id='test', bash_command="true", extra_param=42)
                '''))
            dag_file.flush()
            msgs = list(rule.check(dags_folder=dag_file.name))
>           assert len(msgs) == 1
E           AssertionError: assert 0 == 1
E            +  where 0 = len([])

tests/upgrade/rules/test_no_additional_args_operators.py:49: AssertionError

So i change the assertion. Thinking of it again maybe better way is to remove this test all together. It tries to mock a dag bag but this PR makes it redundant.

@ephraimbuddy
Copy link
Contributor

Hi @kenfinster , most likely the error has nothing to do with your changes, because the same errors already exist in the latest commit in branch v1-10-stable (https://github.com/apache/airflow/commits/v1-10-stable), which is #14493

But on the other hand, the most recent change you made in 4b7f4a7 is not making sense to me. I guess you were making a random try? Otherwise may you please clarify it? Thanks.

@kenfinster Can you remove your last commit 4b7f4a7. This is a known issue(see here #14493) which I think should be fixed in a separate PR.

@kenfinster
Copy link
Author

@ephraimbuddy done.
is there anything else needed?

@kaxil
Copy link
Member

kaxil commented Mar 9, 2021

@kenfinster I have rebased your PR on latest (after merging #14678 and #14680) v1-10-stable -- hopefully the CI failures should be fixed.

@github-actions
Copy link

github-actions bot commented Mar 9, 2021

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.

@kaxil kaxil merged commit 577c82d into apache:v1-10-stable Mar 9, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 9, 2021

Awesome work, congrats on your first merged pull request!

@ashb
Copy link
Member

ashb commented Mar 10, 2021

@kenfinster How were you setting the dag folder? I was unable to reproduce this behaviour with Airflow 1.10.14 and upgrade-check 1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full tests needed We need to run full set of tests for this PR to merge upgrade-check upgrade-check CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants