Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Code smell fixes for BackfillJob#12005

Merged
XD-DENG merged 1 commit intoapache:masterfrom
XD-DENG:code-smell-fix-backfilljob
Nov 1, 2020
Merged

Code smell fixes for BackfillJob#12005
XD-DENG merged 1 commit intoapache:masterfrom
XD-DENG:code-smell-fix-backfilljob

Conversation

@XD-DENG
Copy link
Member

@XD-DENG XD-DENG commented Oct 31, 2020

Three change items in two parts:

Part 1

  • merge IFs when we can;
  • One of the IF condition checks should reuse STATES_COUNT_AS_RUNNING which has been already defined;

Part 2

  • Remove unused parameter for intermediate function _per_task_process()

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Oct 31, 2020
@XD-DENG XD-DENG requested review from ashb, kaxil and turbaszek October 31, 2020 19:33
XD-DENG
XD-DENG commented Oct 31, 2020
View reviewed changes
airflow/jobs/backfill_job.py Outdated
Copy link
Member Author

@XD-DENG XD-DENG Oct 31, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification for the 2nd part: because we have this line, actually whatever we pass to task is ignored

ashb
ashb approved these changes Oct 31, 2020
View reviewed changes
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming tests pass LGTM

@github-actions
Copy link

github-actions bot commented Oct 31, 2020

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Oct 31, 2020
@XD-DENG
Copy link
Member Author

XD-DENG commented Oct 31, 2020

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

Thanks @ashb . May you please advise any specific thing I should address for this step?

It can also help me understand how I can review/process other contributors' PRs later. Cheers!

@ashb
Copy link
Member

ashb commented Oct 31, 2020

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

Thanks @ashb . May you please advise any specific thing I should address for this step?

It can also help me understand how I can review/process other contributors' PRs later. Cheers!

See #12006

@XD-DENG
Code smell fixes for BackfillJob
b37eb8d 
- merge IFs when we can;
- One of the IF condition check should reuse STATES_COUNT_AS_RUNNING which has been already defined;
- Remove unused parameter for intermediate function _per_task_process()
@XD-DENG XD-DENG force-pushed the code-smell-fix-backfilljob branch from 2cb1b2f to b37eb8d Compare November 1, 2020 09:44
@potiuk
Copy link
Member

potiuk commented Nov 1, 2020

Yeah! @XD-DENG -> it worked :) after rebase you run the full matrix now. I'd love to explain it in a clearer way, but It's not easy :). Happy if you could all help in describing it (and maybe make the comments) a little more obvious what to do!

@XD-DENG
Copy link
Member Author

XD-DENG commented Nov 1, 2020

Yeah! @XD-DENG -> it worked :) after rebase you run the full matrix now. I'd love to explain it in a clearer way, but It's not easy :). Happy if you could all help in describing it (and maybe make the comments) a little more obvious what to do!

Thanks @potiuk . Yes I'm also making use of this PR to explore how the new PR test/review process works. Earlier I tried another option ("re-run workflow" as committer), but seems not working correctly. I expect some minor adjustments would be needed to the process, will share with you once I find any.

@potiuk
Copy link
Member

potiuk commented Nov 1, 2020

Thanks @potiuk . Yes I'm also making use of this PR to explore how the new PR test/review process works. Earlier I tried another option ("re-run workflow" as committer), but seems not working correctly. I expect some minor adjustments would be needed to the process, will share with you once I find any.

Yep. I am on a lookout for any adjustments and fixes needed :). Happy to get feedback.

@XD-DENG XD-DENG merged commit e032301 into apache:master Nov 1, 2020
@XD-DENG XD-DENG deleted the code-smell-fix-backfilljob branch November 1, 2020 12:01
@potiuk
Copy link
Member

potiuk commented Nov 1, 2020

@XD-DENG -> the "re-running" problem might be fixed when we merge this one: #12018 . There was a scenario that re-running an old run caused self-cancelling of the Image Build job :) . I've added self-preservation mechanisms to prevent this from happening.

szn pushed a commit to szn/airflow that referenced this pull request Nov 1, 2020
@XD-DENG
Code smell fixes for BackfillJob (apache#12005)
926efae 
- merge IFs when we can;
- One of the IF condition check should reuse STATES_COUNT_AS_RUNNING which has been already defined;
- Remove unused parameter for intermediate function _per_task_process()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashb ashb ashb approved these changes

@kaxil kaxil kaxil approved these changes

@turbaszek turbaszek Awaiting requested review from turbaszek

Assignees

No one assigned

Labels

area:Scheduler including HA (high availability) scheduler full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@XD-DENG @ashb @potiuk @kaxil