Codestin Search App

dimberman · 2020-10-25T16:42:45Z

This PR creates a one-stop document for users looking to update to Airflow 2.0. This ensures that instead of needing to read through the entire changelog, these users can follow a simple set of steps to prepare for the upgrade.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

dimberman · 2020-10-25T16:43:04Z

cc: @vikramkoka @paolaperaza

yangrong688

Hi, @dimberman , I found some formatting issues and need to be fixed.

UPDATING_TO_2.0.md

houqp · 2020-10-26T19:15:31Z

I remember we are unifying roles in RBAC between web UI and stable API, will that be managed automatically without manual actions from user?

dimberman · 2020-10-26T19:36:03Z

#11842 (comment)

cc: @jhtimmins for this question

ryw · 2020-10-27T14:33:02Z

UPDATING.md

not sure we should include "a1" here

paolaperaza

@dimberman I just took a first pass at this! Feel free to take or leave suggestions - my comments are mostly copy/grammar/formatting related 😊

Thanks so much for putting this together - a lotta stuff in here! Appreciate it.

paolaperaza · 2020-10-27T14:54:11Z

UPDATING_TO_2.0.md

@dimberman Should we change this header to # Upgrade to Airflow 2.0+? It's in the file name, but might be good to reiterate in this H1 here as well.

paolaperaza · 2020-10-27T14:56:46Z

UPDATING_TO_2.0.md

Airflow 2.0.0 will only support Python 3.6 and up. > Airflow 2.0.0 will require Python 3.6+.

paolaperaza · 2020-10-27T14:57:34Z

UPDATING_TO_2.0.md

Should we link to info on this Operator?

paolaperaza · 2020-10-27T14:58:34Z

UPDATING_TO_2.0.md

I presume we wanted to specify versions here!

For a list of breaking changes between python 2 and python > For a list of breaking changes between Python 2 and Python 3, please refer to this handy blog from the CouchBaseDB team.

paolaperaza · 2020-10-27T15:10:20Z

UPDATING_TO_2.0.md

To minimize upgrade friction for Airflow 1.10, we are releasing a "bridge" release that has multiple critical features that can prepare users to upgrade without disrupting existing workflows.

-->

To minimize friction for users upgrading from Airflow 1.10 to Airflow 2.0 and beyond, a "bridge" release and final 1.10 version will be made available. Airflow 1.10.13 includes support for various critical features that make it easy for users to test DAGs and make sure they're Airflow 2.0 compatible without forcing breaking changes and disrupting existing workflows. We strongly recommend that all users upgrading to Airflow 2.0 first upgrade to Airflow 1.10.13.

Features in 1.10.13 include:

paolaperaza · 2020-10-27T16:32:36Z

UPDATING_TO_2.0.md

Add link to Stable API reference doc?

paolaperaza · 2020-10-27T16:33:01Z

UPDATING_TO_2.0.md

Exception from DAG callbacks used to crash scheduler. In order to make
scheduler more robust, we have changed this behavior to log the exception
instead. On top of that, a new dag.callback_exceptions counter metric has
been added to help better monitor callback exceptions.

-->

Exception from DAG callbacks used to crash the Airflow Scheduler. As part of our efforts to make the Scheduler more performant and reliable, we have changed this behavior to log the exception
instead. On top of that, a new dag.callback_exceptions counter metric has
been added to help better monitor callback exceptions.

paolaperaza · 2020-10-27T16:34:15Z

UPDATING_TO_2.0.md

--> ### Airflow CLI changes in 2.0

UPDATING_TO_2.0.md

paolaperaza · 2020-10-27T16:35:28Z

UPDATING_TO_2.0.md

This could be its own section, for example

turbaszek · 2020-10-27T16:45:27Z

UPDATING.md

Does deleting this records mean that we won't use UPDATING.md from 2.0 onward?

I think that once 2.0 comes out we should create a new UPDATING.md and maybe move UPDATING to "UPDATING-1.0.md"

UPDATING_TO_2.0.md

potiuk · 2020-10-27T16:30:55Z

UPDATING_TO_2.0.md

Those should be in "Appendix" or "reference" section IMHO. or maybe steps should be grouped under the "Migration steps" .

@potiuk good point. I added an extra level to the TOC and set up an appendix segment. We can even make the primary segments short and link to appendix as needed.

UPDATING_TO_2.0.md

potiuk · 2020-10-27T16:35:10Z

UPDATING_TO_2.0.md

Suggested change

airflow generate_pod_template -o <output file path>

```shell script

airflow generate_pod_template -o <output file path>

` ``

can't add ``` :(

potiuk · 2020-10-27T16:37:02Z

UPDATING_TO_2.0.md

Same here ```shell script would be nice

potiuk · 2020-10-27T17:03:39Z

UPDATING_TO_2.0.md

Suggested change

Note that this backport-provider is *not* the same as the actual provider in 2.0. We have tested these operators for backwards

Note that the backport provider packages are just backports of the provider packages compatible with Airflow 2.0. Those provider packages are installed automatically when you install airflow with extras. For example

pip install airflow[docker]

automatically installs the `apache-airflow-providers-docker` package.

But you can manage/upgrade remove provider packages separately from the airflow core.

potiuk · 2020-10-27T17:11:16Z

Some initial review

potiuk · 2020-10-27T17:11:32Z

(did not have yet time to do it all)

ryw · 2020-10-27T17:17:59Z

@dimberman I'd say address comments then get this merged today, then everyone can improve sections in smaller PRs.

dimberman

@ryw good call. I'll address comments, merge today, and we can clean up in smaller PRs.

dimberman · 2020-10-27T19:55:07Z

@potiuk @ashb @kaxil I'm going by @ryw 's suggestion and merging this first draft to prevent this from becoming a monstrous PR for documentation that we need out ASAP to get people on the right track. Over the next few days we can continue PRing against this document and get it to where we want it.

paolaperaza · 2020-10-27T20:09:06Z

@dimberman Awesome, I'll followup with another PR then! Thanks.

* Separate 2.0 upgrade steps into their own file * shell script * more changes

boring-cyborg bot added the area:dev-tools label Oct 25, 2020

dimberman requested review from ashb, kaxil and potiuk October 25, 2020 16:42

yangrong688 reviewed Oct 26, 2020

View reviewed changes

UPDATING_TO_2.0.md Outdated Show resolved Hide resolved

yangrong688 reviewed Oct 26, 2020

View reviewed changes

UPDATING_TO_2.0.md Outdated Show resolved Hide resolved

dimberman force-pushed the updating-2.0-docs branch from a93eb94 to 286cf50 Compare October 26, 2020 22:34

ryw self-requested a review October 27, 2020 14:31

ryw approved these changes Oct 27, 2020

View reviewed changes

UPDATING.md Outdated

Copy link

Member

ryw Oct 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure we should include "a1" here

paolaperaza reacted with thumbs up emoji

paolaperaza reviewed Oct 27, 2020

View reviewed changes

turbaszek reviewed Oct 27, 2020

View reviewed changes

potiuk reviewed Oct 27, 2020

View reviewed changes

Separate 2.0 upgrade steps into their own file

a533a37

dimberman force-pushed the updating-2.0-docs branch from 018b349 to a533a37 Compare October 27, 2020 19:45

dimberman commented Oct 27, 2020

View reviewed changes

dimberman added 2 commits October 27, 2020 12:49

shell script

4cfb3a9

more changes

60566c4

dimberman merged commit c204a2e into apache:master Oct 27, 2020

dimberman deleted the updating-2.0-docs branch October 27, 2020 19:54

michalmisiewicz pushed a commit to michalmisiewicz/airflow that referenced this pull request Oct 30, 2020

Updating 2.0 docs (apache#11842)

2cf29b2

* Separate 2.0 upgrade steps into their own file * shell script * more changes

szn pushed a commit to szn/airflow that referenced this pull request Nov 1, 2020

Updating 2.0 docs (apache#11842)

972f597

* Separate 2.0 upgrade steps into their own file * shell script * more changes

-        airflow generate_pod_template -o <output file path>
+```shell script
+        airflow generate_pod_template -o <output file path>
+` ``

-Note that this backport-provider is *not* the same as the actual provider in 2.0. We have tested these operators for backwards
+Note that the backport provider packages are just backports of the provider packages compatible with Airflow 2.0. Those provider packages are installed automatically when you install airflow with extras. For example
+    pip install airflow[docker]
+   automatically installs the `apache-airflow-providers-docker` package.
+But you can manage/upgrade remove provider packages separately from the airflow core.

Conversation

dimberman commented Oct 25, 2020

Uh oh!

dimberman commented Oct 25, 2020

Uh oh!

yangrong688 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

houqp commented Oct 26, 2020

Uh oh!

dimberman commented Oct 26, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paolaperaza left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

potiuk commented Oct 27, 2020

Uh oh!

potiuk commented Oct 27, 2020

Uh oh!

ryw commented Oct 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimberman left a comment

Choose a reason for hiding this comment

Uh oh!

dimberman commented Oct 27, 2020

Uh oh!

paolaperaza commented Oct 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ryw commented Oct 27, 2020 •

edited

Loading