Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Comments

Unified project run modes#72

Merged
mingyu89 merged 4 commits intodatabricks:mainfrom
mingyu89:unify-run-modes
Mar 29, 2023
Merged

Unified project run modes#72
mingyu89 merged 4 commits intodatabricks:mainfrom
mingyu89:unify-run-modes

Conversation

@mingyu89
Copy link
Contributor

@mingyu89 mingyu89 commented Mar 22, 2023

Followed option2 in doc https://docs.google.com/document/d/1PO49MfICLpDpCXk5jFq_tcuPbocutotkui30jxSXLXQ/edit#

Tested by adding two projects(receipes & fs) to CUJ repo to make it a monorepo.
The staging and prod workspace workflows succeeded in training, validation, deploy.

@mingyu89 mingyu89 marked this pull request as draft March 24, 2023 22:50
# Parameters:
#
# * env (optional): Name of the environment the notebook is run in (dev, staging, or prod). Defaults to "dev".
# * env (optional): Name of the environment the notebook is run in (test, staging, or prod).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not leave dev here? It will be used when the notebook is triggered manually, right?

Copy link
Contributor Author

@mingyu89 mingyu89 Mar 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are getting the env field without defining the field in the widget, which means only terraform jobs or github actions will be able to set the env parameter.

If users directly run the notebook, they will directly use the dev profile without being able to set the env field.

Copy link
Contributor

@vladimirk-db vladimirk-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please test the train/deploy CUJ to make sure nothing is broken?

Also, how will this affect existing users? E.g., we are changing experiment and model names - I assume existing setups won't be broken, can you confirm please?

@mingyu89
Copy link
Contributor Author

mingyu89 commented Mar 28, 2023

@vladimirk-db Thanks. The testing in CUJ is still in progress.
I'm adding two projects to CUJ repo : with fs and without fs

I went through several issues while setting up the monorepo.

  • One is Move pytest root to the internal project directory #74,
  • One is ML-30083.
  • Another is the resource group, storage account name and key in terraform file provider.tf. I think we can skip this one since bundles will resolve it. Temporary fix in CUJ is to use the same resource group, storage account name as CUJ project and use a different key.
  • dbutils.secrets.get in TriggerModelDeploy should use the key from first project.

I've got the terraform deployed to staging and prod. https://github.com/databricks/mlops-azure-cuj/pull/79
Next step is to go through all workflows.

@mingyu89 mingyu89 marked this pull request as ready for review March 28, 2023 06:19
@mingyu89 mingyu89 merged commit 3f04bfd into databricks:main Mar 29, 2023
@mingyu89 mingyu89 deleted the unify-run-modes branch March 29, 2023 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants