From 4ab6d2867370c99dd6310723ecbaa4bcffaa0af8 Mon Sep 17 00:00:00 2001 From: Shobhit Singh Date: Fri, 22 Mar 2024 22:23:51 +0000 Subject: [PATCH 1/5] docs: update bigquery connection documentation --- README.rst | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/README.rst b/README.rst index ad96382df8..082bf90772 100644 --- a/README.rst +++ b/README.rst @@ -241,7 +241,10 @@ and you must be granted the following IAM roles: ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` * Vertex AI User (roles/aiplatform.user) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default - BigQuery connection, or Browser (roles/browser) if using a pre-created connection + BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. + This requirement can be avoided by setting ``skip_bq_connection_check`` option + to ``True``, in which case the connection (default or pre-configured) would be + used as-is without any existence or permission check. ML locations @@ -288,15 +291,22 @@ into `BigQuery remote functions `_ . Creating a remote function in BigQuery DataFrames (See `code samples `_) -creates a BigQuery remote function, a `BigQuery -connection -`_ , -and a `Cloud Functions (2nd gen) function -`_ . +creates: + +1. A `Cloud Functions (2nd gen) function `_. +2. A `BigQuery connection `_ + if the connection does not exist and ``skip_bq_connection_check`` option is + not set to ``True``. You can use a pre-configured BigQuery connection if you + prefer. If the BigQuery connection is created, the BigQuery service will + create a + `Google Cloud–managed IAM service account https://cloud.google.com/bigquery/docs/working-with-connections#before_you_begin`_ + and attach it to the connection. +3. A BigQuery remote function that talks to the cloud function (1) using the BigQuery + connection (2). BigQuery connections are created in the same location as the BigQuery DataFrames session, using the name you provide in the custom function -definition. To view and manage connections, do the following: +definition. To view and manage connections, do the following: 1. Go to `BigQuery in the Google Cloud Console `__. 2. Select the project in which you created the remote function. @@ -341,7 +351,10 @@ following IAM roles: ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` * Storage Object Viewer (roles/storage.objectViewer) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default - BigQuery connection, or Browser (roles/browser) if using a pre-created connection + BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. + This requirement can be avoided by setting ``skip_bq_connection_check`` option + to ``True``, in which case the connection (default or pre-configured) would be + used as-is without any existence or permission check. **Limitations** From dea94309617a9277acde1ccfad145b20b303c9aa Mon Sep 17 00:00:00 2001 From: Shobhit Singh Date: Fri, 22 Mar 2024 23:21:38 +0000 Subject: [PATCH 2/5] be more specific about the service accounts --- README.rst | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/README.rst b/README.rst index 082bf90772..9e9231b24c 100644 --- a/README.rst +++ b/README.rst @@ -236,9 +236,8 @@ and you must be granted the following IAM roles: * BigQuery Data Editor (roles/bigquery.dataEditor) * BigQuery Connection Admin (roles/bigquery.connectionAdmin) -* Service Account User (roles/iam.serviceAccountUser) on the - `service account `__ - ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` +* Service Account User (roles/iam.serviceAccountUser) in the project or on the + `service account of the BigQuery connection `_ * Vertex AI User (roles/aiplatform.user) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. @@ -299,7 +298,7 @@ creates: not set to ``True``. You can use a pre-configured BigQuery connection if you prefer. If the BigQuery connection is created, the BigQuery service will create a - `Google Cloud–managed IAM service account https://cloud.google.com/bigquery/docs/working-with-connections#before_you_begin`_ + `Google Cloud-managed IAM service account `_ and attach it to the connection. 3. A BigQuery remote function that talks to the cloud function (1) using the BigQuery connection (2). @@ -308,16 +307,16 @@ BigQuery connections are created in the same location as the BigQuery DataFrames session, using the name you provide in the custom function definition. To view and manage connections, do the following: -1. Go to `BigQuery in the Google Cloud Console `__. +1. Go to `BigQuery in the Google Cloud Console `_. 2. Select the project in which you created the remote function. 3. In the Explorer pane, expand that project and then expand External connections. BigQuery remote functions are created in the dataset you specify, or -in a special type of `hidden dataset `__ +in a special type of `hidden dataset `_ referred to as an anonymous dataset. To view and manage remote functions created in a user provided dataset, do the following: -1. Go to `BigQuery in the Google Cloud Console `__. +1. Go to `BigQuery in the Google Cloud Console `_. 2. Select the project in which you created the remote function. 3. In the Explorer pane, expand that project, expand the dataset in which you created the remote function, and then expand Routines. @@ -346,9 +345,12 @@ following IAM roles: * BigQuery Data Editor (roles/bigquery.dataEditor) * BigQuery Connection Admin (roles/bigquery.connectionAdmin) * Cloud Functions Developer (roles/cloudfunctions.developer) -* Service Account User (roles/iam.serviceAccountUser) on the - `service account `__ +* Service Account User (roles/iam.serviceAccountUser) in the project or on the + service accounts + * Cloud Function user provided service account via or the + `default service account `_ ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` + * `Service account of the BigQuery connection `_ * Storage Object Viewer (roles/storage.objectViewer) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. From 495b2c1452c735dc8cd316102b275c5a6ea92c11 Mon Sep 17 00:00:00 2001 From: Shobhit Singh Date: Sat, 23 Mar 2024 00:53:42 +0000 Subject: [PATCH 3/5] more rewording --- README.rst | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/README.rst b/README.rst index 9e9231b24c..6c35ab24dd 100644 --- a/README.rst +++ b/README.rst @@ -241,8 +241,9 @@ and you must be granted the following IAM roles: * Vertex AI User (roles/aiplatform.user) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. - This requirement can be avoided by setting ``skip_bq_connection_check`` option - to ``True``, in which case the connection (default or pre-configured) would be + This requirement can be avoided by setting + ``bigframes.pandas.options.bigquery.skip_bq_connection_check`` option to ``True``, + in which case the connection (default or pre-configured) would be used as-is without any existence or permission check. @@ -293,19 +294,18 @@ function in BigQuery DataFrames (See `code samples creates: 1. A `Cloud Functions (2nd gen) function `_. -2. A `BigQuery connection `_ - if the connection does not exist and ``skip_bq_connection_check`` option is - not set to ``True``. You can use a pre-configured BigQuery connection if you - prefer. If the BigQuery connection is created, the BigQuery service will +2. A `BigQuery connection `_. + If the BigQuery connection is created, the BigQuery service will create a `Google Cloud-managed IAM service account `_ - and attach it to the connection. + and attach it to the connection. You can use a pre-configured BigQuery + connection if you prefer, in which case the connection creation is skipped. 3. A BigQuery remote function that talks to the cloud function (1) using the BigQuery connection (2). BigQuery connections are created in the same location as the BigQuery DataFrames session, using the name you provide in the custom function -definition. To view and manage connections, do the following: +definition. To view and manage connections, do the following: 1. Go to `BigQuery in the Google Cloud Console `_. 2. Select the project in which you created the remote function. @@ -347,15 +347,18 @@ following IAM roles: * Cloud Functions Developer (roles/cloudfunctions.developer) * Service Account User (roles/iam.serviceAccountUser) in the project or on the service accounts - * Cloud Function user provided service account via or the - `default service account `_ - ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` + + * User provided service account for Cloud Function or the + `default service account `_ + ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` * `Service account of the BigQuery connection `_ + * Storage Object Viewer (roles/storage.objectViewer) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. - This requirement can be avoided by setting ``skip_bq_connection_check`` option - to ``True``, in which case the connection (default or pre-configured) would be + This requirement can be avoided by setting + ``bigframes.pandas.options.bigquery.skip_bq_connection_check`` option to ``True``, + in which case the connection (default or pre-configured) would be used as-is without any existence or permission check. **Limitations** From 49e1f4e94fbe7f2b40b11126f49d5978a16c7f3d Mon Sep 17 00:00:00 2001 From: Shobhit Singh Date: Sat, 23 Mar 2024 01:10:03 +0000 Subject: [PATCH 4/5] make rst hyperlink labels non-unique --- README.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.rst b/README.rst index 6c35ab24dd..8aeed88bcb 100644 --- a/README.rst +++ b/README.rst @@ -237,7 +237,7 @@ and you must be granted the following IAM roles: * BigQuery Data Editor (roles/bigquery.dataEditor) * BigQuery Connection Admin (roles/bigquery.connectionAdmin) * Service Account User (roles/iam.serviceAccountUser) in the project or on the - `service account of the BigQuery connection `_ + `service account of the BigQuery connection `__ * Vertex AI User (roles/aiplatform.user) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. @@ -307,16 +307,16 @@ BigQuery connections are created in the same location as the BigQuery DataFrames session, using the name you provide in the custom function definition. To view and manage connections, do the following: -1. Go to `BigQuery in the Google Cloud Console `_. +1. Go to `BigQuery in the Google Cloud Console `__. 2. Select the project in which you created the remote function. 3. In the Explorer pane, expand that project and then expand External connections. BigQuery remote functions are created in the dataset you specify, or -in a special type of `hidden dataset `_ +in a special type of `hidden dataset `__ referred to as an anonymous dataset. To view and manage remote functions created in a user provided dataset, do the following: -1. Go to `BigQuery in the Google Cloud Console `_. +1. Go to `BigQuery in the Google Cloud Console `__. 2. Select the project in which you created the remote function. 3. In the Explorer pane, expand that project, expand the dataset in which you created the remote function, and then expand Routines. @@ -351,7 +351,7 @@ following IAM roles: * User provided service account for Cloud Function or the `default service account `_ ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` - * `Service account of the BigQuery connection `_ + * `Service account of the BigQuery connection `__ * Storage Object Viewer (roles/storage.objectViewer) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default From 01490e3e8d91b5c8575a3245ebdd25c13110b0b7 Mon Sep 17 00:00:00 2001 From: Shobhit Singh Date: Mon, 25 Mar 2024 08:04:29 +0000 Subject: [PATCH 5/5] remove confusing resource level IAM docs until we support that --- README.rst | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/README.rst b/README.rst index 8aeed88bcb..73709641de 100644 --- a/README.rst +++ b/README.rst @@ -232,12 +232,11 @@ you must enable the following APIs: * The BigQuery Connection API (bigqueryconnection.googleapis.com) * The Vertex AI API (aiplatform.googleapis.com) -and you must be granted the following IAM roles: +and you must be granted the following IAM roles in the project: * BigQuery Data Editor (roles/bigquery.dataEditor) * BigQuery Connection Admin (roles/bigquery.connectionAdmin) -* Service Account User (roles/iam.serviceAccountUser) in the project or on the - `service account of the BigQuery connection `__ +* Service Account User (roles/iam.serviceAccountUser) * Vertex AI User (roles/aiplatform.user) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection. @@ -340,19 +339,12 @@ To use BigQuery DataFrames remote functions, you must enable the following APIs: * The Cloud Resource Manager API (cloudresourcemanager.googleapis.com) To use BigQuery DataFrames remote functions, you must be granted the -following IAM roles: +following IAM roles in the project: * BigQuery Data Editor (roles/bigquery.dataEditor) * BigQuery Connection Admin (roles/bigquery.connectionAdmin) * Cloud Functions Developer (roles/cloudfunctions.developer) -* Service Account User (roles/iam.serviceAccountUser) in the project or on the - service accounts - - * User provided service account for Cloud Function or the - `default service account `_ - ``PROJECT_NUMBER-compute@developer.gserviceaccount.com`` - * `Service account of the BigQuery connection `__ - +* Service Account User (roles/iam.serviceAccountUser) * Storage Object Viewer (roles/storage.objectViewer) * Project IAM Admin (roles/resourcemanager.projectIamAdmin) if using default BigQuery connection, or Browser (roles/browser) if using a pre-configured connection.