I had only looked at the artifactregistry, functions and resourcemanager packages, but I suspected the issue may be applicable to other packages. In fact, it seems to be applicable to all of them.

The *AsyncClient objects must use a google.api_core.retry_async.AsyncRetry object, rather than a google.api_core.retry.Retry object. However, all methods document use of the latter, which is incorrect. Some even use a default google.api_core.retry.Retry object, which would have no effect whatsoever for async code. For example, paclages/google-cloud-resource-manager/google/cloud/resourcemanager_v3/services/projects/async_client.py has:

class ProjectsAsyncClient:
    async def get_project(
        self,
        request: Optional[Union[projects.GetProjectRequest, dict]] = None,
        *,
        name: Optional[str] = None,
        retry: OptionalRetry = gapic_v1.method.DEFAULT,
        timeout: Union[float, object] = gapic_v1.method.DEFAULT,
        metadata: Sequence[Tuple[str, str]] = (),
    ) -> projects.Project:
        [...]
        rpc = gapic_v1.method_async.wrap_method(
            self._client._transport.get_project,
            default_retry=retries.Retry(
                initial=0.1,
                maximum=60.0,
                multiplier=1.3,
                predicate=retries.if_exception_type(
                    core_exceptions.ServiceUnavailable,
                ),
                deadline=60.0,
            ),
            default_timeout=60.0,
            client_info=DEFAULT_CLIENT_INFO,
        )

The same issue is with google.cloud.functions_v1.CloudFunctionsServiceAsyncClient and with google.cloud.artifactregistry_v1beta2.ArtifactRegistryAsyncClient. Others, like google.cloud.functions_v2.FunctionServiceAsyncClient do not pass a default_retry at all, which effectively means no retry by default, but they incorrectly document that a google.api_core.retry.Retry instance should be used, both via the docstrings (and hence generated documention), as well as via the annotation:

try:
    OptionalRetry = Union[retries.Retry, gapic_v1.method._MethodDefault]
except AttributeError:  # pragma: NO COVER
    OptionalRetry = Union[retries.Retry, object]  # type: ignore

The reason it doesn't work is that the google.api_core.retry.Retry decorator will effectively wrap the call that returns a coroutine, without awaiting on it, so it doesn't have a chance to actually catch any exceptions, silently not doing anything useful.

Searching in the codebase for packages affected by passing an incorrect object to default_retry with

grep -r default_retry=retries.Retry packages/ | grep async_client.py | cut -d/ -f2 | sort -u

yields:

google-cloud-access-approval
google-cloud-advisorynotifications
google-cloud-alloydb
google-cloud-api-gateway
google-cloud-apigee-connect
google-cloud-apigee-registry
google-cloud-artifact-registry
google-cloud-bigquery-datapolicies
google-cloud-billing
google-cloud-billing-budgets
google-cloud-certificate-manager
google-cloud-confidentialcomputing
google-cloud-contentwarehouse
google-cloud-datacatalog
google-cloud-datalabeling
google-cloud-dataplex
google-cloud-deploy
google-cloud-discoveryengine
google-cloud-documentai
google-cloud-edgecontainer
google-cloud-enterpriseknowledgegraph
google-cloud-essential-contacts
google-cloud-functions
google-cloud-gke-multicloud
google-cloud-ids
google-cloud-rapidmigrationassessment
google-cloud-recommendations-ai
google-cloud-recommender
google-cloud-resource-manager
google-cloud-resource-settings
google-cloud-secret-manager
google-cloud-securitycenter
google-cloud-service-control
google-cloud-shell
google-cloud-storageinsights
google-cloud-support
google-cloud-talent
google-cloud-vmwareengine
google-cloud-webrisk
google-cloud-websecurityscanner
google-cloud-workstations
google-maps-mapsplatformdatasets

Apart from fixing the documentation and the code, I would also like to suggest raising a warning in the google.api_core.retry.retry_target function (also used by the Retry object) if the return value of target() is awaitable (inspect.isawaitable()). Given that the documentation has been incorrect, it is possible that people have been using it incorrectly and they wouldn't know about it, until reliability is affected.