Proxy issue appearing in 10.0.0 #423

thatch · 2021-05-10T13:31:04Z

Describe the bug:

We use an http proxy to communicate using the google-ads library. This worked fine on previous versions by setting the http_proxy and https_proxy env vars. When updating to 10.0.0 or later, I get the following error when using search:

         debug_error_string = "{"created":"@1620489718.036336118","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5419,"referenced_errors":[{"
created":"@1620489718.036326964","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

This problem exists on 10.0.0 unmodified, 10.0.0 with PR 405 applied (with proxy set in yaml), and also on 11.0.0 unmodified. I am 100% sure that the grpc version is the same between working on google-ads 9.0.0 and failing on 10.0.0.

I've run a bisect which points at either 6c242d3, or the changes we had to make in our script to deal with the disruptive api changes. Do you have a more granular version of that commit that we could continue bisecting on?

Steps to Reproduce:
Internal build, sorry. I'm happy to test things to provide simplified versions or add debugging information as you'd like.

Expected behavior:
Proxy support continues to work.

Screenshots:
n/a

Library version and API version:

I see the failed call using /google.ads.googleads.v5.services.GoogleAdsService/Search

google-api-core 1.22.1
grpcio 1.37.1
google-ads 10.0.0 (or 11.0.0 if you'd rather)

Request/Response Logs:
I can generate these if you let me know what you're looking for.

Additional context:
n/a

The text was updated successfully, but these errors were encountered:

laurachevalier4 · 2021-05-10T14:48:08Z

Hey @thatch! Thanks for providing all this info. A couple things:

Are you using distinct endpoints for http and https? As I understand it from the grpc docs, only one of http_proxy and https_proxy will actually be used. Can you retry with setting only http_proxy and not https_proxy, and post the error message that results from the same Search call?
Are you getting this exact error message in all of the cases you outlined (v10 w/o Add http proxy support #405, v10 with Add http proxy support #405 applied, and v11)? In the cases of v10 with PR Add http proxy support #405 applied and v11, are you getting the same errors with and without setting the http_proxy value in your google-ads.yaml? Otherwise, can you post the different errors you're getting per each case (or let us know if they are the same in every case):
- v10
  - http_proxy is set as an env var
- v10 with 405:
  - http_proxy is set as an env var and in the yaml
  - http_proxy is set only as an env var
  - http_proxy is set only in the yaml
- v11
  - http_proxy is set as an env var and in the yaml
  - http_proxy is set only as an env var
  - http_proxy is set only in the yaml

thatch · 2021-05-10T21:46:16Z

Adding @dfaradjev who has more experience with testing this.

dfaradjev · 2021-05-10T22:04:34Z

Hi @laurachevalier4 ,
I will try to jump in to answer some of your questions:

Setting http_proxy in yaml or in the environment does not change anthing on our side. We get
google.auth.exceptions.TransportError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /o/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb15273c400>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
Only if we set https_proxy in the environment things are looking different...

Headers: {}
Fault: {
  "created": "@1620675396.953970565",
  "description": "Failed to pick subchannel",
  "file": "src/core/ext/filters/client_channel/client_channel.cc",
  "file_line": 5419,
  "referenced_errors": [
    {
      "created": "@1620675396.953960685",
      "description": "failed to connect to all addresses",
      "file": "src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc",
      "file_line": 397,
      "grpc_status": 14
    }
  ]
}

We get exactly the same error message with or without setting http_proxy in the yaml file. (after https_proxy env. variable is set)

Really appreciate your help!

thatch · 2021-05-10T22:10:09Z

@dfaradjev two more questions -- on 9.0.0 where it's successful, is it fetching the v5 endpoint? And does it fail identically on 10.0.0 and 11.0.0?

dfaradjev · 2021-05-10T22:47:52Z

v9.0.0 is working fine when I set https_proxy environment variable. It is fetching the v5 endpoint.
v10.0.0 is not working for the same reason as v11.0.0 - google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses

laurachevalier4 · 2021-05-11T23:31:33Z

@dfaradjev I see you confirmed the outcome with the https_proxy env var set and http_proxy not set in the yaml, but can you also make sure the http_proxy env variable is empty and let me know the behavior (in v11)? Also, do you have a longer stack trace you can provide?

Thanks!

dfaradjev · 2021-05-11T23:51:03Z

Confirming, I got the same error when http_proxy env. variable was not set at all.
Here is the full traceback(redacted) using v11.0.0:

[2021-05-11 16:42:18,190 - INFO] Request
-------
Method: /google.ads.googleads.v5.services.GoogleAdsService/Search
Host: googleads.googleapis.com
Headers: {
  "developer-token": "REDACTED",
  "login-customer-id": "<MY_CUSTOMER_ID>",
  "x-goog-api-client": "gl-python/3.8.6 grpc/1.37.1 gax/1.22.1",
  "x-goog-request-params": "customer_id=<MY_CUSTOMER_ID>"
}
Request: customer_id: "<MY_CUSTOMER_ID>"
query: "SELECT customer_client.id FROM customer_client WHERE customer_client.manager = True"
page_size: 3000


Response
-------
Headers: {}
Fault: {
  "created": "@1620776538.189482558",
  "description": "Failed to pick subchannel",
  "file": "src/core/ext/filters/client_channel/client_channel.cc",
  "file_line": 5419,
  "referenced_errors": [
    {
      "created": "@1620776538.189472689",
      "description": "failed to connect to all addresses",
      "file": "src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc",
      "file_line": 397,
      "grpc_status": 14
    }
  ]
}

[2021-05-11 16:42:18,190 - WARNING] Request made: ClientCustomerId: <MY_CUSTOMER_ID>, Host: googleads.googleapis.com, Method: /google.ads.googleads.v5.services.GoogleAdsService/Search, RequestId: None, IsFault: True, FaultMessage: failed to connect to all addresses
Traceback (most recent call last):
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 216, in __call__
    response, ignored_call = self._with_call(request,
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 257, in _with_call
    return call.result(), call
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_channel.py", line 343, in result
    raise self
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 241, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 266, in with_call
    return self._with_call(request,
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 257, in _with_call
    return call.result(), call
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_channel.py", line 343, in result
    raise self
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 241, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 266, in with_call
    return self._with_call(request,
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 254, in _with_call
    call = self._interceptor.intercept_unary_unary(continuation,
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/google/ads/googleads/interceptors/exception_interceptor.py", line 169, in intercept_unary_unary
    self._handle_grpc_failure(response)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/google/ads/googleads/interceptors/exception_interceptor.py", line 141, in _handle_grpc_failure
    raise self._get_error_from_response(response)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_interceptor.py", line 241, in continuation
    response, call = self._thunk(new_method).with_call(
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_channel.py", line 957, in with_call
    return _end_unary_response_blocking(state, call, True, None)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1620776538.189482558","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5419,"referenced_errors":[{"created":"@1620776538.189472689","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 38, in <module>
  File "<string>", line 36, in __run
  File "/local_path/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/local_path/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/my_code_path/ads_report_downloader.py", line 297, in <module>
    manager_accounts = get_accounts_by_type(mcc_client_id, True)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/my_code_path/ads_report_downloader.py", line 44, in get_accounts_by_type
    results = ga_service.search(request=search_request)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/google/ads/googleads/v5/services/services/google_ads_service/client.py", line 2140, in search
    response = rpc(
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/google/api_core/gapic_v1/method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "/data/users/my_user/my_path/ads_report_downloader#link-tree/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses

thatch · 2021-05-12T15:47:52Z

It would appear that the CONNECT is different, and dropped the port number in 10.0.0

# in 9.0.0
CONNECT googleads.googleapis.com:443 HTTP/1.0
# in 10.0.0
CONNECT googleads.googleapis.com HTTP/1.0

thatch · 2021-05-12T16:36:56Z

My proxy admin is advising that grpc is sending non-compliant CONNECT requests. The port is required as part of the "authority form", per

https://datatracker.ietf.org/doc/html/rfc7231#section-4.3.6
https://datatracker.ietf.org/doc/html/rfc7230#section-5.3.3

do you know if grpc has a policy of assuming 443 and should be adding or at least validating it before issuing the request? The source in src/core/ext/filters/client_channel/http_connect_handshaker.cc refers to it as "server_name" without any reference to it including a port.

laurachevalier4 · 2021-05-12T18:19:19Z

That additional info is really useful, thanks. I think I know what's happening here, can you try something to confirm: In your google-ads.yaml, add an endpoint config that's set to googleads.googleapis.com:443?

dfaradjev · 2021-05-12T19:28:21Z

@laurachevalier4 Adding endpoint: googleads.googleapis.com:443 to the yaml seems to help. Will keep testing. Is this a permanent solution or a temporary workaround?
Thanks

laurachevalier4 · 2021-05-12T19:40:27Z

@dfaradjev this is a temporary workaround. Thanks for testing! Please let us know the outcome as you keep testing. Since it seems to work (unless you say otherwise), we're looking at the possible permanent solutions this points us to.

laurachevalier4 · 2021-05-13T17:48:08Z

@dfaradjev and @thatch to help us determine what's going to be the best permanent solution for this, can you provide a couple more details?

With the endpoint field removed from the config (i.e. with the same setup you had when seeing the error), set the following grpc tracing environment variables and rerun the Search query:
- GRPC_TRACE=client_channel_routing,cares_resolver,connectivity_state
- GRPC_VERBOSITY=debug
  What are the output logs when the Search query is rerun?
Confirm whether the proxy is missing ALPN.

ejona86 · 2021-05-13T18:41:21Z

FWIW, I think it is clear (via #423 (comment)) that gRPC is doing the wrong thing here, so fixing it in gRPC (grpc/grpc#26227) seems the appropriate permanent solution. That said, I don't work on C core so we'll need to see what they think.

I don't think ALPN has any impact here, as the proxy is not doing TLS at all (it's a TCP tunnel).

jalopezsilva · 2021-05-13T18:47:51Z

Confirm whether the proxy is missing ALPN.

Our proxy has a TCP endpoint and a TLS endpoint. Through the TLS endpoint we support negotiating h1, h2 through ALPN. That said as @ejona86 mentioned, it's unclear why the ALPN would affect anything here? The H1 CONNECT sent is clearly malformed.

laurachevalier4 · 2021-05-13T19:19:54Z

Looks like this is attributed to the grpc bug, so I'm going to close this and recommend the temporary workaround in the meantime, until that grpc bug is fixed.

thatch added bug Something isn't working triage New issue; requires attention labels May 10, 2021

thatch mentioned this issue May 12, 2021

http_proxy support can issue non-compliant CONNECT requests grpc/grpc#26227

Closed

laurachevalier4 closed this as completed May 13, 2021

BenRKarl added a commit that referenced this issue Jul 12, 2021

Update the minimum version of grpcio to 1.38.1 per Issue #423.

c0da5b6

BenRKarl added a commit that referenced this issue Jul 12, 2021

Update the minimum version of grpcio to 1.38.1 per Issue #423. (#468)

4676d65

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy issue appearing in 10.0.0 #423

Proxy issue appearing in 10.0.0 #423

thatch commented May 10, 2021

laurachevalier4 commented May 10, 2021

thatch commented May 10, 2021

dfaradjev commented May 10, 2021 •

edited

Loading

thatch commented May 10, 2021

dfaradjev commented May 10, 2021

laurachevalier4 commented May 11, 2021

dfaradjev commented May 11, 2021 •

edited

Loading

thatch commented May 12, 2021

thatch commented May 12, 2021

laurachevalier4 commented May 12, 2021

dfaradjev commented May 12, 2021 •

edited

Loading

laurachevalier4 commented May 12, 2021

laurachevalier4 commented May 13, 2021

ejona86 commented May 13, 2021

jalopezsilva commented May 13, 2021 •

edited

Loading

laurachevalier4 commented May 13, 2021

Proxy issue appearing in 10.0.0 #423

Proxy issue appearing in 10.0.0 #423

Comments

thatch commented May 10, 2021

laurachevalier4 commented May 10, 2021

thatch commented May 10, 2021

dfaradjev commented May 10, 2021 • edited Loading

thatch commented May 10, 2021

dfaradjev commented May 10, 2021

laurachevalier4 commented May 11, 2021

dfaradjev commented May 11, 2021 • edited Loading

thatch commented May 12, 2021

thatch commented May 12, 2021

laurachevalier4 commented May 12, 2021

dfaradjev commented May 12, 2021 • edited Loading

laurachevalier4 commented May 12, 2021

laurachevalier4 commented May 13, 2021

ejona86 commented May 13, 2021

jalopezsilva commented May 13, 2021 • edited Loading

laurachevalier4 commented May 13, 2021

dfaradjev commented May 10, 2021 •

edited

Loading

dfaradjev commented May 11, 2021 •

edited

Loading

dfaradjev commented May 12, 2021 •

edited

Loading

jalopezsilva commented May 13, 2021 •

edited

Loading