Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Bug] Python client not able to connect to self-hosted Temporal server via proxy using authorization header #814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dipali-bhatt opened this issue Apr 3, 2025 · 1 comment
Open

[Bug] Python client not able to connect to self-hosted Temporal server via proxy using authorization header #814

dipali-bhatt opened this issue Apr 3, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@dipali-bhatt
Copy link

What are you really trying to do?

We’re trying to connect Python client to our self-hosted Temporal server via a proxy using an authorization header.

Describe the bug

We see that it is able to connect to the worker at startup, but after exactly 60 seconds, it fails with the below error. Note that we do see some transient 503 errors while connecting to the server; but after that it’s able to connect successfully (verified in server logs).

Minimal Reproduction

Here’ the set of errors we see:
1.

Error: Client connection not established Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }.
Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 157, in main
    await conn.create_client()
  File "/home/temporal/temporal_sample_worker_python/src/utils.py", line 105, in create_client
    self.client = await Client.connect("dev.api.data.falcon.sfdcfc.net:7443",
  File "/usr/local/lib64/python3.9/site-packages/temporalio/client.py", line 176, in connect
    await temporalio.service.ServiceClient.connect(connect_config),
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 209, in connect
    return await _BridgeServiceClient.connect(config)
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1117, in connect
    await client._connected_client()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1130, in _connected_client
    self._bridge_client = await temporalio.bridge.client.Client.connect(
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/client.py", line 97, in connect
    await temporalio.bridge.temporal_sdk_bridge.connect_client(
RuntimeError: Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }

2025-03-19T02:17:48.974543Z  WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
2025-03-19T02:17:48.974547Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19T02:17:48.974551Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19 02:17:48,975 - ERROR - Worker failed, shutting down
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed
2025-03-19 02:17:48,975 - INFO - Beginning worker shutdown, will wait 0:00:00 before cancelling activities

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 179, in main
    await run_worker(client, taskqueue_name, args.worker_type)
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 142, in run_worker
    await worker.run()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_worker.py", line 571, in run
    raise exception
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed

Environment/Versions

  • Temporal Version: Server v1.26.2, SDK v1.10.0

Additional context

Our issue looks similar to this this open bug: [Bug] Worker hangs after polling workflow task queue · Issue #631 · temporalio/sdk-python · GitHub
Wanted to check if it is same issue and if not we need help in understanding the fixing it.
@dipali-bhatt dipali-bhatt added the bug Something isn't working label Apr 3, 2025
@dipali-bhatt dipali-bhatt changed the title [Bug] FILL_TITLE_HERE [Bug] Python client not able to connect to self-hosted Temporal server via proxy using authorization header Apr 3, 2025
@cretz
Copy link
Member

cretz commented Apr 3, 2025 •
edited
Loading

Can you confirm that this is not an issue when connecting directly to Temporal and therefore an issue specific to your proxy? What kind of proxy is this?

Also note that the proxy has to be setup with the proper parameters, specifically it should allow individual calls for more than 80s, have a soft max connection age (i.e. http/2 go away) of no less than 6m and hard mas connection age (i.e. connection kill) of 8m. This is because on Temporal server we allow 70s, 5m, and 6m10s for these values respectively so the proxy needs to not intrude on that. The other issue was also an improper ingress/proxy configuration issue where it was failing requests that need to stay open for 70s.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cretz @dipali-bhatt