Interrupt heartbeating activity on pause #854

THardy98 · 2025-05-01T07:38:46Z

What was changed

Adds support to interrupt heartbeating activities from pause requests.

Why?

Functionally support users pausing their activities via the CLI

Closes Heartbeating activities should be interrupted when the activities are paused. #812
How was this tested:
Simple integration test
Any docs updates needed?
Unsure

.vscode/settings.json

cretz · 2025-05-01T13:02:42Z

temporalio/client.py

@@ -6250,6 +6258,9 @@ async def heartbeat_async_activity(
            )
            if resp_by_id.cancel_requested:
                raise AsyncActivityCancelledError()
+            if resp_by_id.activity_paused:


Hrmm, looks like we never added "raises" docs to AsyncActivityHandle.heartbeat, wonder if we should now

temporalio/exceptions.py

temporalio/worker/_activity.py

cretz · 2025-05-05T13:09:22Z

temporalio/activity.py

@@ -135,6 +138,29 @@ def _logger_details(self) -> Mapping[str, Any]:
 _current_context: contextvars.ContextVar[_Context] = contextvars.ContextVar("activity")


+@dataclass


I think this should be frozen

We mutate the fields in this object to reflect changes across running activity & _context

changed to frozen class + holder

cretz · 2025-05-05T13:22:10Z

temporalio/worker/_activity.py

@@ -436,6 +438,7 @@ async def _run_activity(
                    runtime_metric_meter=None
                    if sync_non_threaded
                    else self._metric_meter,
+                    cancellation_details=running_activity.cancellation_details,


The context says it is optional, but this is set with a never-None value. I think you may need some kind of "holder" wrapper.

That would be preferred over a non-optional field?

changed to frozen class + holder

temporalio/activity.py

temporalio/bridge/src/client.rs

temporalio/testing/_activity.py

cretz · 2025-05-05T17:41:07Z

temporalio/testing/_activity.py

+        if cancellation_details:
+            self.cancellation_details.set_details(cancellation_details)


I think this needs to always be set, even if not provided. May want to change param to be non-option w/ a default of temporalio.activity.ActivityCancellationDetails(). Usually using a default param of a full expression is bad in Python because it isn't called for each call, but since the details are immutable, sharing the default for everyone is safe.

I think this needs to always be set, even if not provided.

Why?

May want to change param to be non-option w/ a default of temporalio.activity.ActivityCancellationDetails()

Sort of defeats the purpose of being able to return None without checking all fields are False, in which case, maybe the holder indirection isn't so useful.

I think the logic needs to be "you can guarantee activity.cancellation_details() is not None when the activity is being canceled", but this breaks that guarantee.

Unresolving, I think this is still an issue. We should always set an instance of this on cancel in test environment. So the parameter should default to an empty instance with no optional accepted.

temporalio/activity.py

temporalio/worker/_activity.py

cretz · 2025-05-05T17:47:45Z

tests/worker/test_workflow.py

+        except (CancelledError, asyncio.CancelledError):
+            return activity.cancellation_details()


Need some tests to confirm what happens when this is not caught

Added a test for pausing/unpausing activities, we don't catch cancel errors.

Added the unpause activity method to the bridge client - it was missing.

I made it such that a cancelled error on activity pause returns an activity task failure w/ application failure:

When going through the default failure converter, a canceled error populates cancel failure info, but core expects application failure info because we are populating the failed field of the completion instead of cancel.

A bit of a weird situation, because we can't populate cancel for the reason you mentioned above (we tell server to cancel and server complains that a cancel hasn't been requested yet - like you mention above)

I can revert this change if you prefer, I don't think it causes any notable server effect anyway (state of paused activities are suspended server-side, iiuc), but probably a good thing to have failures sent properly just as a general rule

cretz · 2025-05-06T20:01:11Z

temporalio/testing/_activity.py

+        cancellation_details: Optional[
+            temporalio.activity.ActivityCancellationDetails
+        ] = None,


Updated #854 (comment) thread, I think this parameter should not be Optional

whoops didn't mean to resolve - param is non-optional. Updated callers.

cretz · 2025-05-06T21:28:19Z

temporalio/testing/_activity.py

-    def cancel(self) -> None:
+    def cancel(
+        self,
+        cancellation_details: temporalio.activity.ActivityCancellationDetails,


To clarify, providing the parameter still needs to be optional (this is a backwards incompatible change otherwise), it just needs to not be Optional. So:

Suggested change

cancellation_details: temporalio.activity.ActivityCancellationDetails,

cancellation_details: temporalio.activity.ActivityCancellationDetails = temporalio.activity.ActivityCancellationDetails(cancel_requested=True),

I just picked cancel_requested as the default (might as well set one to True by default). Usually having an expression as a parameter default like this in Python is bad because that same instance is reused for every default so mutations are shared, but this object is immutable so it's safe.

Sorry for the confusion

… fix

THardy98 requested a review from a team as a code owner May 1, 2025 07:38

cretz reviewed May 1, 2025

View reviewed changes

THardy98 force-pushed the support_activity_pause branch from d5107b0 to 9763505 Compare May 3, 2025 06:35

THardy98 requested a review from cretz May 3, 2025 06:39

THardy98 force-pushed the support_activity_pause branch 2 times, most recently from 5c8299a to 4451e77 Compare May 3, 2025 18:07

cretz reviewed May 5, 2025

View reviewed changes

temporalio/activity.py Outdated Show resolved Hide resolved

cretz reviewed May 5, 2025

View reviewed changes

temporalio/activity.py Outdated Show resolved Hide resolved

cretz reviewed May 5, 2025

View reviewed changes

THardy98 requested a review from cretz May 6, 2025 18:00

cretz reviewed May 6, 2025

View reviewed changes

THardy98 force-pushed the support_activity_pause branch 2 times, most recently from c5ca36e to c3634fe Compare May 6, 2025 20:58

cretz reviewed May 6, 2025

View reviewed changes

THardy98 added 15 commits May 16, 2025 11:09

Init commit - waiting for core changes

d8064c8

add activity paused error usage

dfbf3a4

working for async activities

994a0d6

working for sync activities

7cdd3ef

linting and cleanup

af99faa

add cancellation details arg to testing ActivityEnvironment cancel

3c2d5d5

remove .vscode

73c2684

formatting

c50d968

use object reference instead of function, picklable

6bdba9f

nits

3c82a1b

use holder

cff4fb2

docstrings

96cfaee

add test for pause/unpause

f0a8f76

linting, reduce heartbeat timeouts for faster test

523dd0c

make cancellation details non-optional for testing activity env

3e7ab2f

THardy98 added 2 commits May 16, 2025 11:18

address pr suggestion

43b6c1c

rebase conflict fixes

58bd687

THardy98 force-pushed the support_activity_pause branch from c3634fe to 58bd687 Compare May 16, 2025 15:31

include is_worker_shutdown as reason for requested cancellation, test…

e055766

… fix

THardy98 force-pushed the support_activity_pause branch from 4977651 to e055766 Compare May 16, 2025 16:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Interrupt heartbeating activity on pause #854

Interrupt heartbeating activity on pause #854

THardy98 commented May 1, 2025

cretz May 1, 2025

cretz May 5, 2025

THardy98 May 5, 2025

THardy98 May 5, 2025

cretz May 5, 2025

THardy98 May 5, 2025 •

edited

Loading

THardy98 May 5, 2025

cretz May 5, 2025

THardy98 May 5, 2025 •

edited

Loading

cretz May 5, 2025

cretz May 6, 2025

cretz May 5, 2025

THardy98 May 6, 2025 •

edited

Loading

cretz May 6, 2025

THardy98 May 6, 2025 •

edited

Loading

cretz May 6, 2025

		@@ -135,6 +138,29 @@ def _logger_details(self) -> Mapping[str, Any]:
		_current_context: contextvars.ContextVar[_Context] = contextvars.ContextVar("activity")


		@dataclass

		if cancellation_details:
		self.cancellation_details.set_details(cancellation_details)

		except (CancelledError, asyncio.CancelledError):
		return activity.cancellation_details()

	cancellation_details: temporalio.activity.ActivityCancellationDetails,
	cancellation_details: temporalio.activity.ActivityCancellationDetails = temporalio.activity.ActivityCancellationDetails(cancel_requested=True),

Interrupt heartbeating activity on pause #854

Are you sure you want to change the base?

Interrupt heartbeating activity on pause #854

Conversation

THardy98 commented May 1, 2025

What was changed

Why?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THardy98 May 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THardy98 May 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THardy98 May 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THardy98 May 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

THardy98 May 5, 2025 •

edited

Loading

THardy98 May 5, 2025 •

edited

Loading

THardy98 May 6, 2025 •

edited

Loading

THardy98 May 6, 2025 •

edited

Loading