Initial FSM PoC #4669

Bibo-Joshi · 2025-02-04T20:44:29Z

Works on #2770

"It has been a while since we openend #2770" would be an understatement. Yet here we are :D
Finally, I got around to draft up a somewhat proper PoC for my ideas outlined in #2770 (comment). I also attached an example that showcases how the FSM setup can be used. You'll need ≥2 accounts to test it.

To be clear: as of 2025-02-04 this PR does by far not include everything mentioned in #2770 (comment). I hope to work on it in the nearish-future. Continued work on this PR could be greatly supported by intermediate reviews, comments & discussions :)

ToDos / Ideas / Thoughts

(Listing here only things that came to my mind explicitly while working on this PR, not copying everything from #2770)

Using context.job_queue.run_once for conversation timeouts is relatively easy, but would have to repeated everywhere which also holds for the cancellation. Maybe we can think of a clever way to abstract that for the user.
Having both groups and states makes the "who handles what" logic more involved. Do we do "for state in states: for group in state" or "for group in groups: for state in group"? Also State.ANY and things like StateA | StateB make it necessary that one update can be handled by multiple states. All this at least needs proper documentation about what's going on. Additionally, we should think carefully about what we want here.
I'm unsure about if get_active_key_state should be async or not:
- I feel like it should not make too much async operations - since it's part of the logic in App.process_update, it should be as quick as possible
- If we want to support lazy-loading, this would probably be the place were it comes into play - depending on how we design the persistence connection
- If states are set within get_active_key_state (as is done in the fsm example I added), then I'm wondering if one should use a semaphore for that. If one would do that, this could lead to major delays in process_update
Instead of BoundedSemaphore(1), we could probably just use Lock as synchronization primitive

…e update

…llow

Poolitzer

Just some first impressions, once I gain a further understanding I can give more

Poolitzer · 2025-02-08T18:14:46Z

examples/fsmbot.py

+        self.admin_id = admin_id
+        super().__init__()
+
+    def _get_admin_state(self) -> tuple[State, int]:


not sure I like this being private/protected/however the underscore is called

So far I see no reason why someone should be allowed to all USM(…).get_admin_state … then again this is only an example, so I don't care :D

Poolitzer · 2025-02-08T18:19:02Z

examples/fsmbot.py

+        super().__init__()
+
+    def _get_admin_state(self) -> tuple[State, int]:
+        return self._states[self.admin_id]


That should not be the private function right

Poolitzer · 2025-02-08T18:24:11Z

examples/fsmbot.py

+    def _get_admin_state(self) -> tuple[State, int]:
+        return self._states[self.admin_id]
+
+    def get_state_info(self, update: object) -> StateInfo[Optional[int]]:


The type hint here is linked to the FSM class type hint above right? I can't have one be a string one be an integer.

Correct, the type variable specifying the type of keys you use.

Poolitzer · 2025-02-08T18:55:58Z

telegram/ext/_application.py

+                if not state_handlers[group]:
+                    del state_handlers[group]


Does this remove the group if its empty? Can we comment it

yes, that's basically the same as state_handlers.pop(group, None), I think. It will call state_handlers.__del__(group). IMHO this is basic python syntax 😬

Poolitzer · 2025-02-08T19:05:30Z

telegram/ext/_fsm/machine.py

+
+    from telegram.ext import JobQueue
+
+_KT = TypeVar("_KT", bound=Hashable)


Im back with my thinking about private stuff. Why do we private it here, not in other files; should we. Hm.

I feel you :D if machine was a public module like telegram.constants, it would have to be protected IMO. Since it's not and we expose the relevant elements only via telegram.ext, it's not so important. Making it clearer what the content exposed by this module is still doesn't hurt.
If you want to go deeper into the rabbit hole: https://discuss.python.org/t/add-the-export-keyword-to-python/28444/6

Poolitzer · 2025-02-08T19:29:13Z

examples/fsmbot.py

+    application.fsm = UserSupportMachine(admin_id=123456)
+    application.fsm.set_job_queue(application.job_queue)


these should be part of the builder and the second one set by default imo

yes, absolutely. #PoC 👼

Poolitzer · 2025-02-08T19:43:49Z

telegram/ext/_fsm/machine.py

+        return f"FSM_Job_{'_'.join(str(hash(k)) for k in keys)}"
+
+    def set_job_queue(self, job_queue: "JobQueue") -> None:
+        self.__job_queue = weakref.ref(job_queue)


Why weakref wont it/the user experience break if the supplied job queue gets destroyed

There is job_queue.application. that's also weakref b/c we have a cyclic reference: application is application.job_queue.application. Now we also have application is application.fsm.job_queue.application. I'm not 100% sure if this is strictly necessary, but it seemed safer to me 🤔 I can at very least add a comment.

Poolitzer · 2025-02-08T19:44:08Z

telegram/ext/_fsm/machine.py

+            if raise_exception:
+                raise RuntimeError("JobQueue was garbage collected")


ha here see. why.

Poolitzer · 2025-02-08T19:47:29Z

telegram/ext/_fsm/states.py

+    def __init__(self, uid: Optional[str] = None):
+        effective_uid = uid or uuid4().hex
+        if effective_uid in self.__knows_uids:
+            raise ValueError(f"Duplicate UID: {effective_uid} already registered")


is the error saying UID correct? I would understand the optional type hint to be anything hashable

Sorry, I don't get you. What do you think is wrong? the phrasing of the error message?

Poolitzer · 2025-02-08T19:48:02Z

telegram/ext/_fsm/states.py

+
+
+class State(abc.ABC):
+    __knows_uids: ClassVar[set[str]] = set()


Wait I must really misunderstand smth, how does this work across states

uh … that's how class variables work? :D In the init self.__knows_uids is basically the same as State.__knows_uids …

Poolitzer · 2025-02-08T19:50:37Z

A thought I had: Since this is such a huge change for something so integral to our project, should we provide a beta version/implementation after we are satisfied with the status and announce it to get people to try it out and provide feedback before we make a first release?

Bibo-Joshi · 2025-02-08T21:17:47Z

A thought I had: Since this is such a huge change for something so integral to our project, should we provide a beta version/implementation after we are satisfied with the status and announce it to get people to try it out and provide feedback before we make a first release?

That is indeed a very good idea 👍
My thoughts on this are roughly like this:

the overall change should (hopefully) be such that nothing changes if you just don't use the new states.
that way we can introduce the refactoring & deprecate the existing CH without braking changes.
We can mark the status as "public beta feature" and at some point we can say "well, starting with version 25.0, states are now marked as stable".

that way we wouldn't have to do any additional actual beta releases.
If we prefer to have a proper beta release, I'm open to that. We could e.g. do states update on a 23.0bx line that gets only updates for states while other stuff (especially API updates only go into 22.x) . We shouldn't do that for too long IMO, we saw how much effort that takes with v20.
Encouraging people to try out a branch is ofc an easy option.

Poolitzer · 2025-02-08T21:22:37Z

I dont get the first point, what do you mean with not using new states. In conjunction with ConvHandler? I thought this is supposed to fully replace it. In that case I would like a big announcement at least (maybe also in readme or so).

We might not really need an extra release now that I think of it, just make a big fuss about it before calling it stable and doing the deprecation error warning.

Bibo-Joshi · 2025-02-08T21:25:35Z

I dont get the first point, what do you mean with not using new states. In conjunction with ConvHandler? I thought this is supposed to fully replace it. In that case I would like a big announcement at least (maybe also in readme or so).

Yes, this is supposed to replace CH. But, since this is acting on a different level than handlers, the conversationhandlerbot.py example will still run as before. We should throw hella-big warnings if states & CH are used in conjunction, true.
All other features that are independent of CH will also all run the same. all keys will always be in state "IDLE".

We might not really need an extra release now that I think of it, just make a big fuss about it before calling it stable and doing the deprecation error warning.

👍

Poolitzer · 2025-02-08T21:28:05Z

Ah yes you mean in the transition period alright. Okay so lets do a release when we are happy and announce that its there and see if we get feedback, and then do a deprecation later. Perfect.

Initial FSM PoC

0c06ba0

Bibo-Joshi added 🛠 refactor change type: refactor 🔌 enhancement pr description: enhancement labels Feb 4, 2025

Bibo-Joshi requested review from Poolitzer, harshil21 and clot27 February 4, 2025 20:44

This comment was marked as outdated.

Sign in to view

Bibo-Joshi added 8 commits February 5, 2025 10:52

Add State.ANY for fallbacks and allow handling multiple states for on…

07225b9

…e update

Temporarily Disable Tests on this branch

34832d9

Add Some Abstractions for Timeout Jobs

434cbfa

Disable tests harder …

817b71d

Move internal state storage to FiniteStateMachine and add state history

646ba37

Try setting up infrastructure for optimistic locking. Example will fo…

4441543

…llow

Add an example on concurrency in FSM

31af1a9

Use Lock instead of semaphore

b1fff6d

Poolitzer reviewed Feb 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial FSM PoC #4669

Initial FSM PoC #4669

Bibo-Joshi commented Feb 4, 2025 •

edited

Loading

This comment was marked as outdated.

Poolitzer left a comment

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer Feb 8, 2025

Bibo-Joshi Feb 8, 2025

Poolitzer commented Feb 8, 2025

Bibo-Joshi commented Feb 8, 2025

Poolitzer commented Feb 8, 2025

Bibo-Joshi commented Feb 8, 2025

Poolitzer commented Feb 8, 2025


		from telegram.ext import JobQueue

		_KT = TypeVar("_KT", bound=Hashable)

		application.fsm = UserSupportMachine(admin_id=123456)
		application.fsm.set_job_queue(application.job_queue)

		if raise_exception:
		raise RuntimeError("JobQueue was garbage collected")



		class State(abc.ABC):
		__knows_uids: ClassVar[set[str]] = set()

Initial FSM PoC #4669

Are you sure you want to change the base?

Initial FSM PoC #4669

Conversation

Bibo-Joshi commented Feb 4, 2025 • edited Loading

ToDos / Ideas / Thoughts

This comment was marked as outdated.

Poolitzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Poolitzer commented Feb 8, 2025

Bibo-Joshi commented Feb 8, 2025

Poolitzer commented Feb 8, 2025

Bibo-Joshi commented Feb 8, 2025

Poolitzer commented Feb 8, 2025

Bibo-Joshi commented Feb 4, 2025 •

edited

Loading