State persistence #955
State persistence #955
Conversation
PR Change SummaryImplemented state persistence functionality in the graph processing system, enhancing the ability to track and manage state throughout the execution of graph nodes.
Modified Files
How can I customize these reviews?Check out the Hyperlint AI Reviewer docs for more information on how to customize the review. If you just want to ignore it on this PR, you can add the Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add What is Hyperlint?Hyperlint is an AI agent that helps you write, edit, and maintain your documentation. Learn more about the Hyperlint AI reviewer and the checks that we can run on your documentation. |
Docs Preview
|
| state: StateT, | ||
| deps: DepsT, | ||
| auto_instrument: bool, | ||
| snapshot_id: str | None = None, |
There was a problem hiding this comment.
It seems weird to me to have snapshot_id be a kwarg to GraphRun. Maybe you can explain this to me synchronously, but I feel like one of three things should be the case:
- We can rework things so the
GraphRundoesn't need asnapshot_idargument - We can rework the name of the
snapshot_idargument to be more clear what its purpose is - My mental model about what is happening is way off
Regardless of which is right I think a discussion may get us to the bottom of this fairly quickly.
There was a problem hiding this comment.
Discussed over slack, but basically I think we're in agreement on the following:
- we should move some of the APIs to GraphRun instead of on
run - we should make the connection between runs and persistence more explicit in the docs
- we should make "thread creation" one of the methods on the Persistence ABC so that it's more clear how we intend for that to be done (basically, it gives people who want to implement it using postgres a more thorough starting point). (Not sure if you agreed with this but this was the point I was trying to make, if you had something different in mind that's fine, I'm just writing this for the sake of future reference.)
Just ping me once the GraphRun-related changes are made and I'll review further, I think that will get things to be basically in a good state though.
Co-authored-by: David Montague <[email protected]>
Co-authored-by: David Montague <[email protected]>
fix #695.