Feature/datetime conversions #174
Conversation
|
|
| @@ -28,6 +28,7 @@ | |||
| import google.protobuf.json_format | |||
| import google.protobuf.message | |||
| import google.protobuf.symbol_database | |||
| from dateutil import parser | |||
There was a problem hiding this comment.
This is a new dependency just for this feature. Is there any way we can avoid it? Is there any way with the standard library to do this, and/or is the parser easy to vendor/copy into here?
(also, a new dependency would need to be added to project.toml which you can do via poetry add dateutil and then re-alphabetizing)
There was a problem hiding this comment.
Looks like it may be unavoidable to add this dependency unfortunately. Can't easily vendor stdlib impl as it's in C (ref python/cpython#92177). But can we put a if if sys.version_info >= (3, 11): here to use the stdlib version? (also don't forget to add that dependency and to run poe format).
Also, this will need to land after (and merge main after) #172 is merged which will add Python 3.11 support.
There was a problem hiding this comment.
Not an issue on my side, I must have been working on a tainted env because I thought dateutil was in standard. Will rebuild and double-check.
For those wondering about why this is an issue:
https://stackoverflow.com/questions/127803/how-do-i-parse-an-iso-8601-formatted-date
https://stackoverflow.com/questions/969285/how-do-i-translate-an-iso-8601-datetime-string-into-a-python-datetime-object
On doing it within the standard library, we could hack around with the following:
# Example lookup for Z
map_datetime = {
"Z": "+00:00"
}
# ...
# v = "2020-01-01T00:00:00Z"
if v[-1].isalpha() and len(v) == 20:
v = v[:-1] + map_datetime[v[-1].upper()]
# Code as normal
There's a slight catch here that "J" = local time, so we'd have to decide if we treat that string as timezone unaware or pull the timezone from the local machine.
The only reason I avoided this was to rely on a battle-tested date parser that could potentially catch a wider range of formats coming in from other languages. If you're confident we'll only see letters coming in (outside of the +/- hours format) then I'm happy to switch over to an explicit remap of the last digits for the sake of avoiding the external dependency
There was a problem hiding this comment.
I think dateutil is probably an acceptable solution since it doesn't seem to carry dependencies on its own or move much.
There was a problem hiding this comment.
Roger that, will add in the version check and wait for the 3.11 updates you guys have planned
There was a problem hiding this comment.
Added a couple of suggestions and need to merge main, otherwise all good. If the poetry.lock conflict is too annoying to fix, feel free to close this PR and open a new one on a fresh branch off main. Alternatively, you can wait a bit until I get around to this issue where I'll basically just copy what you did here. Thanks!
EDIT: Now that I think about it, we can just use this dependency on < 3.11. See https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies
| @@ -1,20 +1,20 @@ | |||
| [tool.poetry] | |||
There was a problem hiding this comment.
Why did so much of this file change? Not that it seems wrong, just doesn't seem like the issue to do this in.
There was a problem hiding this comment.
Auto-sorting...I noticed you asked to re-alphabetise, I didn't realise the autoformatter did so much! Will revert
There was a problem hiding this comment.
Sorry, I meant just the dependency, heh
| @@ -164,6 +165,37 @@ def test_encode_search_attribute_values(): | |||
| temporalio.converter.encode_search_attribute_values(["foo", 123]) | |||
|
|
|||
|
|
|||
| def test_decode_search_attributes(): | |||
There was a problem hiding this comment.
Is it possible to also add an integration test that sets a datetime attribute in a workflow? Or maybe alter test_workflow_search_attributes so that it will break without your fix?
|
Closing PR to bypass merge conflicts, new issue on #179 |
What was changed
The date parsing system in
temporalio.converter:decode_search_attributesis switched fromdatetime.datetime.fromisoformattodateutils.parser.isoparse. This provides support for ISO8601 compliant datetime strings with timezones attachedWhy?
Python's datetime library is not timezone aware by default, with issues on StackOverflow going back as far as 2008. Prior to Python 3.11,
datetime.datetime.fromisoformatwas not designed to handle timezone-aware datetime strings, but rather the timezone-unaware outputs ofdatetime.isoformat, which was not ISO compliant! (Hardest problem in SE is naming after all.)The new Scheduling features in Temporal handle timezone-aware schedules by default. This caused issues when communicating with the Python SDK, as it could not decode the timezone strings (typically appended with 'Z' for UTC time).
dateutils.parser.isoparseis known to have some problems with greedy evaluation of date strings but this typically occurs in the presence of user input. It should not be a problem for machine-formatted strings.While the release of 3.11 means this won't be an issue in future, much of the Python and cloud ecosystem relies on older versions of Python. This change means most current versions of Python can leverage the scheduling functionality.
Checklist
Closes Search attribute Datetime need to support full ISO 8601 (and maybe more) #144
How was this tested:
No