-
Notifications
You must be signed in to change notification settings - Fork 103
Environment configuration #895
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make sure you get rid of those .DS_Store macOS file nonsense. They should be globally excluded by gitignore.
I dunno if we need to do this in this PR, we can wait and see what @cretz and @dandavison opinions are too, but IMO it would make good sense to add a Client.connect()
overload that takes a ClientConfigProfile
here for easy use.
Also these guys need experimental notices on them too
Whoops, didn't realize I had committed them. Removed, added to gitignore
seems reasonable, will wait to chad gets back as i think he'll have more to suggestions anyway
added to each class (not sure if each method is necessary?) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking good to me, but have Dan or Chad when he's back take a look
temporalio/envconfig.py
Outdated
DataSource: TypeAlias = Union[ | ||
str, bytes | ||
] # str represents a file path, bytes represents raw data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bit confusing. It is very normal to provide PEM file contents as a string in languages. We should differentiate file path better. Maybe Union[pathlib.Path, str, bytes]
and only paths are treated as such?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've changed the typing to your suggestion but I'm unsure whether or not to keep the str
type.
The core TOML parser reads the cert configs into:
pub enum DataSource {
Path(String),
Data(Vec<u8>),
}
so lang only sees these cert configs as bytes or a path (which we now read as an actual Path
type).
Users can provide file contents as string but it would only be possible programmatically, like the (added) test test_read_source_from_string_content
(unless I'm misunderstanding something). I am not sure if we wanted envconfig to be used this way
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The core TOML parser reads the cert configs into:
Rust has a clearer difference between paths and data. It is normal in Python to have a string of data. test_read_source_from_string_content
is actually test_read_source_from_bytes_content
because you have a b
in front. I think it makes sense to accept string content too since PEM data is often a string, but if not, no prob, let's just make sure the exception is clear if the data type is wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test_read_source_from_string_content
reads in string content, but asserts that it is read correctly into bytes.
So it's still from string content so to speak.
temporalio/envconfig.py
Outdated
return source.encode("utf-8") | ||
if isinstance(source, bytes): | ||
return source | ||
return None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should not silently drop/ignore invalid data types in this case at runtime, should raise here IMO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
raise a TypeError
if invalid
from typing_extensions import TypeAlias, TypedDict | ||
|
||
import temporalio.service | ||
from temporalio.bridge.temporal_sdk_bridge import envconfig as _bridge_envconfig |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it may make more sense to have a temporalio/bridge/envconfig.py
module that wraps all the bridge stuff. Every other use of the bridge in this repo goes through that approach. But not required.
temporalio/envconfig.py
Outdated
def load_client_connect_config( | ||
profile: str = "default", | ||
*, | ||
env_vars: Optional[Mapping[str, str]] = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we want this here, can this be moved to the bottom of the kwarg list and be called override_env_vars
to be clear that it's abnormal to set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
moved/renamed
tests/test_envconfig.py
Outdated
ClientConfig.load_profile_from_data(toml_config) | ||
|
||
|
||
def test_load_client_connect_config(base_config_file: Path): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you have somewhere that splats this on to .connect
and makes an actual connection (say to the same dev server used in other tests or start your own if you must)? We need something to confirm type checking on the result of the load_client_connect_config
call that it matches the connect
arg expectations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added
temporalio/envconfig.py
Outdated
return ClientConfig._from_bridge_profiles(loaded_profiles) | ||
|
||
@staticmethod | ||
def load_profiles_from_file( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we can combine all of these load
calls into one to reduce surface area since they are not the commonly called methods. I know Go doesn't combine them because it's a bit harder for that language. I figure you have one load
call that can accept config file override (can be path or data like cert), disable env, override env, etc.
As for the load_profile
stuff, wouldn't that just be load
on ClientConfigProfile
instead of this class? Most of the same parameters should apply.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done, some changes to the bridge to support this
temporalio/envconfig.py
Outdated
|
||
|
||
@dataclass | ||
class ClientConfig: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We want users to be able to convert this (and ClientConfigProfile
) to/from TOML on their own if they want. However, we don't necessarily need to expose literal TOML conversion, just the ability to convert to/from the dict
that represents TOML (they can use some Python TOML library if they want to do the TOML part). Can we expose to_dict
and from_dict
on this and the ClientConfigProfile
class that represents exactly what it looks like in TOML?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added to_dict
methods
the resulting dictionaries are well-typed, not sure if that was necessary or not
one caveat with this, when the given config input is in bytes
to_dict
converts bytes data into strings because strings are more suitable for TOML
I wrote it in a test for this, converting from to_dict
-> from_dict
-> compare:
# We expect the new profile to be the same, but with bytes-based data
# sources converted to strings. This is because to_dict converts
# bytes-based data to a string, suitable for TOML. So we only have
# a string representation to work with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! This is a good model to follow for other SDKs. Only one class name change request (and two other unimportant comments).
temporalio/envconfig.py
Outdated
|
||
|
||
# We define typed dictionaries for what these configs look like as TOML. | ||
class ClientTLSConfigDict(TypedDict, total=False): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was originally going to say you could leave these out since users using to/from dict should not really care about the types since that's low level. But I now see that we also are dogfooding the dict serialization stuff (awesome), so can leave even if just for our benefit.
) | ||
|
||
|
||
class ClientConnectConfig(TypedDict, total=False): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a heads up here, I think we may be having another situation ("plugins") where this actually has value at the client level for others wanting to type a dict for the kwargs of connect/init. So this may move to the client
module in the future.
…ntal to class docstrings, misc
…client_connect_config actually matches connect arguments
What was changed
Environment configuration for Python SDK, using core base envconfig module
Closes [Feature Request] Environment configuration #835
How was this tested:
test_envconfig.py
Any docs updates needed?
probably