remove install dependency on localstack_client #8307

alexrashed · 2023-05-12T09:46:19Z

This PR contains the following changes:

It moves the dependency on localstack_client (and its transitive dependency boto3 from install to the runtime extra).
- It also cleans up the dependencies a bit (moving apispec to runtime, removes an unnecessary version pin on amazon_kclpy).
- This effectively reduces the CLI dependencies (because the install dependencies currently are the CLI dependencies).
It replaces the legacy service:port mapping from the localstack-client with an dynamically detected list of services based on the loaded plugins.
- This means that the localstack-client does not have to be changed and released when introducing a new service emulator in LocalStack.
- The ports from the localstack-client would only be used with the environment variable USE_LEGACY_PORTS, but these wouldn't work with the new "one endpoint for everything" anyways.
- In the long run, we should try to deprecate the rest of localstack-client as well (or maybe completely strip it down to the part patching boto3), however, this is out of scope for this PR. It's rather a first step towards that direction.
Updates the list of "official localstack images" to the new images defined in v2 (this is used by the validation command in the CLI).

I am happy for any feedback!

github-actions · 2023-05-12T10:47:46Z

LocalStack Community integration with Pro

2 043 tests 1 759 ✔️ 1h 20m 59s ⏱️
      2 suites   284 💤
      2 files       0 ❌

Results for commit 71a5b7c.

♻️ This comment has been updated with latest results.

whummer

Nice set of changes, kudos for tackling this cleanup @alexrashed ! 🧹 Added two clarification questions at this point.

Overall, I'm super happy that we're getting rid of localstack-client 🚀 and it looks good to me in general, but will let @thrau do the detailed review/signoff.. 👍

whummer · 2023-05-12T11:39:10Z

localstack/utils/bootstrap.py

@@ -213,7 +214,13 @@ def get_enabled_apis() -> Set[str]:

    The result is cached, so it's safe to call. Clear the cache with get_enabled_apis.cache_clear().
    """
-    return resolve_apis(config.parse_service_ports().keys())
+    services = os.environ.get("SERVICES", "").strip()


I guess we still need to split the string here? (similar to what we had in parse_service_ports() before)

Good point. We would not support the different service ports anyway, but I re-added the parsing (but the ports will just be discarded).

whummer · 2023-05-12T11:44:38Z

localstack/config.py

@@ -1016,6 +1014,44 @@ def get_gateway_listen(gateway_listen: str) -> List[HostAndPort]:
 # Whether to return and parse access key ids starting with an "A", like on AWS
 PARITY_AWS_ACCESS_KEY_ID = is_env_true("PARITY_AWS_ACCESS_KEY_ID")

+# List of services supported by LocalStack
+SUPPORTED_SERVICES = [


Wondering if we should enumerate the list of services explicitly here - would be nice if we can somehow integrate this with the service plugin mechanism (or even enumerate the services from botocore directly). (maybe @thrau has some good ideas here..?)

I thought about this, but the problem is that this is already used when populating the service specific configs (<service>_PROVIDER_OVERRIDE and others) and this list is actually used in controlling the services which should be loaded. That made it a bit too risky for me in the first iteration...
But I am happy for any suggestions and would love to get rid of this list all together!

Might this be a way forward? 5766efb

the populate_config_env_var_names is mostly used for choosing which env vars should be handed to the container. here it makes sense to not use the service plugin manager yet, since this is called in the config.py. however, i think it would be OK to be a bit more permissive, and just use a heuristic based on the key syntax. worst case the container sees some variables it doesn't need. i feel this is an acceptable trade off if it means we don't need to maintain a list of supported services.

The get_enabled_apis control path is only used when already starting the infra and EAGER_SERVICE_LOADING is set to 1, where we've already called SERVICE_PLUGINS.list_available() before, so this seems like a no-brainer

Great! Thanks a lot for the commit. I just directly cherry-picked it from your branch.

thrau

Overall the changes look good to me. Quickly tested with the legacy s3 and lambda providers and they still seem to work 👍 I agree with Waldemar that it would be better to use this opportunity to get rid of a static list of services. We don't really need it and if we merge it now it's just going to stick around.

I made a suggestion of how to proceed in an inline comment.

alexrashed · 2023-05-15T14:31:24Z

I re-added the port parsing in the SERVICES env var (but the ports will actually be discarded).
I also cherry-picked the suggested change by @thrau. I added a small fix (see the last commit) which ensures that:

We actually add the prefixed env vars to the CONFIG_ENV_VARS (instead of replacing the list).
I removed adding env vars which end with _BACKEND and _PORT_EXTERNAL because these are not used anywhere anymore anyways.

thrau

nice! thanks for going the extra mile here. let's

alexrashed added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label May 12, 2023

alexrashed requested a review from whummer May 12, 2023 09:46

alexrashed requested a review from thrau as a code owner May 12, 2023 09:46

alexrashed marked this pull request as draft May 12, 2023 09:47

whummer reviewed May 12, 2023

View reviewed changes

thrau reviewed May 13, 2023

View reviewed changes

alexrashed and others added 4 commits May 15, 2023 15:11

remove install dependency on localstack_client

5e9656d

fix dependency of string utils to botocore

013bccd

remove list of supported services

08bf515

re-add parsing service ports, re-add unit tests

41c12d7

alexrashed force-pushed the remove-localstack-client-install-dep branch from a485f50 to 41c12d7 Compare May 15, 2023 13:42

fix config var population

71a5b7c

thrau approved these changes May 15, 2023

View reviewed changes

alexrashed marked this pull request as ready for review May 15, 2023 14:34

alexrashed merged commit e509193 into master May 16, 2023

alexrashed deleted the remove-localstack-client-install-dep branch May 16, 2023 06:23

alexrashed mentioned this pull request May 16, 2023

remove pin on botocore #8321

Merged

alexrashed mentioned this pull request May 25, 2023

exclude botocore and boto3 dependency localstack/localstack-cli#5

Closed

alexrashed mentioned this pull request Sep 25, 2023

avoid setting LOCALSTACK_CLI in the container #9223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

remove install dependency on localstack_client #8307

remove install dependency on localstack_client #8307

Uh oh!

alexrashed commented May 12, 2023 •

edited

Loading

Uh oh!

github-actions bot commented May 12, 2023 •

edited

Loading

Uh oh!

whummer left a comment •

edited

Loading

Uh oh!

whummer May 12, 2023

Uh oh!

alexrashed May 15, 2023

Uh oh!

whummer May 12, 2023 •

edited

Loading

Uh oh!

alexrashed May 12, 2023

Uh oh!

thrau May 13, 2023 •

edited

Loading

Uh oh!

alexrashed May 15, 2023

Uh oh!

thrau left a comment

Uh oh!

alexrashed commented May 15, 2023

Uh oh!

thrau left a comment

Uh oh!

Uh oh!

Uh oh!

remove install dependency on localstack_client #8307

remove install dependency on localstack_client #8307

Uh oh!

Conversation

alexrashed commented May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

whummer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whummer May 12, 2023

Choose a reason for hiding this comment

Uh oh!

alexrashed May 15, 2023

Choose a reason for hiding this comment

Uh oh!

whummer May 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrashed May 12, 2023

Choose a reason for hiding this comment

Uh oh!

thrau May 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexrashed May 15, 2023

Choose a reason for hiding this comment

Uh oh!

thrau left a comment

Choose a reason for hiding this comment

Uh oh!

alexrashed commented May 15, 2023

Uh oh!

thrau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexrashed commented May 12, 2023 •

edited

Loading

github-actions bot commented May 12, 2023 •

edited

Loading

whummer left a comment •

edited

Loading

whummer May 12, 2023 •

edited

Loading

thrau May 13, 2023 •

edited

Loading