Thanks to visit codestin.com
Credit goes to github.com

Skip to content

add dev container run script #8994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 4, 2023
Merged

add dev container run script #8994

merged 5 commits into from
Sep 4, 2023

Conversation

thrau
Copy link
Member

@thrau thrau commented Aug 26, 2023

Motivation

This PR introduces a script for developers that aims to replace bin/docker-run.sh and ultimately all its derivatives.

We have several ways of starting what I call a "dev container", i.e., a localstack container with various mounted things like source folders or dependencies. Here are some:

  • bin/localstack-start-docker-dev.sh
  • make docker-mount-run
  • make test-docker-mount
  • make docker-run-tests
  • test-docker-mount-code

In ext:

  • bin/docker-run.sh
  • bin/test-in-docker.sh
  • make docker-mount-run
  • ...

With the introduction of #8781 and #9050, there was an opportunity to unify some of the code necessary to start and configure such a dev container.

Moreover, with the growing amount of bootstrap tests (network configs, extensions, persistence, ...) there was a need to run these tests locally without having to build the image every time. So whatever configuration logic needed for a docker-run.sh replacement could be used there as well.

Usage

This is what it looks like when you run a localstack container. The only main difference to localstack start is that it also mounts the source folders by default. The command also outputs the container config in a more human readable way:

Screenshot at 2023-09-04 00-32-55

Here's the help:

 % python -m localstack.dev.run --help                     
Usage: python -m localstack.dev.run [OPTIONS] [COMMAND]...

  A tool for localstack developers to start localstack containers. Run this in
  your localstack or localstack-ext source tree to mount local source files or
  dependencies into the container. Here are some examples::

      python -m localstack.dev.run
      python -m localstack.dev.run -e DEBUG=1 -e LOCALSTACK_API_KEY=test
      python -m localstack.dev.run -- bash -c 'echo "hello"'

  Explanations and more examples:

  Start a normal container localstack container. If you run this from the
  localstack-ext repo, it will start localstack-pro::

      python -m localstack.dev.run

  If you start localstack-pro, you might also want to add the API KEY as
  environment variable::

      python -m localstack.dev.run -e DEBUG=1 -e LOCALSTACK_API_KEY=test

  If your local changes are making modifications to plux plugins (e.g., adding
  new providers or hooks), then you also want to mount the newly generated
  entry_point.txt files into the container::

      python -m localstack.dev.run --mount-entrypoints

  Start a new container with randomized gateway and service ports, and
  randomized container name::

      python -m localstack.dev.run --randomize

  You can also run custom commands:

      python -m localstack.dev.run bash -c 'echo "hello"'

  Or use custom entrypoints:

      python -m localstack.dev.run --entrypoint /bin/bash -- echo "hello"

  You can also mount local dependencies (e.g., pytest and other test
  dependencies, and then use that in the container)::

      python -m localstack.dev.run --mount-dependencies \
          -v ./tests:/opt/code/localstack/tests \
          -- .venv/bin/python -m pytest tests/unit/http_/

  The script generally assumes that you are executing in either localstack or
  localstack-ext source repositories that are organized like this::

      somedir                              <- your workspace directory
      ├── localstack                       <- execute script in here
      │   ├── ...
      │   ├── localstack                   <- will be mounted into the container
      │   ├── localstack_core.egg-info
      │   ├── setup.cfg
      │   ├── tests
      │   └── ...
      ├── localstack-ext                   <- or execute script in here
      │   ├── ...
      │   ├── localstack_ext               <- will be mounted into the container
      │   ├── localstack_ext.egg-info
      │   ├── setup.cfg
      │   ├── tests
      │   └── ...
      ├── moto
      │   ├── AUTHORS.md
      │   ├── ...
      │   ├── moto                         <- will be mounted into the container
      │   ├── moto_ext.egg-info
      │   ├── setup.cfg
      │   ├── tests
      │   └── ...

Options:
  --image TEXT                    Overwrite the container image to be used
                                  (defaults to localstack/localstack or
                                  localstack/localstack-pro.
  --volume-dir DIRECTORY          The localstack volume on the host, default:
                                  ~/.cache/localstack/volume
  --pro / --community             Whether to start localstack pro or
                                  community. If not set, it will guess from
                                  the current directory
  --randomize                     Randomize container name and ports to start
                                  multiple instances
  --mount-source / --no-mount-source
                                  Mount source files from localstack,
                                  localstack-ext, and moto into the container.
  --mount-dependencies / --no-mount-dependencies
                                  Whether to mount the dependencies of the
                                  current .venv directory into the container.
                                  Note this only works if the dependencies are
                                  compatible with the python and platform
                                  version from the venv and the container.
  --mount-entrypoints / --no-mount-entrypoints
                                  Mount entrypoints
  --mount-docker-socket / --no-docker-socket
  -e, --env TEXT                  Additional environment variables that are
                                  passed to the LocalStack container
  -v, --volume TEXT               Additional volume mounts that are passed to
                                  the LocalStack container
  -p, --publish TEXT              Additional ports that are published to the
                                  host
  --entrypoint TEXT               Additional entrypoint flag passed to docker
  --network TEXT                  Docker network to start the container in
  --help                          Show this message and exit.

Changes

  • Add a comprehensive list of ContainerConfigurator implementations to encapsulate the configuration logic needed by the various startup scenarios, and add the new localstack.dev.run script
  • add run_interactive method to localstack.utils.run to attach the tty to a subprocess (in this case docker run)
  • make sure stderr is correctly piped to stdout when using stream_container_logs with the CmdDockerClient

Testing

  • Check out the branch
  • source .venv/bin/activate
  • python -m localstack.dev.run --help

To test ext:

  • keep the .venv of localstack activated
  • navigate into localstack-ext
  • python -m localstack.dev.run -e LOCALSTACK_API_KEY=test should now start a pro container

Future work

Follow up PRs will include:

  • consolidating/re-organizing some of the configurator logic introduced in add container configurators for bootstrapping localstack #9050.
  • making it easier to inject test dependencies (pytest, coverage, ...) in a clean way (for instance using extensions or the python package installer in a "sidecar" venv on the host) to replace test-in-docker.sh
  • actually replace all the different script listed in the motivation

TODO

@thrau thrau force-pushed the dev-container-config branch from e427f5e to e35e3d2 Compare August 26, 2023 18:26
@thrau thrau added the semver: patch Non-breaking changes which can be included in patch releases label Aug 26, 2023
@thrau thrau requested a review from simonrw August 26, 2023 18:27
@coveralls
Copy link

coveralls commented Aug 26, 2023

Coverage Status

coverage: 80.376% (-0.4%) from 80.759% when pulling 5ae16f8 on dev-container-config into 6f5d756 on master.

@github-actions
Copy link

github-actions bot commented Aug 26, 2023

LocalStack Community integration with Pro

       2 files         2 suites   1h 27m 7s ⏱️
2 152 tests 1 676 ✔️ 476 💤 0
2 153 runs  1 676 ✔️ 477 💤 0

Results for commit c2c6a89.

♻️ This comment has been updated with latest results.

@thrau thrau force-pushed the dev-container-config branch from e35e3d2 to 5ae16f8 Compare August 29, 2023 18:23
@thrau thrau force-pushed the dev-container-config branch 2 times, most recently from 1e464e3 to c2c6a89 Compare September 3, 2023 23:11
@thrau thrau marked this pull request as ready for review September 3, 2023 23:29
Copy link
Contributor

@simonrw simonrw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, just a couple of comments about finding the venv dir.

Copy link
Member

@dfangl dfangl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice changes 👍

@thrau thrau force-pushed the dev-container-config branch from c2c6a89 to 3e39943 Compare September 4, 2023 20:33
@thrau thrau merged commit 83274fa into master Sep 4, 2023
@thrau thrau deleted the dev-container-config branch September 4, 2023 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants