This PR finalizes the changes to the filesystem hierarchy we started in #5011 and introduces a semantically well-defined filesystem hierarchy for the localstack system. It cleanly separates directories created on the host (for CLI users or docker users mounting directories into the container), from the filesystem hierarchy in localstack. For host mode, we now use ./.filesystem as filesystem root. The changes can be disabled by setting LEGACY_DIRECTORIES=1.

LocalStack filesystem hierarchy

The hierarchy inside the container will be as follows:

• /var/lib/localstack: the main localstack volume that stores data across localstack runs, this is what user's are interested in

  • /var/lib/localstack/lib: variable packages (like lazy-loaded third-party dependencies, also referred to as var_libs)
  • /var/lib/localstack/logs: logs of the recent localstack run
  • /var/lib/localstack/state: persistent state of third-party services (or what we sometimes call "data" or "assets", e.g., opensearch cluster data)
  • /var/lib/localstack/tmp: temporary data that does not survive localstack runs (is cleared when localstack starts)
  • /var/lib/localstack/cache: temporary data that survives localstack runs (is not cleared when localstack starts)

• /usr/lib/localstack: static third-party packages installed into the container images. this was previously "localstack/infra", and is also referred to as static_libs

• /etc/localstack: 🚧 localstack configuration dir (for future use, currently not in use)

  • /etc/localstack/conf.d: localstack configuration overwrites
  • /etc/localstack/init/: localstack initialization hooks

User experience

From a user perspective, it should be similar to the mariadb and other database containers, where you mount a local directory into /var/lib/localstack, which persist all the data required across localstack runs.
Currently this is achieved by this line:

Users only need to change that to /my/local/localstack/volume:/var/lib/localstack, where the host path could still be /tmp/localstack.

DATA_DIR and HOST_TMP_FOLDER are no longer used:

• DATA_DIR points implicitly to /my/local/localstack/volume/state. if you want to use persistence, set PERSISTENCE=1. if you set DATA_DIR, then we have a deprecation path that we log a warning that DATA_DIR is ignored, but PERSISTENCE is set to 1
• HOST_TMP_FOLDER is determined by inspecting the volume mounts and using the source of the bind mount to /var/lib/localstack.

what they will see is something like this:

thomas@ninox /tmp/localstack % tree -L 3
.
├── cache
│   ├── machine.json
│   └── localstack.opensearch-1.1.0-linux-x64.tar.gz
├── lib
│   └── opensearch
│       └── 1.1.0
├── logs
│   ├── localstack_infra.err
│   └── localstack_infra.log
├── state
└── tmp
    └── zipfile.4986fb95 # <- lambda code

CLI users

Previously, each path was individually configurable (like DATA_DIR, VAR_LIBS, STATIC_LIBS, etc..). this is no longer supported or needed. the only variable that needs to be set is LOCALSTACK_VOLUME_DIR, which is essentially turns into the docker flag -v ${LOCALSTACK_VOLUME_DIR}:/var/lib/localstack. by default, LOCALSTACK_VOLUME_DIR=/tmp/localstack, but that could change to ~/.cache/localstack/volume (~/.cache/localstack is also associated with the CLI)

Developer experience (host mode)

when running in host mode, we don't want to use /usr/lib/localstack or /var/lib/localstack, etc on the host, since those are associated with the system and root permissions. instead, we simply move the filesystem hierarchy root to the environment variable FILESYSTEM_ROOT, which is by default ./.filesystem (from the localstack project root).
the hierarchy will look something like this (from the localstack project root):

thomas@ninox ~/workspace/localstack/localstack
 % tree .filesystem -L 4
.filesystem
├── etc
│   └── localstack
│       ├── conf.d
│       └── init
├── usr
│   └── lib
│       └── localstack
│           ├── amazon-kinesis-client
│           ├── aws-lambda-rie
│           ├── dynamodb
│           ├── kinesis-mock
│           ├── kms
│           ├── lambda.cfn-response.js
│           ├── localstack-utils-fat.jar
│           └── stepfunctions
└── var
    └── lib
        └── localstack
            ├── cache
            ├── state
            ├── lib
            ├── logs
            └── tmp

Developer experience (building)

When building the docker image, we need to do some installation prep of third-party libraries on the host before invoking docker build.
We piggy-back on the host-mode .filesystem directory, and simply ADD .filesystem/usr/lib/localstack /usr/lib/localstack.
This is not great IMO because it mixes concerns (host mode runtime with build time dependency management), but this is also they way it works right now.

PR Changes

Rundown of major changes:

• the main volume target user is no longer /tmp/localstack but /var/lib/localstack, so /tmp/localstack:/tmp/localstack volume mounts need to change to /tmp/localstack:/var/lib/localstack
• config.dirs.tmp is now cleared on every startup and shutdown
• static libs are now /usr/lib/localstack (./.filesystem/usr/lib/localstack in host mode or when running make init)
• var libs are now /var/lib/localstack/libs
• the config.dirs.function dir is no longer needed. it was conceived at the time as a directory in the container where lambda code goes (instead of tmp), but was later used as config variable to hold the HOST_TMP_PATH, which is not needed (since we are inspecting the container now to find the value)
• introduce config variable LEGACY_DIRECTORIES that can be activated as feature flag to enable to old mode
• using the CLI to start localstack no longer creates the directories on the host. directories are only created when actually running localstack, or when we're in the container.
• changed download destination to config.dirs.cache so downloaded archives are kept in case installation was incomplete

Rundown of other changes

• remove explicit elasticmq download from Dockerfile into static_libs, and move it into var_libs (as part of phase-out)
• create soft link from /opt/code/localstack/localstack/infra -> /usr/lib/localstack for backwards compatibility
• move supervisord logs from /tmp/localstack_infra.log to /var/lib/localstack/logs, but create symlinks to not break peoples code

Limitations and future work

• The separation of CLI and runtime configuration is still not clear enough for me. No directory configuration should be shared between the CLI and the runtime IMO.
• There are lots of instances of the form (conceptually) data_store = config.dirs.data or config.dirs.tmp, these should slowly be migrated to simply use config.dirs.data (which will point to a temporary directory if PERSISTENCE=0), or use the config.PERSISTENCE flag if they want to set data_store (or whatever they are using) to None.
• The package directory var_libs and static_libs still have no structure to them, I think we should introduce some well defined package index (like site-packages, node_modules, or maven's .m2)