localstack · joe4dev · Nov 30, 2023 · Nov 28, 2023 · Nov 28, 2023 · Nov 29, 2023
diff --git a/.github/workflows/markdown.links.config.json b/.github/workflows/markdown.links.config.json
@@ -47,6 +47,12 @@
         },
         {
             "pattern": "https://localhost:45139/"
+        },
+        {
+            "pattern": "https://www.cloudflare.com/learning/dns/what-is-1.1.1.1/"
+        },
+        {
+            "pattern": "https://github.com/boto/boto3/blob/develop/CHANGELOG.rst#L892"
         }
     ],
     "httpHeaders": [

diff --git a/content/en/references/configuration.md b/content/en/references/configuration.md
@@ -304,11 +304,16 @@ To learn more about these configuration options, see [DNS Server]({{< ref "dns-s
 
 | Variable | Example Values | Description |
 | - | - | - |
-| `DNS_ADDRESS` | 0.0.0.0 (default) | Address the LocalStack should bind the DNS server on (port 53 tcp/udp). Value `0` to disable.
-| `DNS_SERVER` | 8.8.8.8 (default) | Fallback DNS server for non-modified queries.
-| `DNS_RESOLVE_IP` | 127.0.0.1 | IP address the DNS integration should return as A record for modified queries. This will override any automatic detection of the proper response IP.
-| `DNS_LOCAL_NAME_PATTERNS` | | Names which should be resolved to the LocalStack IP, as python-compatible regex.
+| `DNS_ADDRESS` | `0.0.0.0` (default) | Address the LocalStack should bind the DNS server on (port 53 tcp/udp). Value `0` to disable.
+| `DNS_SERVER` | Default upstream DNS or `8.8.8.8` (default) | Fallback DNS server for queries not handled by LocalStack.
+| `DNS_RESOLVE_IP` | `127.0.0.1` (default) | IP address the DNS server should return as A record for queries handled by LocalStack. If customized, this value will be returned in preference to the DNS server response.
+| `DNS_LOCAL_NAME_PATTERNS` | `.*(ecr\|lambda).*.amazonaws.com` (example) | Skiplist of hostnames that should *NOT* be resolved to the LocalStack container, as a comma-separated list of Python-flavored regex patterns.
 
+## Transparent Endpoint Injection
+
+| Variable | Example Values | Description |
+| - | - | - |
+| `DISABLE_TRANSPARENT_ENDPOINT_INJECTION` | `0` (default in Pro) \| `1` | Whether to disable DNS resolution of AWS hostnames to the LocalStack container. Pro feature. (see [Transparent Endpoint Injection]({{< ref "user-guide/tools/transparent-endpoint-injection" >}}))
 
 ## LocalStack Pro
 

@@ -15,7 +15,7 @@ This documentation provides step-by-step guidance on how to access LocalStack se
 Suppose you have LocalStack installed on your machine and want to access it using the AWS CLI. To connect, you must expose port 4566 from your LocalStack instance and connect to `localhost` or a domain name that points to `localhost`. While the LocalStack CLI does this automatically, when running the Docker container directly or with docker compose, you must configure it manually. Check out the [getting started documentation]({{< ref "getting-started/installation" >}}) for more information.
 
 {{<alert title="Note">}}
-Additionally, if you bind a domain name to `localhost`, ensure that you are not subject to [DNS rebind protection]({{< ref "user-guide/tools/transparent-endpoint-injection/dns-server#dns-rebind-protection" >}}).
+If you bind a domain name to `localhost`, ensure that you are not subject to [DNS rebind protection]({{< ref "dns-server#dns-rebind-protection" >}}).
 {{</alert>}}
 
 You can also use the `GATEWAY_LISTEN` [configuration variable]({{< ref "references/configuration" >}}) to change the exposed port if necessary.

@@ -12,15 +12,16 @@ Suppose you're attempting to access LocalStack, but you're relying on transparen
 
 {{< figure src="../images/2.svg" width="400" >}}
 
-If you're using LocalStack with an [auth token]({{<ref "getting-started/auth-token">}}), then you can utilize the [DNS server]({{<ref "user-guide/tools/transparent-endpoint-injection/dns-server">}}) to perform requests to LocalStack as if it were AWS.
+If you're using LocalStack with an [auth token]({{<ref "getting-started/auth-token">}}), then you can utilize the [DNS server]({{<ref "dns-server">}}) to perform requests to LocalStack as if it were AWS.
 You need to make two changes:
 
 * Publish port 53 from the LocalStack docker container to your host.
 * Configure your host to use the LocalStack DNS server by default.
 
-For more details, see your [DNS server documentation]({{<ref "user-guide/tools/transparent-endpoint-injection/dns-server">}}).
+For more details, see your [DNS server documentation]({{<ref "dns-server">}}).
 
-For the community edition of LocalStack, you can employ your own DNS server to achieve a similar outcome, but it won't be managed by LocalStack. Note that in both cases, SSL verification must be disabled. See the [limitations of the DNS server]({{<ref "user-guide/tools/transparent-endpoint-injection/dns-server#limitations" >}}) for more information.
+For the community edition of LocalStack, you can employ your own DNS server to achieve a similar outcome, but it won't be managed by LocalStack.
+Note that in both cases, SSL verification must be disabled.
 
 ## From a lambda function
 

@@ -39,7 +39,8 @@ $ curl -k https://$domain/hello.txt
 {{< / command >}}
 
 {{< alert title="Note" >}}
-For CloudFront to function properly, ensure your local DNS setup is correctly configured. Refer to the section on [configuring the local DNS server]({{< ref "user-guide/tools/transparent-endpoint-injection/dns-server" >}}) for details.
+If you wish to use CloudFront on system host, ensure your local DNS setup is correctly configured.
+Refer to the section on [System DNS configuration]({{< ref "dns-server#system-dns-configuration" >}}) for details.
 {{< /alert >}}
 
 In the example provided above, be aware that the final command (`curl https://$domain/hello.txt`) might encounter a temporary failure accompanied by a warning message `Could not resolve host`. This can occur because different operating systems adopt diverse DNS caching strategies, causing a delay in the availability of the CloudFront distribution's DNS name (e.g., `abc123.cloudfront.net`) within the system. Typically, after a few retries, the command should succeed. It's worth noting that similar behavior can be observed in the actual AWS environment, where CloudFront DNS names may take up to 10-15 minutes to propagate across the network.

@@ -0,0 +1,252 @@
+---
+title: "DNS Server"
+categories: ["LocalStack Pro", "Tools", "DNS"]
+weight: 11
+description: >
+  Use LocalStack as DNS server to resolve AWS queries to LocalStack
+aliases:
+  - /tools/local-endpoint-injection/dns-server/
+  - /tools/transparent-endpoint-injection/dns-server/
+---
+
+LocalStack includes a DNS server that enables seamless connectivity to LocalStack from different environments using `localhost.localstack.cloud` (Community + Pro).
+The DNS server is available on all IPv4 addresses within the LocalStack container (i.e., listening to `0.0.0.0`) and resolves `localhost.localstack.cloud` to the LocalStack container.
+Therefore, containers that are configured to use the DNS server can reach LocalStack using `localhost.localstack.cloud`.
+This configuration happens automatically for containers created by LocalStack, including compute resources such as Lambda, ECS, and EC2.
+Your container can be configured to use the DNS server as demonstrated in the [Network Troubleshooting guide]({{< ref "references/network-troubleshooting/endpoint-url#from-your-container" >}}).
+If you wish to use the DNS server on your host system, follow the instructions under [System DNS configuration]({{< ref "dns-server#system-dns-configuration" >}}).
+
+LocalStack Pro additionally offers [Transparent Endpoint Injection]({{< ref "/user-guide/tools/transparent-endpoint-injection" >}}) (active by default),
+which enables seamless connectivity to LocalStack without changing your application code targeting AWS.
+The DNS server resolves AWS domains such as `*.amazonaws.com` including subdomains to the LocalStack container.
+Therefore, your application seamlessly accesses the LocalStack APIs instead of the real AWS APIs.
+
+{{< alert title="Notes" >}}
+On your host machine, `localhost.localstack.cloud` and any subdomains such as `mybucket.s3.localhost.localstack.cloud` resolve to `localhost` using a public DNS entry by LocalStack
+unless your router has [DNS rebind protection]({{< ref "dns-server#dns-rebind-protection" >}}) enabled.
+{{< / alert >}}
+
+
+### Fallback DNS server
+
+If you want to use another upstream DNS resolver than your default system DNS resolver or Google DNS (`8.8.8.8` fallback if detection fails),
+specify the fallback DNS server where all non-redirected queries (i.e., not matching `DNS_LOCAL_NAME_PATTERNS`) will be forwarded to:
+
+```bash
+DNS_SERVER=1.1.1.1
+```
+
+By default, LocalStack attempts to detect the default system DNS resolver upon startup.
+If this detection fails, LocalStack uses Google DNS `8.8.8.8` as a fallback.
+
+### Skip LocalStack DNS resolution
+
+If you want to resolve certain AWS URLs to AWS instead of LocalStack,
+specify a comma-separated list of skip patterns using Python-flavored regex such as:
+
+```bash
+DNS_LOCAL_NAME_PATTERNS='.*(ecr|lambda).*.amazonaws.com'
+```
+
+Using this configuration, the LocalStack DNS server resolves all AWS domains to LocalStack _except_ ECR and Lambda domains which will be resolved via the `DNS_SERVER` (i.e., the real DNS entry by default).
+For example, `https://123456789012.dkr.ecr.us-west-2.amazonaws.com` will be forwarded to the upstream DNS resolver and reach real AWS.
+This can be used for hybrid setups, where certain API calls (e.g., ECR, Lambda) target AWS, whereas other services will target LocalStack.
+The regex pattern follows Python flavored-regex and can be tested at [regex101.com](https://regex101.com/r/OzIsQa/1).
+
+[The regex101 link is maintained by Joel Scheuner (requires linking to GitHub or Google account).
+It redirects to the main page if the saved example would not work.]: #
+
+{{< alert title="Warning" color="warning">}}
+Use this configuration with caution because we generally do not recommend connecting to real AWS from within LocalStack. 
+{{< /alert >}}
+
+### DNS Server bind address
+
+If you experience problems when running LocalStack and the DNS server is the issue, you can disable the DNS server using:
+
+```bash
+DNS_ADDRESS=0
+```
+
+{{< alert title="Warning" color="warning" >}}
+We do not recommend disabling the DNS server since this disables resolving `localhost.localstack.cloud` to the LocalStack container.
+{{< / alert >}}
+
+This option is primarily used by [LocalStack developers]({{< ref "contributing/development-environment-setup" >}}) in host mode because binding port 53 requires root privileges and port 53 might be occupied.
+
+### LocalStack endpoints
+
+If you operate behind an enterprise proxy and wish to customize the domain name returned by LocalStack services (e.g., SQS queue URL),
+check out the [Configuration]({{< ref "configuration#core" >}}) `LOCALSTACK_HOST`.
+
+If you wish to customize internal LocalStack DNS routing of `localhost.localstack.cloud`,
+refer to the instructions in the [Route53 documentation]({{< ref "route53#customizing-internal-endpoint-resolution" >}}).
+
+
+## DNS rebind protection
+
+If you rely on your local network's DNS, your router/DNS server might block requests due to the DNS Rebind Protection.
+This feature is enabled by default in pfSense, OPNSense, OpenWRT, AVM FritzBox, and potentially also other devices.
+Some of the vendors might allow upstream responses in the 127.0.0.0/8 range (like OpenWRT).
+
+{{< alert title="Note" >}}
+If you are using the LocalStack DNS server, DNS rebind protection should not cause any issues.
+{{< / alert >}}
+
+You can check if your DNS setup works correctly by resolving a subdomain of `localhost.localstack.cloud`:
+{{< command "hl_lines=16">}}
+$ dig test.localhost.localstack.cloud
+
+; <<>> DiG 9.16.8-Ubuntu <<>> test.localhost.localstack.cloud
+;; global options: +cmd
+;; Got answer:
+;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45150
+;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
+
+;; OPT PSEUDOSECTION:
+; EDNS: version: 0, flags:; udp: 65494
+;; QUESTION SECTION:
+;test.localhost.localstack.cloud. IN	A
+
+;; ANSWER SECTION:
+test.localhost.localstack.cloud. 10786 IN CNAME	localhost.localstack.cloud.
+localhost.localstack.cloud. 389	IN	A	127.0.0.1
+
+;; Query time: 16 msec
+;; SERVER: 127.0.0.53#53(127.0.0.53)
+;; WHEN: Fr Jän 14 11:23:12 CET 2022
+;; MSG SIZE  rcvd: 90
+{{< /command >}}
+
+If the DNS resolves the subdomain to your localhost (127.0.0.1), your setup is working.
+If not, please check the configuration of your router / DNS if the Rebind Protection is active or [enable the LocalStack DNS on your system]({{< ref "dns-server#system-dns-configuration" >}}).
+
+
+## System DNS configuration
+
+If you wish to use the DNS server on your host system, you need to expose the LocalStack DNS server and configure your operating system.
+This is necessary if you want to test unmodified application code directly on your system against LocalStack and cannot configure the endpoint URL.
+
+{{< alert title="Warning" color="warning">}}
+Please be careful when changing the network configuration on your system, as this may have undesired side effects.
+Remember to save the default configuration and restore it after testing.
+{{< /alert >}}
+
+1. Expose the LocalStack DNS server:
+
+    a) The LocalStack CLI automatically publishes port 53 if it can be bound on the host.
+
+    b) For Docker Compose, add the following port mappings to your `docker-compose.yml`:
+
+     ```yaml
+     ports:
+       - "127.0.0.1:53:53"                # Expose DNS server to host
+       - "127.0.0.1:53:53/udp"            # Expose DNS server to host
+     ```
+
+{{< alert title="Notes" >}}
+If port 53 is already bound, `docker-compose up` fails with the error:
+```plain
+Error response from daemon: Ports are not available: exposing port UDP 127.0.0.1:53 -> 0.0.0.0:0: command failed
+```
+
+To find out if a program is listening on a port, run the following command:
+
+```bash
+# sudo is required if the port is < 1024
+# [sudo] lsof -P -i :<port> | grep LISTEN
+sudo lsof -P -i :53 | grep LISTEN
+```
+
+In macOS, a common process that listens on port 53 is `mDNSResponder`.
+Docker for Mac 4.24 has a [known issue](https://docs.docker.com/desktop/release-notes/#4240) and suggests the following workaround:
+> Deactivate network acceleration by adding `"kernelForUDP": false`, in the `settings.json` file located at `~/Library/Group Containers/group.com.docker/settings.json`.
+
+Additionally, ensure that "Internet Sharing" is disabled in the system preferences as suggested in [this GitHub issue](https://github.com/docker/for-mac/issues/7008#issuecomment-1748344545).
+
+{{< / alert >}}
+
+2. Configure LocalStack to use a `DNS_SERVER` other than the host, for example using [CloudFlare DNS](https://www.cloudflare.com/learning/dns/what-is-1.1.1.1/) `DNS_SERVER=1.1.1.1`.
+3. Configure your system to use the LocalStack DNS depending on your operating system:
+
+### macOS
+
+Search for "DNS servers" in the system preferences and add a new DNS server with the IP `127.0.0.1`.
+Updates in the system settings are automatically reflected in `/etc/resolv.conf` and should add such an entry such as `nameserver 127.0.0.1`.
+
+<img src="macos-dns-server-configuration.png" alt="macOS DNS server configuration" title="Configure DNS server in macOS system preferences" width="500" />
+
+### Linux
+
+In Linux, the configuration depends on your network manager/DNS configuration.
+
+#### systemd-resolved
+
+[//]: # (TODO: fix docs for Linux)
+On many modern systemd-based distributions, like Ubuntu, systemd-resolved is used for name resolution.
+LocalStack provides a CLI command for exactly this scenario.
+To use systemd-resolved and the LocalStack domain resolution, try the following steps.
+
+Start LocalStack Pro with `DNS_ADDRESS=127.0.0.1` as environment variable.
+This makes LocalStack bind port 53 on 127.0.0.1, whereas systemd-resolved binds its stub resolver to 127.0.0.53:53, which prevents a conflict.
+Once LocalStack is started, you can test the DNS server using `dig @127.0.0.1 s3.amazonaws.com` versus `dig @127.0.0.53 s3.amazonaws.com`, the former should return an A record `127.0.0.1`, the latter the real AWS DNS result.
+
+Run:
+{{< command >}}
+$ localstack dns systemd-resolved
+{{< / command >}}
+
+To revert, please run:
+{{< command >}}
+$ localstack dns systemd-resolved --revert
+{{< / command >}}
+
+{{< alert title="Note">}}
+You need sudo privileges to execute this command.
+{{< /alert >}}
+
+This command sets the DNS server of the bridge interface of the docker network LocalStack currently runs in to the LocalStack container's IP address.
+(The command does not work with host networking or without LocalStack running for this reason.)
+Also, it configures the DNS route to exclusively (and only) route the following DNS names (and its subdomains) to the LocalStack DNS:
+
+```text
+~amazonaws.com
+~aws.amazon.com
+~cloudfront.net
+~localhost.localstack.cloud
+```
+
+If you want to perform this action manually, please do the following steps:
+
+1. Find out the bridge interface and container IP of your LocalStack container.
+    Use `docker inspect localstack-main` to get the IP address and network, then `docker inspect network` to get the interface name.
+    If the interface name is not mentioned, it is usually the first 12 characters of the network ID prefixed with `br-`, like `br-0ae393d3345e`.
+    If you use the default bridge network, it is usually `docker0`.
+
+1. Configure the DNS resolver for the bridge network:
+
+    {{< command >}}
+    # resolvectl dns <network_name> <container_ip>
+    {{< / command >}}
+
+3. Set the DNS route to route only the above mentioned domain names (and subdomains) to LocalStack:
+
+    {{< command >}}
+    # resolvectl domain <network_name> ~amazonaws.com ~aws.amazon.com ~cloudfront.net ~localhost.localstack.cloud
+    {{< / command >}}
+
+In both cases, you can use `resolvectl query s3.amazonaws.com` or `resolvectl query example.com` to check which interface your DNS request is routed through, to confirm only the above mentioned domains (and its subdomains) are routed to LocalStack.
+
+When correctly configured, either using the LocalStack CLI command or manually, only the requests for the mentioned domain names are routed to LocalStack, all other queries will resolve as usual.
+
+#### Other resolution settings
+
+Depending on your Linux distribution, the settings to set a DNS server can be quite different.
+In some systems, directly editing `/etc/resolv.conf` is possible, like described in [macOS]({{< ref "#mac-os" >}}).
+If your `/etc/resolv.conf` is overwritten by some service, it might be possible to install and enable/start `resolvconf` and specify the nameserver in `/etc/resolvconf/resolv.conf.d/head` with `nameserver 127.0.0.1`.
+This will prepend this line in the resolv.conf file even after changes.
+
+{{< alert title="Note">}}
+Using these options, every DNS request is forwarded to LocalStack, which will forward queries it does not need to modify (in essence all but certain AWS domains).
+LocalStack does not share or store any forwarded DNS requests, except for local exception logging in debug mode.
+{{< /alert >}}