Rework dns server and transparent endpoint injection #941

joe4dev · 2023-11-28T09:05:21Z

Motivation

The current DNS server page is badly outdated, misleading, and incomplete. With the increasing importance of DNS-based resolution in LocalStack networking, the DNS server page needs to be reworked.

Changes

Document missing config DISABLE_TRANSPARENT_ENDPOINT_INJECTION
Restructure the DNS server page
- Add more subheaders
- Adjust order by de-emphasizing the system configuration because that is more of a special case. Information that is relevant for LocalStack container users is more important and should come first.
Refine and extend the general introduction to this page
Re-write system DNS config section with a general start and OS-specific subsections. macOS is done and tested.
Generally clarify writing and warnings
UPDATE: Move the DNS server page to a top-level tool and describe transparent endpoint injection more self-contained
Ignore problematic URLs in link checker:
- CloudFlare using bot detection returning 403 forbidden
- GithHub boto3 link with line reference frequently flakes. Unfortunately, retries only happen on 429 (too many requests) according to the link checker tool.
UPDATE: Consistently use "Transparent Endpoint Injection" instead of transparent execution mode

Preview

Transparent Endpoint Injection: https://localstack-docs-preview-pr-941.surge.sh/user-guide/tools/transparent-endpoint-injection
DNS Server: https://localstack-docs-preview-pr-941.surge.sh/user-guide/tools/dns-server

Discussion

The explanations around "transparent execution mode" / "transparent endpoint injection" could be improved. We used to have multiple implementations (legacy SDK replacement + DNS-based) and with the introduction of AWS_ENDPOINT_URL, this could be considered another implementation because it does transparently inject the endpoint. However, the configuration DISABLE_TRANSPARENT_ENDPOINT_INJECTION could be confusing if we do not clarify that this specific implementation refers to the DNS-based implementation.
It is not 100% clear where transparent endpoint injection is explained. The page Transparent Endpoint Injection (preview) gives a short 1-sentence intro and then refers to the DNS server page for more details. Not yet ideal (but much better than before ;).
The default for DNS_SERVER mentioned to be 8.8.8.8 can be confusing because, in the current implementation, the upstream DNS will be used as default (i.e., typically the system DNS). That is also the reason why we need to explicitly set an external DNS server when configuring LocalStack as system DNS because otherwise, we would create an infinite loop ;)

TODO

Clarify default DNS server behavior
Fix CloudFlare link yielding 403 (probably checking user agent).
- CloudFlare doesn't like link checkers an blocks them using some advanced fingerprinting (user-agent worked on my machine). Ignoring the URL for now.

Follow Up

@simonrw @dfangl Update Linux instructions for system DNS configuration
@simonrw Adjust docs for new DNS_SKIP_LOCAL_RESOLUTION Introduce DNS_NAME_PATTERNS_TO_RESOLVE_UPSTREAM config localstack#9692

github-actions · 2023-11-28T09:05:34Z

🎊 PR Preview has been successfully built and deployed to https://localstack-docs-preview-pr-941.surge.sh 🎊

simonrw · 2023-11-29T11:41:12Z

The DNS server page is still under the "transparent endpoint injection" section, is this a conscious decision? I think it would be better either at the top level in the user guide, or at least elevated to the "LocalStack tools" section, though it's not an external tool like the section naming suggests.

joe4dev · 2023-11-29T12:12:59Z

The DNS server page is still under the "transparent endpoint injection" section, is this a conscious decision? I think it would be better either at the top level in the user guide, or at least elevated to the "LocalStack tools" section, though it's not an external tool like the section naming suggests.

@HarshCasper suggested the same in #866 and that makes sense if it would remain the only subsection

I foresee that we are going to have 3 scenarios on how to connect to LocalStack:

[Community] Manual endpoint configuration: Specify the endpoint_url using AWS_ENDPOINT_URL for AWS clients that do not yet support it natively (all default Lambda runtimes ATM: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html). This is not really a tool but rather something we should document for each (major) runtime (currently missing). Maybe in the Lambda docs?
[Community] Auto AWS_ENDPOINT_URL detection: Newer SDKs automatically detect AWS_ENDPOINT_URL and no code changes are necessary. This is likely going to be the default option in the future. This is also not really a tool and does not fit here nicely.
[Pro] LocalStack Transparent Endpoint injection: The current way of DNS-based seamless connectivity to LocalStack for Pro users is enabled by default in Lambda (i.e., opt-out). This is a LocalStack tool and fits here.

Given that only the LS transparent endpoint injection is a real (internal) "tool", it might best to only cover that one here (including DISABLE_TRANSPARENT_ENDPOINT_INJECTION) and move the "DNS server" somewhere else.

I think it would be better either at the top level in the user guide, or at least elevated to the "LocalStack tools" section, though it's not an external tool like the section naming suggests.

Transparent endpoint injection and Lambda are also not really "external" tools (I think the term is used broadly for extra features beyond AWS emulation). I agree/suggest moving the "DNS server" to the top-level Tools section for now. What do you think?

simonrw

This is a huge improvement thanks, and most of my comments are suggestions I'll let you decide how to proceed, but the DNS_LOCAL_NAME_PATTERNS section needs updating.

content/en/references/configuration.md

content/en/user-guide/tools/transparent-endpoint-injection/dns-server.md

simonrw · 2023-11-29T12:10:37Z

content/en/user-guide/tools/transparent-endpoint-injection/dns-server.md

+{{< alert title="Warning" color="warning">}}
+Use this configuration with caution because we generally do not recommend connecting to real AWS from within LocalStack. 
+{{< /alert >}}
+

 ## Self-signed certificates


I think this step could remain with the TEI docs, since it is not specific to the DNS server but rather TEI.

I think this option does indeed refer to DNS_LOCAL_NAME_PATTERNS because skipping LocalStack resolution enables connecting to real AWS 🤔 (at least in it's original use)

I agree that it also fits for disabling transparent endpoint injection.

But that is disabling transparent endpoint injection, right?!

right, disabling transparent endpoint injection deserves a bigger warning than partially disabling it through ...

Maybe the thought was that people use alternative ways (custom DNS or manual endpoint injection) if they disable it completely.

👉 I added to both places for now because the effect is ultimately the same

content/en/user-guide/tools/transparent-endpoint-injection/dns-server.md

joe4dev · 2023-11-29T17:09:13Z

content/en/user-guide/tools/transparent-endpoint-injection/_index.md

+process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0"
+```
+
+If you are using the Java AWS SDK v2 in Lambda, you can opt in to automatically disable SSL validation using the configuration `LAMBDA_DISABLE_JAVA_SDK_V2_CERTIFICATE_VALIDATION=1`.


@dfangl @dominikschubert sadly, we missed making this breaking change upon v3 announced at the 2.3.0 release:

Transparent endpoint injection for Lambdas running AWS Java SDK v2 is now available with the feature flag LAMBDA_DISABLE_JAVA_SDK_V2_CERTIFICATE_VALIDATION=1 . This will be default behaviour by 3.0.
https://discuss.localstack.cloud/t/localstack-release-v2-3-0/533#other-changes-and-enhancements-14

Should we add it to the Lambda configurations (so we won't miss it at v4)?

general config discussion: do we strive to have all defaults 0 for configurations?

If we ever change the default for a single config, it gets inconsistent. Hence avoiding double negations might be clearer in most cases 💭

simonrw

Great stuff, thanks for making this sooo much clearer!

simonrw · 2023-11-29T14:32:15Z

content/en/user-guide/tools/transparent-endpoint-injection/dns-server.md

+{{< alert title="Warning" color="warning">}}
+Use this configuration with caution because we generally do not recommend connecting to real AWS from within LocalStack. 
+{{< /alert >}}
+

 ## Self-signed certificates


But that is disabling transparent endpoint injection, right?!

content/en/user-guide/tools/transparent-endpoint-injection/dns-server.md

joe4dev added 2 commits November 28, 2023 09:46

Rework DNS server

a651342

Use light-themed image

0abb011

joe4dev requested review from simonrw and dfangl November 28, 2023 09:05

joe4dev self-assigned this Nov 28, 2023

simonrw suggested changes Nov 29, 2023

View reviewed changes

joe4dev added 4 commits November 29, 2023 15:30

Fix 403 forbidden link check using user agent

459f634

Refine with review feedback

4bfa803

Move DNS Server to separate tools page

ae15593

Re-add customized part for DNS_RESOLVE_IP

cc3bfe6

joe4dev requested a review from giograno as a code owner November 29, 2023 16:24

joe4dev added 5 commits November 29, 2023 17:27

Add pointer to customize skip patterns

672f507

Try a user agent matching the OS

4a12f64

Ignore CloudFlare for link checking

0ef856c

Clarify default DNS detection

9ff5bae

Ignore flaky boto3 link

5ec8636

joe4dev commented Nov 29, 2023

View reviewed changes

simonrw approved these changes Nov 30, 2023

View reviewed changes

Add warning for disabling transparent endpoint injection

8717042

joe4dev changed the title ~~Rework dns server~~ Rework dns server and transparent endpoint injection Nov 30, 2023

joe4dev merged commit 2b08cdc into main Nov 30, 2023

joe4dev deleted the rework-dns-server branch November 30, 2023 16:43

joe4dev mentioned this pull request Nov 30, 2023

Fix dns server redirect #944

Merged

simonrw mentioned this pull request Nov 30, 2023

reformat transparent endpoint injection docs #866

Closed

Rework dns server and transparent endpoint injection #941

Rework dns server and transparent endpoint injection #941

Uh oh!

Conversation

joe4dev commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Preview

Discussion

TODO

Follow Up

Uh oh!

github-actions bot commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonrw commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joe4dev commented Nov 29, 2023

Uh oh!

simonrw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonrw Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

joe4dev Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

simonrw Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

joe4dev Nov 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joe4dev Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

joe4dev Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

simonrw left a comment

Choose a reason for hiding this comment

Uh oh!

simonrw Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joe4dev commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading

simonrw commented Nov 29, 2023 •

edited

Loading