Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Handle analytics publishing errors more gracefully #10185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Handle analytics publishing errors more gracefully #10185

wants to merge 2 commits into from

Conversation

whummer
Copy link
Member

@whummer whummer commented Feb 6, 2024 •
edited
Loading

Motivation

In a support channel, we've seen error stack trace related to SSL verification when connecting to the analytics service (can happen if the user is behind a corporate proxy/firewall).

This behavior can be reproduced by setting analytics.localstack.cloud to an invalid IP address (where the TLS certificate doesn't match our hostname), then shutting down the container:

$ DOCKER_FLAGS='--add-host analytics.localstack.cloud:1.1.1.1' DEBUG=1 localstack start
...
Ready.

[CTRL-C]
...
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: MyHTTPSConnectionPool(host='analytics.localstack.cloud', port=443): Max retries exceeded with url: /v1/events (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'analytics.localstack.cloud'. (_ssl.c:1006)")))
LocalStack supervisor: localstack process (PID 15) returned with exit code 0

These logs should not be presented to the user, and the error also causes the shutdown hooks not to be executed properly on shutdown in stop_infra().

Changes

Add a try/except block to catch the exception, and log it with an INFO log message.

Let me know if you'd like to see this covered with a test - happy to try and wire something up, if we think it could be useful.

@whummer whummer requested a review from thrau as a code owner February 6, 2024 14:29
@whummer whummer added the semver: patch Non-breaking changes which can be included in patch releases label Feb 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 6, 2024 •
edited
Loading

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 22m 7s ⏱️ -40s
2 623 tests ±0  2 374 ✅ ±0  249 💤 ±0  0 ❌ ±0 
2 625 runs  ±0  2 374 ✅ ±0  251 💤 ±0  0 ❌ ±0 

Results for commit 2d81292. ± Comparison against base commit e2e054a.

♻️ This comment has been updated with latest results.

alexrashed
alexrashed approved these changes Feb 6, 2024
View reviewed changes
Copy link
Member

@alexrashed alexrashed left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for jumping on this! The fix is straight forward and minimal. 💯

@coveralls
Copy link

Coverage Status

coverage: 83.84% (-0.002%) from 83.842%
when pulling 9d504f6 on analytics-errors
into e2e054a on master.

thrau
thrau reviewed Feb 6, 2024
View reviewed changes
Copy link
Member

@thrau thrau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying the culprit! Before we merge I just want to understand why we are using a AnalyticsClientPublisher directly, rather than using localstack.utils.analytics.log, which already handles errors and logging.
@dominikschubert do you remember?

localstack/utils/analytics/usage.py
Comment on lines +123 to +125
publisher.publish(
[Event(name="ls:usage_analytics", metadata=metadata, payload=aggregated_payload)]
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason for not using the global analytics bus here? it already does buffering and graceful handling of errors

@alexrashed alexrashed added this to the 3.4 milestone Mar 26, 2024
@alexrashed alexrashed modified the milestones: 3.4, 3.5 Apr 24, 2024
@dominikschubert dominikschubert modified the milestones: 3.5, 3.6 Jun 12, 2024
@whummer
Copy link
Member Author

whummer commented Jul 12, 2024

This PR is outdated, wasn't able to replicate the error anymore with the latest version, as the code path no longer gets called with the new runtime framework introduced in #10942 . Closing this one for now - we can revisit in the future if the issue comes up again. 👍

@whummer whummer closed this Jul 12, 2024
@whummer whummer deleted the analytics-errors branch July 12, 2024 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thrau thrau thrau left review comments

@alexrashed alexrashed alexrashed approved these changes

@dominikschubert dominikschubert Awaiting requested review from dominikschubert

Assignees

@dominikschubert dominikschubert

Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
3.6
Development

Successfully merging this pull request may close these issues.
5 participants
@whummer @coveralls @alexrashed @thrau @dominikschubert