Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add functionality for export of latency logs via telemetry #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 90 commits into from
Jul 3, 2025
Merged

Add functionality for export of latency logs via telemetry #608

merged 90 commits into from
Jul 3, 2025
+475 −440

Conversation

saishreeeee
Copy link
Collaborator

@saishreeeee saishreeeee commented Jun 19, 2025 •
edited
Loading

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

Add functionality for export of latency logs via telemetry

Example latency log:
{
"frontend_log_event_id": "30565427-1299-4fbc-bec0-25378cfe912b",
"context": {
"client_context": {
"timestamp_millis": 1750592858108,
"user_agent": "PyDatabricksSqlConnector/4.0.4"
}
},
"entry": {
"sql_driver_log": {
"session_id": "01f04f5e-b1a8-1dd1-962a-e40efd092bdd",
"system_configuration": {
"driver_version": "4.0.4",
"os_name": "Darwin",
"os_version": "24.5.0",
"os_arch": "arm64",
"runtime_name": "Python 3.13.3",
"runtime_version": "3.13.3",
"runtime_vendor": "CPython",
"driver_name": "Databricks SQL Python Connector",
"char_set_encoding": "utf-8",
"locale_name": "en_US"
},
"driver_connection_params": {
"http_path": "/sql/1.0/warehouses/864004c1b3961382",
"mode": "THRIFT",
"host_info": {
"host_url": "e2-dogfood.staging.cloud.databricks.com",
"port": 443
},
"auth_mech": "PAT"
},
"sql_statement_id": "01f04f5e-b1d6-1a1e-afe4-e99e1ccb8805",
"sql_operation": {
"statement_type": "sql",
"is_compressed": true,
"execution_result": "inline_arrow",
"retry_count": 0
},
"operation_latency_ms": 518
}
}
}

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

PECOBLR-554

saishreeeee added 30 commits June 17, 2025 21:35
@saishreeeee
added functionality for export of failure logs
65a75f4 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
changed logger.error to logger.debug in exc.py
5305308 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
Fix telemetry loss during Python shutdown
ba83c33 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
unit tests for export_failure_log
131db92 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
try-catch blocks to make telemetry failures non-blocking for connecto…
3abc40d 
…r operations

Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed redundant try/catch blocks, added try/catch block to initiali…
ffa4787 
…ze and get telemetry client

Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
skip null fields in telemetry request
cc077f3 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed dup import, renamed func, changed a filter_null_values to lamda
2c6fd44 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed unnecassary class variable and a redundant try/except block
89540a1 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
public functions defined at interface level
52a1152 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
changed export_event and flush to private functions
3dcdcfa 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
formatting
b2714c9 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
changed connection_uuid to thread local in thrift backend
377a87b 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
made errors more specific
c9376b8 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
revert change to connection_uuid
bbfadf2 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
reverting change in close in telemetry client
9bce26b 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
JsonSerializableMixin
ef4514d 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
isdataclass check in JsonSerializableMixin
8924835 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
convert TelemetryClientFactory to module-level functions, replace Noo…
65361e7 
…pTelemetryClient class with NOOP_TELEMETRY_CLIENT singleton, updated tests accordingly

Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
renamed connection_uuid as session_id_hex
1722a77 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
added NotImplementedError to abstract class, added unit tests
e841434 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
formatting
2f89266 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
added PEP-249 link, changed NoopTelemetryClient implementation
5564bbb 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed unused import
1e4e8cf 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
made telemetry client close a module-level function
55b29bc 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
unit tests verbose
93bf170 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
debug logs in unit tests
45f5ccf 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
debug logs in unit tests
8ff1c1f 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
removed ABC from mixin, added try/catch block around executor shutdown
8bdd324 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@saishreeeee
checking stuff
f99f7ea 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee
removed the changes in test_fetches
ae90dee 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee
test_telemetry
acc9904 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee saishreeeee requested a review from jprakash-db July 3, 2025 05:21
jprakash-db
jprakash-db approved these changes Jul 3, 2025
View reviewed changes
Copy link
Contributor

@jprakash-db jprakash-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks for the making the changes

@saishreeeee
removed test
a847122 
Signed-off-by: Sai Shree Pradhan <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee saishreeeee merged commit 0dfe0f4 into telemetry Jul 3, 2025
23 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vikrantpuppala vikrantpuppala vikrantpuppala approved these changes

@jprakash-db jprakash-db jprakash-db approved these changes

@deeksha-db deeksha-db Awaiting requested review from deeksha-db deeksha-db is a code owner

@samikshya-db samikshya-db Awaiting requested review from samikshya-db samikshya-db is a code owner

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db jackyhu-db is a code owner

@madhav-db madhav-db Awaiting requested review from madhav-db madhav-db is a code owner

@gopalldb gopalldb Awaiting requested review from gopalldb gopalldb is a code owner

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db jayantsing-db is a code owner

@shivam2680 shivam2680 Awaiting requested review from shivam2680 shivam2680 is a code owner

Assignees

@saishreeeee saishreeeee

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@saishreeeee @vikrantpuppala @jprakash-db