Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Complete Fetch Phase (EXTERNAL_LINKS disposition and ARROW format) #598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 296 commits into
base: sea-migration
Choose a base branch
from
Open

Complete Fetch Phase (EXTERNAL_LINKS disposition and ARROW format) #598

wants to merge 296 commits into from
+1,608 −355

Conversation

varun-edachali-dbx
Copy link
Collaborator

@varun-edachali-dbx varun-edachali-dbx commented Jun 16, 2025 •
edited
Loading

What type of PR is this?

  • Feature

Description

Complete the Fetch phase of the SEA implementation for EXTERNAL_LINKS disposition and ARROW_STREAM format. Introduce a new CloudFetchQueue.

Note that this is not yet feature complete, with some known failures in the following domains (for the SEA backend only):

  • retries
  • native, complex params
  • metadata queries

These will be resolved in future PRs.

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually - using the test scripts invoked in examples/experimental/sea_connector_test.py.
  • N/A

The coverage of the key classes by the unit tests are as below:

Module Statements Missing Coverage Notes
backend/sea/queue.py (SeaCloudFetchQueue and SeaResultSetQueueFactory classes) 92 2 98% Missing lines are pyarrow import exception handling (lines 10-11) which are not meant to be covered in runtime tests
backend/sea/result_set.py (SeaResultSet class) 85 3 96% Missing lines are pyarrow import exception handling (lines 13-14) and TYPE_CHECKING import (line 17) which are not meant to be covered in runtime tests

Related Tickets & Documents

Design Doc
PECOBLR-553

@varun-edachali-dbx
move description to List[Tuple]
8a014f0 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
frmatting (black)
39c41ab 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
reduce diff (remove explicit tuple conversion)
2cd04df 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove has_more_rows from ExecuteResponse
067a019 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove un-necessary has_more_rows aclc
48c83e0 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
default has_more_rows to True
281a9e9 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
return has_more_rows from ExecResponse conversion during GetRespMetadata
192901d 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove unnecessary replacement
55f5c45 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
better mocked backend naming
edc36b5 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove has_more_rows test in ExecuteResponse
81280e7 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
introduce replacement of original has_more_rows read test
c1d3be2 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
call correct method in test_use_arrow_schema
5ee4136 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
call correct method in test_fall_back_to_hive_schema
b881ab0 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
re-introduce result response read test
53bf715 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
simplify test
45a32be 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove excess fetch_results mocks
e3fe299 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
more minimal changes to thrift_backend tests
e8038d3 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
move back to old table types
2f6ec19 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove outdated arrow_schema_bytes return
73bc282 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
backend from cloudfetch-sea
e385d5b 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove filtering, metadata ops
484064e 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
raise NotImplementedErrror for metadata ops
030edf8 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
align SeaResultSet with new structure
4e07f1e 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
correct sea res set tests
65e7c6b 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
add metadata commands
30f8266 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
formatting (black)
033ae73 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
add metadata command unit tests
33821f4 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
minimal fetch phase intro
71b451a 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
Merge branch 'exec-resp-norm' into fetch-json-inline
170f339
@varun-edachali-dbx
Merge branch 'sea-res-set' into fetch-json-inline
40f79b5
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
reduce diff
f84578a 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
introduce SEA cloudfetch e2e tests
c621c0c 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
allow empty cloudfetch result
7958cd9 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx varun-edachali-dbx marked this pull request as ready for review July 3, 2025 06:04
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
skip pyarrow dependent tests
d348b35 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 3, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deeksha-db deeksha-db Awaiting requested review from deeksha-db

@samikshya-db samikshya-db Awaiting requested review from samikshya-db

@jprakash-db jprakash-db Awaiting requested review from jprakash-db

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db

@madhav-db madhav-db Awaiting requested review from madhav-db

@gopalldb gopalldb Awaiting requested review from gopalldb

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db

@vikrantpuppala vikrantpuppala Awaiting requested review from vikrantpuppala

@shivam2680 shivam2680 Awaiting requested review from shivam2680

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@varun-edachali-dbx