-
Notifications
You must be signed in to change notification settings - Fork 5.5k
Databricks vector search endpoints #18256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
- Introduced actions for creating, deleting, getting, and listing vector search endpoints. - Added endpoint name prop for dynamic endpoint selection. - Updated package version to 0.2.0 and dependencies to the latest version. - Incremented version for existing actions to 0.0.2 where applicable.
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
Warning Rate limit exceeded@luancazarine has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 4 minutes and 47 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughAdds Databricks Vector Search endpoint support: app propDefinition and methods for create/get/list/delete endpoints, four new Pipedream actions (create, get, list with pagination, delete), and updates package metadata and dependency versions. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant Action as Pipedream Action
participant App as Databricks App
participant API as Databricks Vector Search API
rect rgba(230,245,255,0.6)
note right of Action: Create / Get / Delete endpoint flows
User->>Action: Invoke (props)
Action->>App: method(args, $)
alt Create
App->>API: POST /vector-search/endpoints
API-->>App: 201 Created (id,...)
else Get
App->>API: GET /vector-search/endpoints/{name}
API-->>App: 200 OK (details)
else Delete
App->>API: DELETE /vector-search/endpoints/{name}
API-->>App: 200 OK
end
App-->>Action: Response
Action-->>User: Summary + Response
end
sequenceDiagram
autonumber
actor User
participant Action as List Endpoints Action
participant App as Databricks App
participant API as Databricks Vector Search API
rect rgba(240,255,240,0.6)
note right of Action: Paginated listing until no next_page_token or maxResults reached
User->>Action: Invoke (maxResults)
loop paginate
Action->>App: listEndpoints({ page_token })
App->>API: GET /vector-search/endpoints?page_token=...
API-->>App: 200 OK (endpoints[], next_page_token)
App-->>Action: { endpoints[], next_page_token }
Action->>Action: Accumulate results
end
Action-->>User: Summary + endpoints[]
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Assessment against linked issues
Out-of-scope changes
Suggested labels
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
components/databricks/actions/list-runs/list-runs.mjs (1)
52-54
: Bug: loop condition compares array to number, truncating pagination prematurely.
allRuns < this.maxResults
should beallRuns.length < this.maxResults
. Current code can exit early even when more pages exist.- } while (total === params.limit && allRuns < this.maxResults); + } while (total === params.limit && allRuns.length < this.maxResults);
🧹 Nitpick comments (9)
components/databricks/actions/list-runs/list-runs.mjs (1)
42-47
: Nice-to-have: respect maxResults when paging to avoid over-fetching.
Throttleparams.limit
per-iteration to remaining budget.- do { - const { runs } = await this.databricks.listRuns({ + do { + params.limit = Math.min(this.maxResults - allRuns.length, params.limit); + const { runs } = await this.databricks.listRuns({ params, $, });components/databricks/databricks.app.mjs (1)
70-77
: Optional: Broaden workspace host support
Hardcoding.cloud.databricks.com
restricts this client to AWS; Azure workspaces use*.azuredatabricks.net
and GCP uses*.gcp.databricks.com
. Accept a full host in$auth.domain
if it contains a dot, otherwise append the AWS pattern.- _baseUrl() { - return `https://${this.$auth.domain}.cloud.databricks.com/api/2.0`; - }, + _baseUrl() { + const host = this.$auth.domain.includes('.') + ? this.$auth.domain + : `${this.$auth.domain}.cloud.databricks.com`; + return `https://${host}/api/2.0`; + },components/databricks/actions/create-endpoint/create-endpoint.mjs (3)
28-31
: Harden success summary against missingid
Some responses may not include
id
(or use a different field). Fall back to the provided name so the summary is still useful.- if (response) { - $.export("$summary", `Successfully created endpoint with ID ${response.id}.`); - } + if (response) { + const idOrName = response?.id ?? response?.endpoint_id ?? this.name; + $.export("$summary", `Successfully created endpoint "${idOrName}".`); + }
18-26
: Validate input earlyReject empty or whitespace-only names before making the API call.
- async run({ $ }) { - try { + async run({ $ }) { + const name = this.name?.trim(); + if (!name) { + throw new ConfigurationError("Endpoint Name is required."); + } + try { const response = await this.databricks.createEndpoint({ data: { - name: this.name, + name, endpoint_type: "STANDARD", }, $, });
10-17
: Consider exposing endpoint type as a prop (default STANDARD)Let users pick the endpoint type if Databricks adds more types, while defaulting to "STANDARD".
props: { databricks, name: { type: "string", label: "Endpoint Name", description: "The name of the vector search endpoint", }, + endpointType: { + type: "string", + label: "Endpoint Type", + description: "Type of vector search endpoint.", + options: [ + { label: "STANDARD", value: "STANDARD" }, + ], + default: "STANDARD", + }, },And pass it:
- endpoint_type: "STANDARD", + endpoint_type: this.endpointType,components/databricks/actions/delete-endpoint/delete-endpoint.mjs (1)
18-26
: Wrap in try/catch and return a stable shapeDelete APIs often return empty bodies. Wrap errors for clearer UX and always return a predictable object.
- async run({ $ }) { - const response = await this.databricks.deleteEndpoint({ - endpointName: this.endpointName, - $, - }); - - $.export("$summary", `Successfully deleted endpoint "${this.endpointName}".`); - return response; - }, + async run({ $ }) { + try { + const response = await this.databricks.deleteEndpoint({ + endpointName: this.endpointName, + $, + }); + $.export("$summary", `Successfully deleted endpoint "${this.endpointName}".`); + return response ?? { success: true, endpointName: this.endpointName }; + } catch (err) { + const message = err?.response?.data?.message + || err?.message + || `Failed to delete endpoint "${this.endpointName}".`; + throw new Error(message); + } + },Optional: swap
Error
forConfigurationError
and import it from@pipedream/platform
for consistency.components/databricks/actions/get-endpoint/get-endpoint.mjs (1)
18-29
: Add defensive error handling and resilient summaryGuard against missing bodies and surface better errors.
- async run({ $ }) { - const response = await this.databricks.getEndpoint({ - endpointName: this.endpointName, - $, - }); - - if (response) { - $.export("$summary", `Successfully retrieved endpoint "${this.endpointName}".`); - } - - return response; - }, + async run({ $ }) { + try { + const response = await this.databricks.getEndpoint({ + endpointName: this.endpointName, + $, + }); + $.export("$summary", `Successfully retrieved endpoint "${this.endpointName}".`); + return response ?? {}; + } catch (err) { + const message = err?.response?.data?.message + || err?.message + || `Failed to retrieve endpoint "${this.endpointName}".`; + throw new Error(message); + } + },components/databricks/actions/list-endpoints/list-endpoints.mjs (2)
11-16
: ConstrainmaxResults
Add a minimal bound to avoid confusion from zero or negative values.
maxResults: { type: "integer", label: "Max Results", description: "Maximum number of endpoints to return", default: 100, + min: 1, },
35-43
: Clarify truncation in the summaryMake it explicit when results were truncated to
maxResults
.- $.export("$summary", `Successfully retrieved ${allEndpoints.length} endpoint${allEndpoints.length === 1 - ? "" - : "s"}.`); + const truncated = allEndpoints.length > this.maxResults ? " (truncated)" : ""; + $.export("$summary", `Successfully retrieved ${allEndpoints.length} endpoint${allEndpoints.length === 1 ? "" : "s"}${truncated}.`);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
pnpm-lock.yaml
is excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (9)
components/databricks/actions/create-endpoint/create-endpoint.mjs
(1 hunks)components/databricks/actions/delete-endpoint/delete-endpoint.mjs
(1 hunks)components/databricks/actions/get-endpoint/get-endpoint.mjs
(1 hunks)components/databricks/actions/get-run-output/get-run-output.mjs
(1 hunks)components/databricks/actions/list-endpoints/list-endpoints.mjs
(1 hunks)components/databricks/actions/list-runs/list-runs.mjs
(1 hunks)components/databricks/actions/run-job-now/run-job-now.mjs
(1 hunks)components/databricks/databricks.app.mjs
(2 hunks)components/databricks/package.json
(2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2024-12-12T19:23:09.039Z
Learnt from: jcortes
PR: PipedreamHQ/pipedream#14935
File: components/sailpoint/package.json:15-18
Timestamp: 2024-12-12T19:23:09.039Z
Learning: When developing Pipedream components, do not add built-in Node.js modules like `fs` to `package.json` dependencies, as they are native modules provided by the Node.js runtime.
Applied to files:
components/databricks/package.json
🧬 Code graph analysis (4)
components/databricks/actions/delete-endpoint/delete-endpoint.mjs (1)
components/databricks/actions/get-endpoint/get-endpoint.mjs (1)
response
(19-22)
components/databricks/actions/get-endpoint/get-endpoint.mjs (1)
components/databricks/actions/delete-endpoint/delete-endpoint.mjs (1)
response
(19-22)
components/databricks/actions/list-endpoints/list-endpoints.mjs (2)
components/databricks/actions/list-runs/list-runs.mjs (1)
params
(33-38)components/databricks/databricks.app.mjs (1)
params
(29-32)
components/databricks/actions/create-endpoint/create-endpoint.mjs (2)
components/databricks/actions/delete-endpoint/delete-endpoint.mjs (1)
response
(19-22)components/databricks/actions/get-endpoint/get-endpoint.mjs (1)
response
(19-22)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: pnpm publish
- GitHub Check: Publish TypeScript components
- GitHub Check: Verify TypeScript components
- GitHub Check: Lint Code Base
🔇 Additional comments (11)
components/databricks/actions/run-job-now/run-job-now.mjs (2)
7-7
: Version bump only — looks good.
No behavioral changes detected.
6-6
: Update documentation link anchor to “run-now”- description: "Run a job now and return the id of the triggered run. [See the documentation](https://docs.databricks.com/en/workflows/jobs/jobs-2.0-api.html#runs-list)", + description: "Run a job now and return the id of the triggered run. [See the documentation](https://docs.databricks.com/en/workflows/jobs/jobs-2.0-api.html#run-now)",components/databricks/actions/get-run-output/get-run-output.mjs (1)
7-7
: Version bump only — looks good.
No functional changes.components/databricks/package.json (2)
3-3
: Package version bump — OK.
No issues spotted.
16-16
: axios export unchanged in @pipedream/platform@^3.1.0
Verified v3.1.0 changelog (PR #5544) adds the Pipedream-wrapped axios export without breaking existingimport { axios } from "@pipedream/platform"
usage.components/databricks/databricks.app.mjs (1)
114-121
: Confirmed Vector Search endpoints and pagination params: The endpoints map to the correct paths under/api/2.0/vector-search/endpoints
(POST
for create,GET
for list, andGET
/DELETE
with/{endpoint_name}
), and pagination usespage_token
in requests withnext_page_token
in responses.components/databricks/actions/create-endpoint/create-endpoint.mjs (1)
20-26
: Idempotency and conflict UXIf the endpoint already exists, the API may return a 409/conflict. Consider catching that case and turning it into a friendly message instead of a hard failure.
Would you like me to add a small helper that detects conflict status codes and exports a “already exists” summary?
components/databricks/actions/delete-endpoint/delete-endpoint.mjs (1)
3-17
: LGTM overallProps wiring and summary message look consistent with other actions.
components/databricks/actions/get-endpoint/get-endpoint.mjs (1)
3-17
: LGTMProp definition reuse and action metadata are consistent.
components/databricks/actions/list-endpoints/list-endpoints.mjs (2)
3-17
: LGTMAction metadata and basic pagination approach look consistent with the rest of the suite.
23-31
: Optional: pass page size to reduce overfetchIf the Databricks API supports
page_size
/max_results
, set it based on remaining capacity to minimize extra requests.Would you like me to adjust
params
to include a dynamicpage_size
once we confirm the parameter name in the Databricks API?
- Incremented version numbers for multiple Databricks SQL Warehouse actions to 0.0.2 and 0.0.3 as applicable. - Added missing closing brackets in the `databricks.app.mjs` file. - Updated the `get-run-output` and `list-runs` actions to version 0.0.3. - Ensured consistency in action descriptions and documentation links.
Resolves #18126
Summary by CodeRabbit
New Features
Chores