Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@lovincyrus
Copy link
Contributor

@lovincyrus lovincyrus commented Oct 17, 2025

This PR supports HMAC and Google application credentials authentication in GCS form. In addition to creating connector files, we now support step-based form flow: Step 1 (Connector) → Step 2 (Model/Source)

Closes https://linear.app/rilldata/issue/APP-353/gcs-explicit-credential-flow-for-models

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@lovincyrus lovincyrus marked this pull request as ready for review October 22, 2025 07:28
@royendo
Copy link
Contributor

royendo commented Oct 22, 2025

Repost from Linear: One thing here is that you dont need to plug the service account key in the model JSON 🙂

Other than that, demo looks great! thanks for the fast addition.

@ericpgreen2
Copy link
Contributor

@claude Please review this code for clarity and maintainability. One component that I feel is at-risk is "AddDataForm.svelte", so please include an inspection of that code.

@claude
Copy link

claude bot commented Oct 23, 2025

Claude finished @ericpgreen2's task —— View job


Code Review for PR #8142: Explicit credentials for GCS

I've completed a thorough review of this PR, focusing especially on AddDataForm.svelte as requested. Here's my detailed analysis:

🔍 Overall Assessment

This PR successfully implements explicit credential support for GCS with a step-based form flow. The implementation is functionally sound but has significant maintainability concerns in AddDataForm.svelte that need attention.

⚠️ Major Concerns with AddDataForm.svelte

1. Component Complexity (High Priority)

web-common/src/features/sources/modal/AddDataForm.svelte:1-986

Issues:

  • 986 lines in a single component - far exceeds maintainable size
  • Multiple responsibilities: form validation, state management, UI rendering, YAML generation, file handling
  • Deep conditional nesting makes logic flow difficult to follow
  • Reactive statement overuse creates hard-to-debug dependencies

Specific Problem Areas:

// Lines 77-96: Complex reactive logic mixing step state and form updates
$: if (
  isMultiStepConnector &&
  stepState.step === "source" &&
  stepState.connectorConfig
) {
  // 20+ lines of form manipulation logic
}

// Lines 347-378: Massive yamlPreview reactive statement
$: yamlPreview = (() => {
  // 30+ lines of complex conditional logic
})();

2. State Management Complexity

AddDataForm.svelte:63-100

Issues:

  • 7 different form states (paramsForm, dsnForm, clickhouseParamsForm, etc.)
  • Multiple reactive variables that depend on each other creating circular dependency risks
  • Step state management mixed with form state creates coupling

3. Duplicated Form Logic

AddDataForm.svelte:550-866

Issues:

  • Form field rendering logic is repeated 4+ times with slight variations
  • Error handling patterns duplicated across different form types
  • Validation logic scattered throughout the component

Positive Aspects

1. Clean Supporting Files

  • connectorStepStore.ts is well-designed with clear separation of concerns
  • Radio.svelte is properly structured with good prop interfaces
  • constants.ts provides good type safety and configuration management

2. Security Implementation

runtime/drivers/gcs/gcs.go:109-135

  • Proper credential validation in Go backend
  • Secure handling of both credential types (JSON credentials vs HMAC)
  • Good error handling for authentication failures

3. Type Safety

The TypeScript interfaces and types are well-defined, providing good compile-time safety.

🛠️ Specific Recommendations

Immediate Actions (High Priority)

  1. Break down AddDataForm.svelte:

    AddDataForm.svelte (coordinator, <200 lines)
    ├── FormRenderer.svelte (field rendering logic)
    ├── GCSMultiStepForm.svelte (GCS-specific flow)
    ├── ClickHouseForm.svelte (already exists, good!)
    ├── FormValidation.ts (validation logic)
    └── YamlPreview.svelte (preview generation)
    
  2. Extract Common Form Logic:

    // useFormManager.ts
    export function useFormManager(connector, formType) {
      // Centralize form state management
    }
    
    // FormRenderer.svelte
    export function renderFields(properties, form, errors) {
      // Centralize field rendering
    }
  3. Simplify Reactive Statements:

    $: formConfig = computeFormConfig(stepState, connector, formType);
    $: validationState = computeValidation(formConfig, formValues);

Medium Priority Improvements

  1. Create Step Management Hook:

    // useStepManager.ts  
    export function useStepManager(connector, isMultiStep) {
      // Centralize step transitions and state
    }
  2. Standardize Error Handling:

    // errorUtils.ts
    export function handleFormError(error, formType) {
      // Centralize error processing
    }

Code Quality Issues

  1. Lines 195-224: The isSubmitDisabled reactive statement is too complex
  2. Lines 406-479: handleOnUpdate function is doing too much
  3. Lines 678-815: GCS multi-step form logic should be extracted

🧪 Testing Coverage

The test file shows good coverage of the new functionality, testing both authentication methods and step transitions.

📋 Action Items Summary

Must Fix:

  • Split AddDataForm.svelte into smaller, focused components
  • Extract common form logic into reusable utilities
  • Simplify reactive statement dependencies

Should Fix:

  • Create dedicated step management utilities
  • Standardize error handling patterns
  • Add JSDoc comments for complex functions

Nice to Have:

  • Add storybook stories for new components
  • Create form field registry pattern
  • Add performance monitoring for form operations

🎯 Bottom Line

While this PR delivers the required functionality successfully, AddDataForm.svelte has grown into a maintenance liability. The component should be refactored into smaller, focused components before adding more features. The supporting files and backend implementation are well-crafted.

Recommendation: Consider this PR functionally approved but plan immediate refactoring work to address the maintainability debt.


Copy link
Contributor

@ericpgreen2 ericpgreen2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see Claude's assessment above and break down the "AddDataForm" into modular components. A 986-line component with so many conditionals is far too complex!

@lovincyrus
Copy link
Contributor Author

  1. Break down AddDataForm.svelte:
    AddDataForm.svelte (coordinator, <200 lines)
    ├── FormRenderer.svelte (field rendering logic)
    ├── GCSMultiStepForm.svelte (GCS-specific flow)
    ├── ClickHouseForm.svelte (already exists, good!)
    ├── FormValidation.ts (validation logic)
    └── YamlPreview.svelte (preview generation)
    

These are acceptable codesmell as we plan on tackling this in https://linear.app/rilldata/issue/APP-449/composable-adddataform We covered this in our 1:1

@ericpgreen2
Copy link
Contributor

ericpgreen2 commented Oct 24, 2025

These are acceptable codesmell as we plan on tackling this in https://linear.app/rilldata/issue/APP-449/composable-adddataform We covered this in our 1:1

This PR introduces significant complexity with the multi-step form logic. It expands the AddDataForm.svelte component from 689 lines to 990 lines, which is over-the-line. At this point, let's try to make this component incrementally smaller in every PR, not larger. At the very least, this PR should factor out the multi-step form logic.

royendo added a commit that referenced this pull request Oct 24, 2025
- Document new two-step authentication flow (Step 1: Auth → Step 2: Source)
- Add UI instructions for uploading service account JSON files
- Add detailed HMAC key generation and configuration steps
- Update YAML examples to show proper connector/source separation
- Clarify deployment requirements for Rill Cloud

Related to PR #8142

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@lovincyrus
Copy link
Contributor Author

lovincyrus commented Oct 28, 2025

These are acceptable codesmell as we plan on tackling this in https://linear.app/rilldata/issue/APP-449/composable-adddataform We covered this in our 1:1

This PR introduces significant complexity with the multi-step form logic. It expands the AddDataForm.svelte component from 689 lines to 990 lines, which is over-the-line. At this point, let's try to make this component incrementally smaller in every PR, not larger. At the very least, this PR should factor out the multi-step form logic.

I've broken down AddDataForm.svelte into modular components and w.r.t https://linear.app/rilldata/issue/APP-449/composable-adddataform. Tested the connectors manually and they look good to me!

@lovincyrus lovincyrus changed the title Explicit credentials for GCS GCS Explicit Credentials Oct 29, 2025
@lovincyrus lovincyrus mentioned this pull request Oct 29, 2025
8 tasks
@lovincyrus lovincyrus requested a review from royendo October 30, 2025 15:55
@royendo
Copy link
Contributor

royendo commented Oct 30, 2025

PRD review: fixed the issues reported in Linear, (gs text checker, public buckets via skip.).

Looks good! 💯

No need to fix in this PR, but if I type something in HMAC, hit skip and hit back the text is still there. also, click out of HMAC should clear the HMAC so it clears from Connector Preview.

Screenshot 2025-10-30 at 12 27 57

Copy link
Contributor

@royendo royendo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a comment to review, maybe a quick followup PR instead of this one.

@lovincyrus lovincyrus merged commit a643a6c into main Oct 30, 2025
12 of 13 checks passed
@lovincyrus lovincyrus deleted the cyrus/gcs-form branch October 30, 2025 16:48
grahamplata pushed a commit that referenced this pull request Oct 30, 2025
* remove gcs from maybeRewriteToDuckDb

* init spec change

* file input for big query

* working

* revert and new file input component

* swap spec bigquery sequence

* clean up credentials input, restrict to credentials file type

* add input label to credentials input

* css tweaks

* css tweaks

* clean up config properties

* lint

* lint

* auth method radio

* prettier

* rearrange, styles

* ui tweaks

* extract project_id from credentials JSON

* external link

* custom slot to support nested

* copy and clear credentials on method change

* hide content when switching option

* trash icon to remove uploaded credentials

* icon buttno

* copy command interaction

* include bigquery as part of form height

* hide credentials input when inferred is selected

* radio style tweaks

* credentials first

* refine custom slot for credentials option

* style tweaks for code copy block

* style

* persist credentials input filename

* copy update

* bump link

* inital

* revert and remove bigquery auth options

* revert

* remove source properties from gcs

* specify gcs in the connector type

* clean up

* clean up

* clean up

* remove special gcs form display for path and name

* attempt to fix e2e

* secret key id

* re-add source props path and name

* gcs: connector + model

* multi step connectors

* centralize connector step store

* hoist form height

* add e2e for multi step

* lint

* tweak multi step connectors

* filter out connector properties from step 2

* hide back btn in step 2

* add skip button for multi step connector

* reset connector step when closing modal

* hmac key id secret

* path validaton yupschema

* fix handle back logic

* initial break down AddDataForm

* lint

* clean up
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants