Jgk2k4/eng 84 import data from platforms#150
Closed
george-bobby wants to merge 2 commits intomainfrom
Closed
Conversation
…eng-84-import-data-from-platforms
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review infoConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Cache: Disabled due to Reviews > Disable Cache setting ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (48)
📝 WalkthroughSummary by CodeRabbitRelease Notes
WalkthroughThis PR introduces multi-provider import capabilities for Linear and Todoist alongside existing Slack support, adds comprehensive OAuth flows, implements a reusable import pipeline infrastructure, expands the assistant agent with additional tools and context, and includes new database tables for tracking imported data across platforms. Changes
Sequence Diagram(s)sequenceDiagram
participant User as User/Browser
participant NextApp as Next.js App
participant OAuth as OAuth Provider<br/>(Linear/Todoist)
participant ConvexHTTP as Convex HTTP<br/>Callback
participant ConvexMutation as Convex<br/>Mutation
User->>NextApp: Click "Connect [Platform]"
NextApp->>ConvexMutation: Call initiate[Platform]OAuth<br/>(workspaceId)
ConvexMutation->>ConvexMutation: Validate workspace membership<br/>Load credentials
ConvexMutation->>ConvexMutation: Build redirect URI + CSRF state
ConvexMutation-->>NextApp: Return OAuth URL
NextApp->>OAuth: Redirect to OAuth flow
OAuth->>User: Show consent screen
User->>OAuth: Grant permissions
OAuth->>NextApp: Redirect with code + state
NextApp->>ConvexHTTP: GET /import/[platform]/callback<br/>?code=...&state=...
ConvexHTTP->>OAuth: Exchange code for token
OAuth-->>ConvexHTTP: Return access/refresh tokens
ConvexHTTP->>ConvexMutation: Call store[Platform]Connection<br/>(tokens, metadata)
ConvexMutation->>ConvexMutation: Persist connection record
ConvexMutation-->>ConvexHTTP: Success
ConvexHTTP->>NextApp: 302 Redirect to workspace
NextApp->>User: Connection complete
sequenceDiagram
participant User as User
participant UI as UI Component
participant ConvexMutation as Convex<br/>start[Platform]Import
participant BackendJob as Background<br/>Job Runner
participant Provider as [Platform]<br/>ImportProvider
participant Storage as Storage &<br/>Database
participant Notify as OneSignal<br/>Notifications
User->>UI: Click "Start Import"<br/>+ configuration
UI->>ConvexMutation: Call start[Platform]Import<br/>(workspaceId, config)
ConvexMutation->>ConvexMutation: Validate member<br/>Load connection
ConvexMutation->>Storage: Create import job record<br/>with config
ConvexMutation-->>UI: Return jobId
UI->>User: "Import started..."
BackendJob->>Provider: process[Platform]Import(jobId)
Provider->>Provider: Validate connection
Provider->>Provider: Fetch workspace metadata
Provider->>Provider: Fetch users, channels/projects
Provider->>Storage: Store imported channels<br/>with external ID mapping
Provider->>Provider: Fetch messages (paginated)
Provider->>Storage: Store imported messages<br/>with idempotency keys
Provider->>Storage: Update progress<br/>& job status
BackendJob->>Notify: sendImportNotification<br/>(status, counts)
Notify-->>User: Push notification<br/>Import complete!
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds multi-platform data import support (Todoist + Linear alongside Slack) by introducing a reusable Convex import pipeline, wiring OAuth callback routes, and exposing new UI controls to connect/start imports from the workspace manage area.
Changes:
- Introduces a reusable Convex import pipeline and new import providers for Slack/Todoist/Linear.
- Adds OAuth callback handling (Next.js + Convex HTTP routes) and updates middleware to allow callback routes.
- Expands import UI and schema to support platform-specific import configuration and metadata tracking.
Reviewed changes
Copilot reviewed 48 out of 49 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test-tool-schemas.ts | Formatting/cleanup in tool schema test output. |
| tests/test-message.ts | Minor formatting + template literal cleanup for test conversation IDs. |
| tests/test-assistant.ts | Import order + formatting changes in assistant test harness. |
| tests/test-assistant-message.ts | Formatting cleanup for test client + separator output. |
| src/middleware.ts | Allows OAuth callback routes to bypass auth middleware. |
| src/lib/unified-tool-manager.ts | Refactors typings/import order and formatting; minor logging formatting. |
| src/lib/assistant-orchestration.ts | Formats connected-app tool policy prompt generation. |
| src/lib/ai-tool-selector.ts | Formatting + minor refactors for caching/filtering/tool catalog prompt. |
| src/lib/ai-query-classifier.ts | Formatting + minor refactors; keeps AI classification + fallback behavior. |
| src/lib/ai-multi-step-planner.ts | Formatting + minor refactors for planning/execution logging and loops. |
| src/lib/ai-confirmation-logic.ts | Formatting + minor refactors for prompt building and fallbacks. |
| src/lib/ai-cache.ts | Formats periodic cache eviction interval setup. |
| src/features/manage/import-data-management.tsx | Enables Todoist/Linear in UI, adds config toggles, and wires new connect/import hooks. |
| src/features/imports/api/use-start-todoist-import.ts | New hook to start Todoist import via Convex mutation. |
| src/features/imports/api/use-start-linear-import.ts | New hook to start Linear import via Convex mutation. |
| src/features/imports/api/use-initiate-todoist-oauth.ts | New hook to initiate Todoist OAuth via Convex mutation. |
| src/features/imports/api/use-initiate-linear-oauth.ts | New hook to initiate Linear OAuth via Convex mutation. |
| src/features/dashboard/components/dashboard-chatbot.tsx | Minor formatting / prop ordering tweak. |
| src/app/workspace/[workspaceId]/manage/page.tsx | Renames “Integrations” tab label to “AI Integrations”. |
| src/app/api/import/todoist/callback/route.ts | New Next.js route that redirects OAuth callback params to Convex handler. |
| src/app/api/import/linear/callback/route.ts | New Next.js route that redirects OAuth callback params to Convex handler. |
| src/app/api/assistant/composio/tools/route.ts | Minor formatting; adjusts import order and tool fetch call formatting. |
| src/app/api/assistant/chatbot/stream/route.ts | Import ordering/formatting and minor response formatting for streaming route. |
| src/app/api/assistant/chatbot/route.ts | Import ordering/formatting and minor response formatting for non-stream route. |
| convex/tsconfig.json | Formatting-only adjustments. |
| convex/todoistImportProvider.ts | New Todoist import provider + executor using shared import pipeline. |
| convex/testComposio.ts | Minor cleanup (unused ctx) + formatting. |
| convex/slackImportProvider.ts | New Slack import provider + executor using shared import pipeline. |
| convex/schema.ts | Adds import config fields + new import metadata tables (channel/message/file). |
| convex/ragchat.ts | Formatting-only changes in semantic reranking logic. |
| convex/onesignal.ts | Adds OneSignal notification internal actions (push + placeholder in-app). |
| convex/linearImportProvider.ts | New Linear import provider + executor using shared import pipeline. |
| convex/integrations.ts | Removes noisy debug logs and formats workspace account query. |
| convex/importPipeline.ts | Introduces shared import pipeline types/utilities (rate limiting, retry, idempotency, etc.). |
| convex/http.ts | Adds Convex HTTP OAuth callbacks for Todoist/Linear; adjusts site URL sourcing. |
| convex/chatbot.ts | Formatting-only changes to tool selection/external audit logging. |
| convex/assistantTools.ts | Formatting-only tweaks in date range filters. |
| convex/assistantComposioTools.ts | Formatting-only changes + minor cleanup of unused catch vars. |
| convex/assistantChat.ts | Formatting-only changes and minor log formatting adjustments. |
| convex/assistant/tools/internalTools.ts | Formatting-only changes to zod tool argument schemas. |
| convex/assistant/tools/index.ts | Import order tweak for tool exports. |
| convex/assistant/tools/composioTools.ts | Import ordering/formatting changes. |
| convex/assistant/errorHandling.ts | Formatting-only changes for long error message string. |
| convex/assistant/errorHandling.test.ts | Formatting-only changes in assertions. |
| convex/assistant/context.ts | Removes unused imports + formats mapping of recent tools. |
| convex/assistant/agent.ts | Import order/formatting for agent construction. |
| convex/_generated/api.d.ts | Updated generated API typings for newly added Convex modules. |
| README.md | Updates local dev URL in setup instructions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+723
to
+725
| disabled={ | ||
| startSlackImport.isPending || startTodoistImport.isPending | ||
| } |
There was a problem hiding this comment.
The “Start Import” button is disabled for Slack/Todoist pending states but not for Linear. This allows duplicate Linear import submissions while startLinearImport is in-flight. Include startLinearImport.isPending in the disabled condition (and ideally gate based on selectedPlatform).
| const siteUrl = | ||
| process.env.NEXT_PUBLIC_APP_URL || | ||
| process.env.SITE_URL || | ||
| "https://localhost:3000"; |
There was a problem hiding this comment.
Defaulting siteUrl to https://localhost:3000 is likely incorrect for local development (most dev servers run on http://localhost:3000). This affects OAuth redirect URIs and can break the auth flow locally. Prefer http://localhost:3000 as the fallback (or require NEXT_PUBLIC_APP_URL/SITE_URL to be set).
Suggested change
| "https://localhost:3000"; | |
| "http://localhost:3000"; |
|
|
||
| 5. **Run the app** | ||
| - Terminal 1 (Next.js): `bun next` → open [http://localhost:3000](http://localhost:3000) | ||
| - Terminal 1 (Next.js): `bun next` → open [https://localhost:3000](https://localhost:3000) |
There was a problem hiding this comment.
The README instructs opening https://localhost:3000, but the dev server is typically http://localhost:3000 unless TLS is configured. This can confuse contributors and break OAuth redirect assumptions in local dev.
Suggested change
| - Terminal 1 (Next.js): `bun next` → open [https://localhost:3000](https://localhost:3000) | |
| - Terminal 1 (Next.js): `bun next` → open [http://localhost:3000](http://localhost:3000) |
Comment on lines
+32
to
+34
| // Get user's OneSignal player ID from preferences or external player table | ||
| // For now, we'll broadcast to all users in the workspace | ||
| // In production, you'd store player IDs per user |
There was a problem hiding this comment.
This comment says “broadcast to all users in the workspace”, but the payload targets include_aliases.external_user_id for a single userId and sets include_player_ids to an empty array. Either update the comment to match the current behavior or implement actual workspace-wide targeting (e.g., by maintaining and supplying player IDs / tags / segments).
Suggested change
| // Get user's OneSignal player ID from preferences or external player table | |
| // For now, we'll broadcast to all users in the workspace | |
| // In production, you'd store player IDs per user | |
| // Target this notification to the specific user via OneSignal aliasing | |
| // (external_user_id). Workspace-wide broadcasting would require | |
| // maintaining player IDs / tags / segments for all workspace members. |
Comment on lines
+698
to
+703
| const _messageId = await ctx.runMutation<string>( | ||
| internal.importIntegrations.storeImportedMessage, | ||
| { | ||
| workspaceId: ctx.workspaceId, | ||
| memberId: ctx.memberId, | ||
| channelId: "" as any, // Not needed for replies |
There was a problem hiding this comment.
storeImportedMessage requires a valid channelId (Convex validator is v.id("channels")), but comment replies are being stored with channelId: "" as any. This will fail at runtime and also loses the channel association for replies. Pass the parent task's channelId (projectMap lookup) when storing comments, or change the storage API to accept optional channelId for replies and derive it from the parent message.
Suggested change
| const _messageId = await ctx.runMutation<string>( | |
| internal.importIntegrations.storeImportedMessage, | |
| { | |
| workspaceId: ctx.workspaceId, | |
| memberId: ctx.memberId, | |
| channelId: "" as any, // Not needed for replies | |
| // Derive the channelId from the Todoist project associated with this comment | |
| // so that replies are stored in the same channel as their parent task. | |
| const projectChannel: ExternalChannel | undefined = | |
| ctx.projectMap && comment.project_id | |
| ? ctx.projectMap.get(comment.project_id) | |
| : undefined; | |
| if (!projectChannel || !projectChannel.channelId) { | |
| throw new Error( | |
| `Missing channel mapping for Todoist project ${comment.project_id} when storing comment ${comment.id}` | |
| ); | |
| } | |
| const _messageId = await ctx.runMutation<string>( | |
| internal.importIntegrations.storeImportedMessage, | |
| { | |
| workspaceId: ctx.workspaceId, | |
| memberId: ctx.memberId, | |
| channelId: projectChannel.channelId, |
Comment on lines
+193
to
+210
| const url = "https://api.todoist.com/rest/v2/projects"; | ||
| console.log("[TodoistValidate] Calling URL:", url); | ||
| console.log("[TodoistValidate] Token present:", !!ctx.accessToken); | ||
|
|
||
| try { | ||
| await ctx.log("info", "Validating Todoist connection"); | ||
| // Test the connection by fetching projects (lightweight validation) | ||
| const response = await fetch(url, { | ||
| headers: { | ||
| Authorization: `Bearer ${ctx.accessToken}`, | ||
| }, | ||
| }); | ||
|
|
||
| console.log("[TodoistValidate] Response status:", response.status); | ||
|
|
||
| if (!response.ok) { | ||
| const errorText = await response.text(); | ||
| console.log("[TodoistValidate] Error response:", errorText); |
There was a problem hiding this comment.
validateConnection contains multiple console.log statements (URL, token presence, response status, error text, exception). This will be noisy in production Convex logs and may leak operational details. Prefer ctx.log with appropriate level, and avoid logging token presence/error bodies unless explicitly needed for debugging behind a flag.
|
|
||
| constructor() { | ||
| this.rateLimiter = new RateLimiter({ | ||
| minDelay: 100, // Todoist allows ~50 requests per minute |
There was a problem hiding this comment.
The rate limiter configuration is internally inconsistent: minDelay: 100ms implies up to ~600 requests/minute, but the comment says Todoist allows ~50 requests/minute. Either increase minDelay (e.g., ~1200ms) or update the comment so it matches the intended request rate to avoid accidental 429s and confusion.
Suggested change
| minDelay: 100, // Todoist allows ~50 requests per minute | |
| minDelay: 1200, // ~50 requests per minute |
| { | ||
| workspaceId: ctx.workspaceId, | ||
| memberId: ctx.memberId, | ||
| channelId: "" as any, |
There was a problem hiding this comment.
Comment replies are stored with channelId: "" as any, but storeImportedMessage validates channelId as v.id("channels"). This will throw at runtime. Replies should carry the same channelId as the parent issue/team channel (look it up when calling storeComment, or include it in the function signature).
Suggested change
| channelId: "" as any, | |
| channelId: ctx.channelId as any, |
Contributor
DeepSource Code ReviewWe reviewed changes in See full review on DeepSource ↗ Code Review Summary
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recordings
📹 View Recording
Code Quality
/deslopbun typeand ensured type safetybun checkand passed biome checksPerformance & Security
bun iafter adding dependenciesPR Comments
Reviewer Checklist: