Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Comments

♻️ refactor: refactor database schema#10860

Merged
arvinxx merged 2 commits intonextfrom
refactor/database-schema
Dec 20, 2025
Merged

♻️ refactor: refactor database schema#10860
arvinxx merged 2 commits intonextfrom
refactor/database-schema

Conversation

@arvinxx
Copy link
Member

@arvinxx arvinxx commented Dec 20, 2025

💻 Change Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 👷 build
  • ⚡️ perf
  • ✅ test
  • 📝 docs
  • 🔨 chore

🔗 Related Issue

🔀 Description of Change

🧪 How to Test

  • Tested locally
  • Added/updated tests
  • No tests needed

📸 Screenshots / Videos

Before After
... ...

📝 Additional Information

Summary by Sourcery

Update database and type schemas to support richer threading, messaging, agent, and user personalization metadata, along with corresponding migrations and tooling adjustments.

New Features:

  • Add thread metadata, agent, and group associations to enable tracking of agent task execution and grouping within topics.
  • Extend message groups and messages with compression-related fields and per-message summaries.
  • Add user interests, onboarding state, and memory settings to support personalized onboarding and memory configuration.
  • Introduce user onboarding types and validation schema for strongly-typed onboarding data.

Enhancements:

  • Refine user memories context typing, remove unused title vector and index, and rely solely on description vectors.
  • Add user- and tool-related indexes (AI providers/models, message groups, message plugins, threads) to improve query performance.
  • Ensure accessed_at timestamps auto-update on modification and adjust agents schema to make plugins nullable and support pinned agents.
  • Align lint and type-check npm scripts for consistency with tooling expectations.

Build:

  • Rename the typecheck script to type-check and update the main lint command to use the new name.

@vercel
Copy link

vercel bot commented Dec 20, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
lobehub Ready Ready Preview, Comment Dec 20, 2025 5:18am

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Dec 20, 2025
@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Dec 20, 2025

Reviewer's Guide

Refactors and extends the database schema and shared types to support richer threading/agent metadata, compressed message groups, memory/onboarding fields on users, and new indexes, with a matching migration and minor tooling/script updates.

ER diagram for updated database schema (threads, users, agents, memories, AI infra)

erDiagram
  users {
    text id
    text email
    varchar[] interests
    jsonb onboarding
  }

  user_settings {
    text id
    text user_id
    jsonb system_agent
    jsonb default_agent
    jsonb market
    jsonb memory
    jsonb tool
    jsonb image
  }

  agents {
    text id
    text user_id
    jsonb plugins
    boolean pinned
  }

  chat_groups {
    text id
    text user_id
  }

  threads {
    text id
    text topic_id
    text parent_thread_id
    text client_id
    text user_id
    text agent_id
    text group_id
    jsonb metadata
    text type
    text status
  }

  message_groups {
    text id
    text topic_id
    text user_id
    text client_id
    text title
    text description
    text type
    text content
    jsonb editor_data
  }

  messages {
    text id
    text group_id
    text user_id
    text role
    text content
    jsonb editor_data
    text summary
  }

  message_plugins {
    text id
    text message_id
    text tool_call_id
    text user_id
    text client_id
  }

  user_memories_contexts {
    text id
    text user_id
    jsonb user_memory_ids
    jsonb metadata
    text[] tags
    jsonb associated_objects
    jsonb associated_subjects
    text title
    text description
    vector description_vector
  }

  ai_providers {
    text id
    text user_id
    text provider
  }

  ai_models {
    text id
    text provider_id
    text user_id
    text model
  }

  %% Relationships
  users ||--o{ user_settings : has
  users ||--o{ agents : owns
  users ||--o{ chat_groups : owns
  users ||--o{ threads : owns
  users ||--o{ message_groups : owns
  users ||--o{ messages : owns
  users ||--o{ message_plugins : owns
  users ||--o{ user_memories_contexts : owns
  users ||--o{ ai_providers : configures
  users ||--o{ ai_models : configures

  agents ||--o{ threads : assigned_to

  chat_groups ||--o{ threads : grouped_in

  message_groups ||--o{ messages : contains

  messages ||--o{ message_plugins : uses

  ai_providers ||--o{ ai_models : offers

  threads ||--o{ threads : parent_of

  user_memories_contexts ||--o{ user_memories_contexts : related_to
Loading

Class diagram for updated types and metadata (ThreadMetadata, UserOnboarding, and related models)

classDiagram
  class ThreadMetadata {
    +string completedAt
    +number duration
    +string error
    +string operationId
    +string startedAt
    +number totalCost
    +number totalMessages
    +number totalTokens
    +number totalToolCalls
  }

  class UserOnboarding {
    +number currentStep
    +string finishedAt
    +number version
  }

  class AgentRow {
    +string id
    +string userId
    +string[] plugins
    +boolean pinned
  }

  class UserRow {
    +string id
    +string email
    +string firstName
    +string lastName
    +string fullName
    +string[] interests
    +UserOnboarding onboarding
  }

  class UserSettingsRow {
    +string id
    +string userId
    +any systemAgent
    +any defaultAgent
    +any market
    +any memory
    +any tool
    +any image
  }

  class MessageGroupRow {
    +string id
    +string topicId
    +string userId
    +string clientId
    +string title
    +string description
    +string type
    +string content
    +any editorData
  }

  class MessageRow {
    +string id
    +string groupId
    +string userId
    +string role
    +string content
    +any editorData
    +string summary
  }

  class UserMemoriesContextRow {
    +string id
    +string userId
    +string[] userMemoryIds
    +Record~string, unknown~ metadata
    +string[] tags
    +AssociatedObject[] associatedObjects
    +AssociatedObject[] associatedSubjects
    +string title
    +string description
  }

  class AssociatedObject {
    +Record~string, unknown~ extra
    +string name
    +string type
  }

  class ThreadRow {
    +string id
    +string topicId
    +string parentThreadId
    +string clientId
    +string userId
    +string agentId
    +string groupId
    +ThreadMetadata metadata
    +string type
    +string status
  }

  UserRow "1" --> "1" UserOnboarding : has
  ThreadRow "1" --> "1" ThreadMetadata : has
  UserRow "1" --> "*" AgentRow : owns
  UserRow "1" --> "1" UserSettingsRow : owns
  UserRow "1" --> "*" MessageGroupRow : owns
  MessageGroupRow "1" --> "*" MessageRow : contains
  UserRow "1" --> "*" UserMemoriesContextRow : owns
  UserMemoriesContextRow "*" --> "*" AssociatedObject : references
  AgentRow "1" --> "*" ThreadRow : assigned_to
Loading

File-Level Changes

Change Details Files
Extend thread and agent-related schema to support agent execution metadata and relationships.
  • Add ThreadMetadata interface in shared types for task timing, cost, and usage metrics
  • Extend threads table with new status values (completed, failed), agentId, groupId, and metadata JSON, plus indexes on agentId and groupId
  • Wire threads.agent_id and threads.group_id to agents and chat_groups via foreign keys in migration
  • Add pinned flag and remove default on plugins in agents table and migration
packages/types/src/topic/thread.ts
packages/database/src/schemas/topic.ts
packages/database/src/schemas/agent.ts
packages/database/migrations/0063_add_columns_for_several_tables.sql
Support compressed/parallel message groups and richer message/plugin querying.
  • Add type, content, and editorData columns to messageGroups schema for compression metadata and index type
  • Add summary column to messages schema
  • Add index on toolCallId for messagePlugins and align Drizzle callback shape to an array
  • Create corresponding columns and indexes in the migration
packages/database/src/schemas/message.ts
packages/database/migrations/0063_add_columns_for_several_tables.sql
Refine user memories context structure and drop unused title vector index.
  • Strongly type userMemoryIds, associatedObjects, and associatedSubjects JSON fields for userMemoriesContexts
  • Remove titleVector column and its HNSW index from schema and migration, keeping only descriptionVector index
packages/database/src/schemas/userMemories.ts
packages/database/migrations/0063_add_columns_for_several_tables.sql
Add onboarding and memory-related fields and types for users and user settings.
  • Introduce UserOnboarding interface and Zod schema and export from user types index
  • Add interests array and onboarding JSON to users table
  • Add memory JSON column to userSettings table
  • Create matching columns in migration
packages/types/src/user/onboarding.ts
packages/types/src/user/index.ts
packages/database/src/schemas/user.ts
packages/database/src/schemas/userMemories.ts
packages/database/migrations/0063_add_columns_for_several_tables.sql
Improve AI infra querying and timestamp behavior helpers.
  • Add userId indexes to aiProviders and aiModels tables and wire up in migration
  • Update accessedAt helper column to auto-update on modification using $onUpdate
packages/database/src/schemas/aiInfra.ts
packages/database/src/schemas/_helpers.ts
packages/database/migrations/0063_add_columns_for_several_tables.sql
Align package scripts with new type-check command name and keep migration metadata in sync.
  • Rename root npm script from typecheck to type-check and update lint pipeline to call it
  • Add SQL migration entry and snapshot files to migrations metadata and core migrations list
  • Update DBML and migration journal/snapshot to reflect new schema (no functional logic changes)
package.json
packages/database/src/core/migrations.json
packages/database/migrations/meta/_journal.json
docs/development/database-schema.dbml
packages/database/migrations/meta/0063_snapshot.json

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@gru-agent
Copy link
Contributor

gru-agent bot commented Dec 20, 2025

⏳ Processing in progress

@arvinxx arvinxx changed the title refactor/database-schema ♻️ refactor: refactor database schema Dec 20, 2025
@codecov
Copy link

codecov bot commented Dec 20, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.31%. Comparing base (5cc9141) to head (378e368).
⚠️ Report is 2 commits behind head on next.

Additional details and impacted files
@@           Coverage Diff            @@
##             next   #10860    +/-   ##
========================================
  Coverage   80.31%   80.31%            
========================================
  Files         980      980            
  Lines       66983    66983            
  Branches     9153     8780   -373     
========================================
  Hits        53800    53800            
  Misses      13183    13183            
Flag Coverage Δ
app 73.10% <ø> (ø)
database 98.25% <ø> (ø)
packages/agent-runtime 98.08% <ø> (ø)
packages/context-engine 91.61% <ø> (ø)
packages/conversation-flow 98.05% <ø> (ø)
packages/electron-server-ipc 93.76% <ø> (ø)
packages/file-loaders 92.21% <ø> (ø)
packages/model-bank 100.00% <ø> (ø)
packages/model-runtime 89.60% <ø> (ø)
packages/prompts 79.17% <ø> (ø)
packages/python-interpreter 96.50% <ø> (ø)
packages/utils 95.31% <ø> (ø)
packages/web-crawler 96.81% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Store 73.05% <ø> (ø)
Services 56.44% <ø> (ø)
Server 75.19% <ø> (ø)
Libs 39.30% <ø> (ø)
Utils 82.05% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • The agents.plugins column no longer has a default while the Drizzle type is still string[]; consider either keeping a non-null default or updating the type/usage to safely handle null to avoid runtime assumptions of an array.
  • The new users.onboarding jsonb column is nullable in the schema but the UserOnboarding type requires a version field; make sure call sites either treat the column as optional/nullable or initialize a default shape to prevent undefined version values at runtime.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `agents.plugins` column no longer has a default while the Drizzle type is still `string[]`; consider either keeping a non-null default or updating the type/usage to safely handle `null` to avoid runtime assumptions of an array.
- The new `users.onboarding` jsonb column is nullable in the schema but the `UserOnboarding` type requires a `version` field; make sure call sites either treat the column as optional/nullable or initialize a default shape to prevent undefined `version` values at runtime.

## Individual Comments

### Comment 1
<location> `packages/database/src/schemas/agent.ts:40` </location>
<code_context>
     backgroundColor: text('background_color'),
     marketIdentifier: text('market_identifier'),

-    plugins: jsonb('plugins').$type<string[]>().default([]),
+    plugins: jsonb('plugins').$type<string[]>(),

     clientId: text('client_id'),
</code_context>

<issue_to_address>
**issue (bug_risk):** Dropping the `plugins` default may introduce `null` where code expects an array.

Without the `[]` default (and with `ALTER TABLE "agents" ALTER COLUMN "plugins" DROP DEFAULT;`), `plugins` can now be `NULL` for both existing and new rows. Any code that assumes this is always an array (e.g., iterating/spreading without a null check) may now throw at runtime. Consider keeping the default, adding a non-null constraint with a default, or updating all call sites to handle `null` explicitly.
</issue_to_address>

### Comment 2
<location> `packages/types/src/user/onboarding.ts:14` </location>
<code_context>
+export interface UserOnboarding {
+  /** Current step number (1-based), for resuming onboarding */
+  currentStep?: number;
+  /** Timestamp when onboarding was completed (ISO 8601) */
+  finishedAt?: string;
+  /** Onboarding flow version for future upgrades */
+  version: number;
</code_context>

<issue_to_address>
**suggestion:** The schema doesn’t enforce the documented ISO 8601 format for `finishedAt`.

Docs promise an ISO 8601 timestamp, but the schema only uses a plain `string`. To align them, consider `z.string().datetime()` (if supported) or a custom refinement to validate ISO 8601; otherwise the documented contract and actual validation will diverge.

```suggestion
  finishedAt: z.string().datetime().optional(),
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.


export const UserOnboardingSchema = z.object({
currentStep: z.number().optional(),
finishedAt: z.string().optional(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: The schema doesn’t enforce the documented ISO 8601 format for finishedAt.

Docs promise an ISO 8601 timestamp, but the schema only uses a plain string. To align them, consider z.string().datetime() (if supported) or a custom refinement to validate ISO 8601; otherwise the documented contract and actual validation will diverge.

Suggested change
finishedAt: z.string().optional(),
finishedAt: z.string().datetime().optional(),

@arvinxx arvinxx merged commit 5c489bc into next Dec 20, 2025
53 checks passed
@arvinxx arvinxx deleted the refactor/database-schema branch December 20, 2025 12:20
@lobehubbot
Copy link
Member

❤️ Great PR @arvinxx ❤️

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.

lobehubbot pushed a commit that referenced this pull request Dec 20, 2025
## [Version&nbsp;2.0.0-next.174](v2.0.0-next.173...v2.0.0-next.174)
<sup>Released on **2025-12-20**</sup>

#### ♻ Code Refactoring

- **misc**: Refactor database schema.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### Code refactoring

* **misc**: Refactor database schema, closes [#10860](#10860) ([5c489bc](5c489bc))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
@lobehubbot
Copy link
Member

🎉 This PR is included in version 2.0.0-next.174 🎉

The release is available on:

Your semantic-release bot 📦🚀

JamieStivala pushed a commit to jaworldwideorg/OneJA-Bot that referenced this pull request Dec 29, 2025
## [Version&nbsp;1.146.0](v1.145.1...v1.146.0)
<sup>Released on **2025-12-29**</sup>

#### ♻ Code Refactoring

- **misc**: Refactor database schema.

#### ✨ Features

- **ci**: Add bundle analyzer workflow.
- **misc**: Mobile native better auth support.

#### 🐛 Bug Fixes

- **ci**: Always continue build to upload bundle analyzer report, skip backend routes in bundle analyzer build.
- **scripts**: Fix syntax error in prebuild.mts.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### Code refactoring

* **misc**: Refactor database schema, closes [lobehub#10860](https://github.com/jaworldwideorg/OneJA-Bot/issues/10860) ([5c489bc](5c489bc))

#### What's improved

* **ci**: Add bundle analyzer workflow, closes [lobehub#10932](https://github.com/jaworldwideorg/OneJA-Bot/issues/10932) ([c470cfb](c470cfb))
* **misc**: Mobile native better auth support, closes [lobehub#10871](https://github.com/jaworldwideorg/OneJA-Bot/issues/10871) ([8c42a93](8c42a93))

#### What's fixed

* **ci**: Always continue build to upload bundle analyzer report, closes [lobehub#10946](https://github.com/jaworldwideorg/OneJA-Bot/issues/10946) ([8d37811](8d37811))
* **ci**: Skip backend routes in bundle analyzer build, closes [lobehub#10944](https://github.com/jaworldwideorg/OneJA-Bot/issues/10944) ([0276b87](0276b87))
* **scripts**: Fix syntax error in prebuild.mts, closes [lobehub#10952](https://github.com/jaworldwideorg/OneJA-Bot/issues/10952) ([3d46c13](3d46c13))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released on @next size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants