-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Description
Practical action for this issue: update documentation and explain somewhere that knex will never implement returning of nested queries the way an ORM would.
Knex does not transform flat SQL result sets into nested objects
Knex is a SQL query builder, not an ORM (Object-Relational Mapper). One of the most common questions from users is how to get nested/hierarchical data structures from joined queries, where child records are grouped into arrays under their parent records. This issue consolidates all related questions and explains why this is outside Knex's scope.
The Core Issue
When you perform a JOIN query in SQL, the database returns a flat result set with repeated parent data for each child record. For example:
// This query
knex('user_groups')
.leftJoin('users', 'users.user_group_id', 'user_groups.id')
.where('user_groups.branch_id', 10)
.select('*');Returns a flat structure like:
[
{ id: 1, name: 'Admin', branch_id: 10, user_id: 1, user_name: 'Alice' },
{ id: 1, name: 'Admin', branch_id: 10, user_id: 2, user_name: 'Bob' },
{ id: 1, name: 'Admin', branch_id: 10, user_id: 3, user_name: 'Charlie' }
]Many users expect or want nested results like:
[
{
id: 1,
name: 'Admin',
branch_id: 10,
users: [
{ user_id: 1, user_name: 'Alice' },
{ user_id: 2, user_name: 'Bob' },
{ user_id: 3, user_name: 'Charlie' }
]
}
]Why Knex Doesn't Do This
As stated by maintainer @tgriesser in #61: "this is a limitation of sql... and not something knex will try to handle for you provide out of the box".
Knex's philosophy is to be a thin, portable SQL query builder that mirrors SQL semantics across different databases. Data transformation and object mapping are explicitly outside its scope. This design decision keeps Knex:
- Lightweight - No complex data mapping or hydration logic
- Predictable - What you see in SQL is what you get in JavaScript
- Portable - Works identically across all supported databases
- Fast - No transformation overhead
- Simple - Easier to maintain and reason about
Available Solutions
If you need nested results from joined queries, you have several options:
1. Use an ORM that provides this feature
ORMs like Objection.js (built on Knex) and Bookshelf provide relationship management and nested data fetching:
// Objection.js example
const groups = await UserGroup.query()
.where('branch_id', 10)
.withGraphFetched('users');As mentioned by maintainer @elhigu in #882: "objection.js and bookshelf allow fetching relations as nested data."
2. Use database-specific JSON functions
Modern databases provide native JSON aggregation functions that can build nested structures in SQL:
PostgreSQL:
knex('user_groups')
.select([
'user_groups.*',
knex.raw('COALESCE(json_agg(users.*) FILTER (WHERE users.id IS NOT NULL), \'[]\') as users')
])
.leftJoin('users', 'users.user_group_id', 'user_groups.id')
.where('user_groups.branch_id', 10)
.groupBy('user_groups.id');MySQL (with `nestTables` driver option):
knex('user_groups')
.leftJoin('users', 'users.user_group_id', 'user_groups.id')
.where('user_groups.branch_id', 10)
.options({ nestTables: true }); // MySQL-specific driver optionThis returns objects grouped by table name:
[
{
user_groups: { id: 1, name: 'Admin', branch_id: 10 },
users: { user_id: 1, user_name: 'Alice' }
},
// ... more rows
]Note: These approaches are database-specific and require manual post-processing.
3. Manual transformation with JavaScript
Transform the flat results yourself using libraries like lodash or custom logic:
const results = await knex('user_groups')
.leftJoin('users', 'users.user_group_id', 'user_groups.id')
.where('user_groups.branch_id', 10)
.select(
'user_groups.id',
'user_groups.name',
'user_groups.branch_id',
'users.id as user_id',
'users.name as user_name'
);
// Group manually
const grouped = results.reduce((acc, row) => {
let group = acc.find(g => g.id === row.id);
if (!group) {
group = {
id: row.id,
name: row.name,
branch_id: row.branch_id,
users: []
};
acc.push(group);
}
if (row.user_id) {
group.users.push({
user_id: row.user_id,
user_name: row.user_name
});
}
return acc;
}, []);4. Use a dedicated nesting library
Libraries like KnexNest (NestHydrationJS) provide automatic nesting based on column naming conventions:
const KnexNest = require('knex-nest');
const results = await knex('user_groups')
.leftJoin('users', 'users.user_group_id', 'user_groups.id')
.where('user_groups.branch_id', 10)
.select(
'user_groups.id AS _id',
'user_groups.name AS _name',
'user_groups.branch_id AS _branch_id',
'users.id AS _users__id',
'users.name AS _users__name'
);
const nested = KnexNest(results);5. Make separate queries
Often the cleanest approach is to fetch parent and child records separately:
// Fetch parent records
const groups = await knex('user_groups')
.where('branch_id', 10)
.select('*');
// Fetch all related users in one query
const groupIds = groups.map(g => g.id);
const users = await knex('users')
.whereIn('user_group_id', groupIds)
.select('*');
// Group users by parent ID
const usersByGroupId = users.reduce((acc, user) => {
if (!acc[user.user_group_id]) acc[user.user_group_id] = [];
acc[user.user_group_id].push(user);
return acc;
}, {});
// Attach users to groups
const result = groups.map(group => ({
...group,
users: usersByGroupId[group.id] || []
}));This approach:
- Avoids duplicate parent data in result sets
- Often performs better than complex joins
- Makes it easier to cache parent and child data separately
- Is the pattern used by most ORMs internally
Common Related Issues
Column name collisions: When joining tables with identically named columns (like `id`), the child table values overwrite parent values. Solution: Use explicit column selection with aliases:
knex('posts')
.leftJoin('comments', 'comments.post_id', 'posts.id')
.select(
'posts.id as post_id',
'posts.name as post_name',
'comments.id as comment_id',
'comments.name as comment_name'
);Multiple executions: Remember that Knex query builders are "thenable" objects. Awaiting or calling `.then()` multiple times on the same query builder executes the query each time. Store the promise result, not the query builder.
Maintainer Position
This is a deliberate design decision that will not change. From the maintainers:
- @tgriesser (creator): "this is a limitation of sql... and not something knex will try to handle for you provide out of the box"
- @elhigu (maintainer): Directed users to ORMs like Objection.js and Bookshelf for this functionality
Knex intentionally remains a query builder, not an ORM. Users needing ORM features should use an ORM built on top of Knex (like Objection.js) or alongside it.
Related Issues (Duplicates)
Exact duplicates asking for nested results from joins:
- How to generated nested results ? #5974 - How to generate nested results?
- How to return nested Json object MSSQL #4773 - How to return nested JSON object MSSQL
- Muliple Join response in nested json format #4432 - Multiple Join response in nested json format
- Getting nested data when using Joins #882 - Getting nested data when using Joins
- Question around a simple join... #260 - Question around a simple join
Related to column collisions in joins:
- Joined tables overwrite columns with the same name #61 - Joined tables overwrite columns with the same name
Related to extending query builder for nested data:
- Correct way to extends query builder to get nested data? #6181 - Correct way to extends query builder to get nested data?
Related ORM/relationship discussions:
- Error: Connection lost: The server closed the connection. #1072 - Possibility of ORM-like extensions (closed as not planned)
Recommendation
If you're reading this because you opened an issue asking how to get nested results: Knex is not the right tool for this job. Use an ORM like Objection.js, use database-specific JSON functions, or transform the data manually. Knex will continue to return flat result sets that mirror standard SQL behaviour.