Migration System Meta-Issue

This meta-issue tracks all problems and feature requests related to the Knex migration system. Migrations are a critical part of database schema management, and issues in this system affect many users across different databases and environments.

General

• Issue: Add --seeds-directory option #3356 — Add --seeds-directory CLI option to mirror --migrations-directory. Today seed directory is only set via knexfile (config.seeds.directory); no CLI override exists. Code: bin/cli.js:83–119 (global options) and 320–370 (seed commands); no seeds‑directory flag. - Impact: Improves DX for multi‑env tooling and scripts that pass directories
  at runtime. - Proposal: Add Commander option --seeds-directory, merge into config.seeds.directory similar to migrations; ensure seeder merger respects it.
• Status: Needs design/PR; low risk, backwards‑compatible.

Core Migration System Issues

Logging & Observability (Runner/CLI)

• Issue: Is there an option to log the sucessfully completed migration file name #5256 - Log each completed migration name. Today the runner records
  { name, batch, migration_time } and returns [batchNo, log]; the CLI
  prints batch summaries and, with --verbose, file names, but there is no
  dedicated per‑migration completion log/hook for programmatic use. Code: lib/
  migrations/migrate/Migrator.js:498–530; bin/cli.js:236–258 (latest), 275–
  302 (rollback).
• Impact: Improves auditability/observability, especially outside the CLI and
  in centralised logging setups.
• Current behaviour: Names are available indirectly (tuple return, CLI
  verbose), but not emitted as events or via a config toggle.
• Proposal: Add a configuration flag or an event (e.g. migrate:completed)
  that emits { name, batch, time } per migration. Default off to avoid noise;
  document alongside CLI verbose.
• Status: Needs design/PR; low risk and backward‑compatible. Consider
  cross‑link with Unify logging, getting rid of debug #6231 (logging unification).

Migration Corruption and File Management

• Migration Directory has been corrupted.  #5383 - Migration Directory has been corrupted
  • Error: "The migration directory is corrupt, the following files are missing"
  • Happens when migration files referenced in DB don't exist in filesystem
  • Can occur when switching projects or incorrectly managing migration directories
  • Related: knex migrations directory is corrupt error in production environment  #5177 (closed as duplicate)

Migration Order Ergonomics

• Issue: Execute migration with wrong numer after merge lead to unexpected number error #5543 — Migration ordering relies on lexicographic filenames. After
  merges, non‑padded/duplicated numeric prefixes can produce surprising
  execution order and user‑facing confusion. Behaviour confirmed in
  code: filenames are string‑sorted in lib/migrations/migrate/sources/
  fs-migrations.js:34–53; completed migrations are ordered by id in lib/
  migrations/migrate/migration-list-resolver.js:5–17, which amplifies
  mismatches post‑merge. (closed as duplicate)
  • Impact: Unexpected run order, especially for cross‑branch merges; can cause
    “relation does not exist” or similar when dependencies are created out of
    order.
  • Current guidance: Use consistent, sortable prefixes (timestamps preferred).
    Rename conflicting files post‑merge to enforce correct order.
  • Proposal: Add an optional configuration flag to enforce a prefix scheme
    (timestamp or strictly padded numeric), validate at startup, and emit a
    descriptive error listing offending filenames. Consider a CLI opt‑in (e.g.
    --enforce-migration-prefix) and a docs note. This would be non‑breaking by
    default (off unless configured).

Transaction Handling

• disableTransactions: true uses transaction when migrating #5359 - disableTransactions: true uses transaction when migrating (NG_bug)
  • Configuration option disableTransactions: true is ignored
  • Causes "CREATE INDEX CONCURRENTLY cannot run inside a transaction block" errors
  • Affects PostgreSQL concurrent index creation
  • Related: migrate:up not respecting disabled transactions #4551 (closed as duplicate)

Migration Locking and Concurrency

• Can't take lock to run migrations: Migration table is already locked #5086 - Can't take lock to run migrations: Migration table is already locked (NG_bug)
  • Race conditions when multiple processes try to run migrations simultaneously
  • Lock table initialization is not safe for concurrent access
  • Can leave database in locked state requiring manual intervention
  • Related issues:
    • Migrations lock index sequence already exists #3457 - Migrations lock index sequence already exists
    • creation of rows in knex_migration_lock table is not safe concurrently #3538 - Creation of rows in knex_migration_lock table is not safe concurrently

ESM and TypeScript Support

• Proper support for Typescript and ESM by migrating to better libraries/frameworks: Jest to Vitest, ts-node/ts-register to tsx #5800 - Proper support for TypeScript and ESM by migrating to better libraries/frameworks (NG_feature_request, NG_PRIORITY)
  • Request to upgrade tooling: Jest → Vitest, ts-node → tsx
  • Better ESM and TypeScript support for migrations and seeds
  • Enable .mjs migration files
  • Related issues:
    • Feature Request: .mjs (es module) migrations and seeds (no cli) #3850 - Feature Request: .mjs (es module) migrations and seeds (closed as duplicate)
    • Add 'cjs' to list of Seeder.js extensions to support Node "type": "Module" projects #4381 - Add 'cjs' to list of Seeder.js extensions to support Node "type": "Module" projects

Schema Management

• Create schema as well as knex_migrations table when running migrations for the first time. #5374 - Create schema as well as knex_migrations table when running migrations for the first time (NG_feature_request)
  • Migration tables should respect and create configured schema
  • Currently only creates tables in default schema
  • Related: PostgreSQL: migrations table created both in public and target schema #4563 - PostgreSQL: migrations table created both in public and target schema

Impact Areas

Database Support

• PostgreSQL: Transaction handling (disableTransactions: true uses transaction when migrating #5359), concurrent operations, schema management (Create schema as well as knex_migrations table when running migrations for the first time. #5374, PostgreSQL: migrations table created both in public and target schema #4563)
• Multi-database: All databases affected by locking issues (Can't take lock to run migrations: Migration table is already locked #5086, Migrations lock index sequence already exists #3457, creation of rows in knex_migration_lock table is not safe concurrently #3538) and corruption (Migration Directory has been corrupted.  #5383)

Deployment Scenarios

• Multi-server deployments: Concurrency and locking issues critical
• CI/CD pipelines: Migration reliability essential
• Development workflows: File management and corruption issues

Developer Experience

• Modern JavaScript: ESM/TypeScript support (Proper support for Typescript and ESM by migrating to better libraries/frameworks: Jest to Vitest, ts-node/ts-register to tsx #5800, Feature Request: .mjs (es module) migrations and seeds (no cli) #3850, Add 'cjs' to list of Seeder.js extensions to support Node "type": "Module" projects #4381)
• Configuration: Transaction settings not working as expected (disableTransactions: true uses transaction when migrating #5359)
• Error handling: Unclear errors and locked states difficult to debug

Root Causes

1. Concurrency: Migration lock table initialization has race conditions
2. File tracking: Migration system doesn't handle file deletions/moves gracefully
3. Configuration: Some migration options not properly implemented
4. Modern tooling: Built on older CommonJS patterns, lacking full ESM support
5. Schema awareness: Migration tables don't properly respect schema configuration

Proposed Solutions

Short Term

1. Fix transaction configuration handling (disableTransactions: true uses transaction when migrating #5359)
2. Improve migration lock table initialization safety (Can't take lock to run migrations: Migration table is already locked #5086)
3. Better error messages for migration corruption (Migration Directory has been corrupted.  #5383)
4. Add .cjs extension support (Add 'cjs' to list of Seeder.js extensions to support Node "type": "Module" projects #4381)

Long Term

1. Migrate to modern tooling for ESM/TypeScript (Proper support for Typescript and ESM by migrating to better libraries/frameworks: Jest to Vitest, ts-node/ts-register to tsx #5800)
2. Redesign migration locking mechanism for better concurrency
3. Add schema-aware migration table creation (Create schema as well as knex_migrations table when running migrations for the first time. #5374)
4. Improve migration file tracking and validation

Related Work

This meta-issue focuses on the migration system itself. Related but separate concerns:

• Seed system: Tracked separately (proposed META)
• Schema builder: Schema/DDL issues tracked separately
• Database drivers: Driver-specific issues in their respective METAs

Conclusion

The migration system is fundamental to Knex's value proposition. Addressing these issues will improve reliability, support modern JavaScript patterns, and provide better developer experience across all deployment scenarios.