GitLab to GitHub Migrator

A comprehensive service for migrating repositories from GitLab (self-hosted or cloud) to GitHub, with support for nested repositories, complete metadata migration, and flexible naming strategies.

Features

1. Nested Repository Handling

• Problem: GitLab supports nested groups/subdirectories (e.g., group/subgroup/repo), but GitHub doesn't support nested repositories
• Solution: Multiple naming strategies to convert nested GitLab paths to flat GitHub repository names:
  • Flatten Strategy: group/subgroup/repo → group-subgroup-repo
  • Prefix Strategy: group/subgroup/repo → group-repo
  • Last Group Strategy: Uses starting group (from config) + repo name. Example: with group_path: "sikatech/eshop", sikatech/eshop/services/core-service → eshop-core-service
  • Last Segment Strategy: group/subgroup/repo → repo
  • Custom Strategy: Use custom mappings from configuration

2. Complete Migration Capabilities

• ✅ Code Migration: Full git history, branches, and tags
• ✅ Issues: All issues with labels, comments, and metadata
• ✅ Merge Requests: Converted to Pull Requests
• ✅ Labels: All labels with colors and descriptions
• ✅ Milestones: All milestones with due dates
• ✅ Wikis: Wiki pages migration
• ✅ Branches & Tags: All branches and tags preserved

3. Flexible Configuration

• Configurable migration options (what to migrate)
• Dry-run mode for testing
• Custom name mappings for specific repositories
• Support for both GitLab.com and self-hosted GitLab instances
• Support for GitHub.com and GitHub Enterprise

4. Import/Export Capabilities

• Export: Export all GitLab repositories to a JSON file with GitHub name mappings
• Import: Import repository mappings from a JSON file for validation
• Two-Phase Migration: Review and edit mappings before migration
• Manual Review: Edit JSON file to adjust names, skip repositories, or add notes

Quick Start

1. Configuration

Copy config.sample.json to config.json and configure:

{
  "gitlab": {
    "base_url": "https://gitlab.com",
    "api_token": "your-gitlab-api-token",
    "group_path": "your-group/subgroup"
  },
  "github": {
    "base_url": "https://api.github.com",
    "api_token": "your-github-api-token",
    "org": "sika365"
  },
  "migration": {
    "naming_strategy": "flatten",
    "name_mappings": {
      "group/subgroup/old-name": "new-name"
    },
    "migrate_code": true,
    "migrate_issues": true,
    "migrate_merge_requests": true,
    "migrate_wikis": true,
    "migrate_labels": true,
    "migrate_milestones": true,
    "migrate_branches": true,
    "migrate_tags": true,
    "preserve_history": true,
    "dry_run": false
  }
}

2. Run the Service

go run main.go -c config.json

3. API Endpoints

List Repositories

GET /api/v1/migration/repositories

Get Repository Details

GET /api/v1/migration/repositories/:path

Migrate Single Repository

POST /api/v1/migration/repositories/:path/migrate
Content-Type: application/json

{
  "dry_run": false
}

Migrate All Repositories

POST /api/v1/migration/migrate-all
Content-Type: application/json

{
  "dry_run": false
}

Get Migration Status

GET /api/v1/migration/status/:path

Export Repositories to JSON File

Exports all GitLab repositories to a JSON file with GitHub name mappings. This is useful for reviewing and editing mappings before migration.

POST /api/v1/migration/export
Content-Type: application/json

{
  "output_path": "repositories.json"
}

Response:

{
  "message": "Repositories exported successfully",
  "output_path": "repositories.json"
}

Import Repositories from JSON File

Imports repository mappings from a JSON file. This validates the file and returns repository count.

POST /api/v1/migration/import
Content-Type: application/json

{
  "input_path": "repositories.json"
}

Response:

{
  "message": "Repositories imported successfully",
  "repositories": 150,
  "gitlab_group": "sikatech/eshop",
  "github_org": "sika365"
}

Migrate from JSON File

Migrates repositories from a JSON mapping file. This allows you to review and edit the mappings before migration.

POST /api/v1/migration/migrate-from-file
Content-Type: application/json

{
  "input_path": "repositories.json",
  "dry_run": false
}

Two-Phase Migration Approach

The migrator supports a two-phase approach for safer and more controlled migrations:

Phase 1: Discovery and Export

1. List Repositories: Discover all repositories in the GitLab group
2. Export to JSON: Export all repositories with their GitHub name mappings to a JSON file
3. Review and Edit: Review the JSON file and manually adjust any name mappings if needed
4. Validate: Ensure all names conform to the GitHub Repository Naming Standard

Phase 2: Migration

1. Import from JSON: Load the reviewed JSON file
2. Migrate from File: Execute migration using the mappings from the JSON file
3. Track Status: Monitor migration progress and status

Example Workflow

# Step 1: Export repositories to JSON
curl -X POST http://localhost:8080/api/v1/migration/export \
  -H "Content-Type: application/json" \
  -d '{"output_path": "repositories.json"}'

# Step 2: Review and edit repositories.json if needed
# (Edit the file to adjust GitHub names, mark repositories to skip, etc.)

# Step 3: Import and validate the file
curl -X POST http://localhost:8080/api/v1/migration/import \
  -H "Content-Type: application/json" \
  -d '{"input_path": "repositories.json"}'

# Step 4: Migrate from the JSON file
curl -X POST http://localhost:8080/api/v1/migration/migrate-from-file \
  -H "Content-Type: application/json" \
  -d '{"input_path": "repositories.json", "dry_run": false}'

JSON File Structure

The exported JSON file has the following structure:

{
  "version": "1.0",
  "created_at": "2025-01-15T10:00:00Z",
  "updated_at": "2025-01-15T10:00:00Z",
  "gitlab_group": "sikatech/eshop",
  "github_org": "sika365",
  "repositories": [
    {
      "gitlab_id": 123,
      "gitlab_path": "core-service",
      "gitlab_full_path": "sikatech/eshop/services/core-service",
      "gitlab_name": "core-service",
      "gitlab_description": "Core service description",
      "gitlab_visibility": "private",
      "gitlab_default_branch": "main",
      "gitlab_ssh_url": "[email protected]:sikatech/eshop/services/core-service.git",
      "gitlab_http_url": "https://gitlab.com/sikatech/eshop/services/core-service.git",
      "is_nested": true,
      "parent_group_path": "sikatech/eshop/services",
      "github_name": "core-service",
      "github_description": "Core service description",
      "github_private": true,
      "migrated": false,
      "skip": false
    }
  ]
}

You can edit this file to:

• Adjust GitHub repository names to match the naming standard
• Mark repositories to skip ("skip": true)
• Add skip reasons ("skip_reason": "...")
• Review all mappings before migration

GitHub Repository Naming Standard

This migrator follows the GitHub Repository Naming Standard for Sika365 Microservices Architecture. For complete details, see NAMING_STANDARD.md.

Overview

The naming standard defines category-based patterns for different types of repositories:

• Core Services (/services/): {service-name}-service (e.g., core-service, auth-service)
• Finance Services (/payment-gateway-adapters/): {gateway-name}-finance-service (e.g., zarinpal-finance-service, sibank-finance-service)
• Connector Services (/external/): {provider-name}-connector-service (e.g., darian-connector-service, tipax-connector-service)
• CDC Services (/cdc/): cdc-{agent-name}-agent or cdc-{component-name} (e.g., cdc-sepidar-agent, cdc-debezium)
• Mobile Apps (/apps/shevi/): mobile-app-{version} or {name}-mobile-app-v3-template (e.g., mobile-app-v3, market-mobile-app-v3-template)
• Admin Panels (/admin-panels/): admin-panel-{version} (e.g., admin-panel-v3)
• SDKs: {sdk-name}-sdk-{language}-{version} or {prefix}-sdk (e.g., eshop-sdk-go-v1, cdc-sdk)

Naming Rules

1. Use hyphens, not underscores: core-service ✅, core_service ❌
2. Use lowercase only: auth-service ✅, Auth-Service ❌
3. Be descriptive but concise: zarinpal-finance-service ✅, zarinpal-payment-gateway-adapter-service ❌
4. Always use -service suffix for microservices: core-service ✅, core ❌

For complete naming patterns, examples, and migration mappings, refer to NAMING_STANDARD.md.

Naming Strategies (Technical Implementation)

The migrator provides multiple technical strategies to convert nested GitLab paths to flat GitHub repository names. These strategies are used to generate names that should then conform to the naming standard above.

Last Group Strategy (Default)

Uses the starting group name (from config group_path) and repository name:

• With group_path: "sikatech/eshop":
  • sikatech/eshop/services/core-service → eshop-core-service
  • sikatech/eshop/payment-gateway-adapters/zarinpal-adapter-service → eshop-zarinpal-adapter-service
• The starting group is the last part of the group_path in config (e.g., "eshop" from "sikatech/eshop")

Flatten Strategy

Converts all slashes to hyphens:

• sikatech/eshop/services/core-service → sikatech-eshop-services-core-service
• sikatech/eshop/payment-gateway-adapters/zarinpal-adapter-service → sikatech-eshop-payment-gateway-adapters-zarinpal-adapter-service

Prefix Strategy

Uses first and last segment:

• sikatech/eshop/services/core-service → sikatech-core-service
• sikatech/eshop/payment-gateway-adapters/zarinpal-adapter-service → sikatech-zarinpal-adapter-service

Last Segment Strategy

Uses only the repository name:

• sikatech/eshop/services/core-service → core-service
• sikatech/eshop/payment-gateway-adapters/zarinpal-adapter-service → zarinpal-adapter-service

Custom Strategy

Use custom mappings in configuration to ensure names conform to the naming standard:

{
  "migration": {
    "naming_strategy": "custom",
    "name_mappings": {
      "sikatech/eshop/services/core-service": "core-service",
      "sikatech/eshop/payment-gateway-adapters/zarinpal-adapter-service": "zarinpal-finance-service",
      "sikatech/eshop/external/darian-endpoint": "darian-connector-service"
    }
  }
}

Note: When using custom strategy, ensure all mappings follow the GitHub Repository Naming Standard.

Migration Process

The migrator supports two approaches:

Direct Migration

1. Discovery: Lists all repositories in the specified GitLab group (including nested groups)
2. Name Mapping: Converts GitLab paths to GitHub-compatible names using the selected strategy, ensuring names conform to the GitHub Repository Naming Standard
3. Validation: Validates repository names against the naming standard and suggests fixes if needed
4. Code Migration: Clones repository with full history and pushes to GitHub
5. Metadata Migration: Migrates issues, labels, milestones, merge requests, and wikis
6. Verification: Checks migration status and reports results

Two-Phase Migration (Recommended)

See the Two-Phase Migration Approach section above for detailed steps. This approach allows you to:

• Export all repositories to a JSON file
• Review and edit name mappings before migration
• Validate all mappings against the naming standard
• Migrate from the reviewed JSON file

Requirements

• Go 1.22.3 or higher
• Git installed and available in PATH
• GitLab API token with appropriate permissions
• GitHub API token with repository creation permissions

API Tokens

GitLab Token

Create a personal access token with:

• api scope
• read_repository scope
• read_user scope

GitHub Token

Create a personal access token or organization token with:

• repo scope (full control of private repositories)
• admin:org scope (if migrating to an organization)

Limitations

• Wiki migration requires manual steps (GitHub wikis are separate git repositories)
• Some GitLab-specific features may not have direct GitHub equivalents
• Large repositories may take significant time to migrate
• Rate limiting may affect migration speed for large numbers of repositories

Error Handling

The service provides detailed error reporting:

• Per-repository error tracking
• Warnings for name changes
• Migration status for each repository
• Detailed logs for troubleshooting

License

See LICENSE file for details.