npm-package-tester

🧪 Automatically test npm packages by discovering their CLI commands and running them in isolated Docker environments

🤖 AI-Powered: Generate realistic test scenarios using Claude, GPT-4, Gemini, or Groq 🔐 Private Packages: Full support for private npm packages and custom registries ⚡ Fast & Reliable: Run tests in parallel across multiple Node versions

The Problem

When you publish an npm package with CLI commands, you need to verify:

✅ Does it install correctly?
✅ Do all CLI commands work?
✅ Does it work across different Node versions?
✅ Does the --help flag work?
✅ Does the --version flag work?
✅ Does it run without crashing?

Manual testing is tedious and error-prone.

The Solution

npm-package-tester automatically:

📦 Analyzes your package.json to detect CLI commands
🐳 Creates fresh Docker containers with specified Node versions
📥 Installs your package
🧪 Runs each CLI command with common flags (--help, --version, etc.)
✅ Reports which commands work and which don't

Features

🔍 Auto-Discovery: Automatically detects all CLI commands from package.json
🐳 Docker Isolation: Tests in clean environments
🔄 Multi-Version: Test across Node 16, 18, 20, etc.
⚡ Parallel Testing: Run tests concurrently for speed
📊 Detailed Reports: Clear pass/fail results with complete test breakdown
🎯 Smart Testing: Automatically tries --help, --version, and no-args
🤖 AI-Powered Scenarios: Generate realistic test scenarios using Claude, GPT-4, Gemini, or Groq
🔐 Private Packages: Full support for private npm packages with authentication
📦 Custom Registries: Works with private npm registries
💨 Lightweight: Minimal dependencies, fast execution

Quick Start

# Install globally
npm install -g npm-package-tester

# Test a package from npm
npt test eslint

# Test your local package
npt test ./my-package

# Test across multiple Node versions
npt test prettier --node 16,18,20

# AI-powered intelligent testing (generates realistic scenarios)
npt test env-type-generator --ai-provider anthropic --ai-token YOUR_API_KEY

# Use OpenAI instead
npt test my-package --ai-provider openai --ai-token YOUR_OPENAI_KEY

# Test private packages
npt test @your-org/private-package --npm-token YOUR_NPM_TOKEN

# Test from custom registry
npt test @company/package --npm-registry https://npm.company.com --npm-token YOUR_TOKEN

# Keep containers for debugging
npt test --keep-containers

Example Output

Default Testing

📦 Package: eslint
   Version: 8.50.0

🐳 Node 20
  ✓ eslint --help (125ms)
  ✓ eslint --version (98ms)
  ✓ eslint [no args] (102ms)

📊 Summary
──────────────────────────────────────────────────
  Total: 3 tests
  Passed: 3
  Failed: 0
  Duration: 325ms

📋 Test Details
──────────────────────────────────────────────────

  🎯 Default Tests
    ✓ eslint --help (125ms)
    ✓ eslint --version (98ms)
    ✓ eslint (no args) (102ms)

✅ All tests passed!

AI-Powered Testing

📦 Package: env-type-generator
   Version: 1.0.0
   Auto-generate TypeScript types from .env files with zero config

🐳 Node 20
  ✓ env-type-gen (1512ms)
  ✓ env-type-gen (4871ms)
  ✓ env-type-gen (1307ms)

📊 Summary
──────────────────────────────────────────────────
  Total: 3 tests
  Passed: 3
  Failed: 0
  Duration: 7690ms

📋 Test Details
──────────────────────────────────────────────────

  🤖 AI-Generated Tests
    ✓ basic-type-generation (1512ms)
    ✓ multiple-env-files-with-parsing -e .env .env.local -p (4871ms)
    ✓ strict-mode-all-required -e .env.example -o ./types/env.d.ts (1307ms)

✅ All tests passed!

How It Works

1. Command Detection

// Reads package.json
{
  "name": "my-cli-tool",
  "bin": {
    "mycli": "./dist/cli.js",      // Primary command
    "my": "./dist/cli.js"           // Alias
  }
}

// Detects: mycli, my

2. Docker Environment

# Spins up for each Node version
FROM node:20-alpine
RUN npm install -g my-cli-tool
RUN mycli --help
RUN mycli --version

3. Smart Testing

For each command, automatically tests:

command --help (should show help)
command --version (should show version)
command (no args - should not crash)
Custom scenarios you define

Use Cases

1. Pre-Publish Validation

# Before publishing
npm run build
npt test .
npm publish

2. CI/CD Integration

# .github/workflows/test.yml
- name: Test CLI
  run: |
    npm install -g npm-package-tester
    npt test . --node 16,18,20

3. Testing Published Packages

# Verify your package works after publishing
npt test your-package-name@latest

4. Debugging Installation Issues

# Keep containers to inspect
npt test . --keep-containers
docker exec -it <container-id> sh

Configuration

Create .npmtestrc.json:

{
  "nodeVersions": ["16", "18", "20"],
  "parallel": true,
  "timeout": 30000,
  "customTests": [
    {
      "name": "Generate help",
      "command": "mycli generate --help",
      "expect": {
        "exitCode": 0,
        "stdout": "Usage:"
      }
    }
  ]
}

API Usage

import { testPackage } from 'npm-package-tester';

const results = await testPackage('eslint', {
  nodeVersions: ['20'],
  parallel: false,
});

console.log(`Passed: ${results.passed}/${results.total}`);

CLI Options

Usage: npt test [options] <package>

Options:
  -n, --node <versions>        Node versions (comma-separated)
  -p, --parallel               Run tests in parallel
  -k, --keep-containers        Keep containers after test
  -t, --timeout <ms>           Timeout per test (default: 30000)
  --npm-token <token>          npm authentication token for private packages
  --npm-registry <url>         Custom npm registry URL
  --ai-provider <provider>     AI provider (anthropic, openai, google, groq)
  --ai-token <token>           AI API token/key
  --ai-model <model>           AI model name (optional, auto-detects best)
  --no-help                    Skip --help tests
  --no-version                 Skip --version tests
  -v, --verbose                Verbose output

AI-Powered Testing

Instead of just testing --help and --version, use AI to generate realistic test scenarios:

npt test env-type-generator --ai-provider anthropic --ai-token YOUR_TOKEN

The AI will:

📖 Analyze the package README and CLI help
🧠 Understand what the package does
🎯 Generate realistic test scenarios with:
- Appropriate input files
- Realistic command arguments
- Expected output validation
✅ Validate file creation, content, and exit codes

Example AI-Generated Scenarios:

🤖 Generating test scenarios with AI...
✨ Generated 4 AI test scenarios

🐳 Node 20
  ✓ basic-env-file-generation (1400ms)
  ✓ env-with-comments (1706ms)
  ✓ nested-env-variables (5040ms)
  ✓ strict-mode-with-custom-output (1517ms)

📊 4/4 tests passed ✅

Supported AI Providers

Provider	Model	Setup
Anthropic	Claude Sonnet 4.5	Get key from console.anthropic.com
OpenAI	GPT-4o	Get key from platform.openai.com
Google	Gemini 2.0 Flash	Get key from ai.google.dev
Groq	Llama 3.3 70B	Get key from console.groq.com

The best model for each provider is automatically selected.

Testing Private Packages

npm-package-tester fully supports private npm packages with authentication:

Using npm Token

# Test private package from npm registry
npt test @your-org/private-package --npm-token YOUR_NPM_TOKEN

# Get your npm token from ~/.npmrc or create one with:
# npm token create --read-only

Using Custom Registry

# Test from private registry (like Verdaccio, Artifactory, GitHub Packages)
npt test @company/package \
  --npm-registry https://npm.company.com \
  --npm-token YOUR_REGISTRY_TOKEN

How it Works

The tool creates a .npmrc file in the Docker container
Configures authentication with your token
Installs the package securely
Runs all tests normally

Security Note: Tokens are only used within isolated Docker containers and are not stored or logged.

Supported Registries

✅ npm (private packages)
✅ GitHub Packages
✅ GitLab Package Registry
✅ Verdaccio
✅ JFrog Artifactory
✅ Azure Artifacts
✅ Any npm-compatible registry

Architecture

npm-package-tester
├── PackageAnalyzer      # Reads package.json, detects commands
├── CommandDetector      # Identifies CLI binaries
├── DockerManager        # Creates/manages containers
├── TestRunner           # Executes commands in containers
├── ScenarioRunner       # Runs AI-generated test scenarios
├── ScenarioGenerator    # Generates scenarios using AI
├── AIProvider           # Multi-provider AI abstraction (Claude, GPT-4, Gemini, Groq)
├── ResultFormatter      # Pretty output
└── CLI                  # User interface

Clean Architecture

Domain: Types, interfaces (no dependencies)
Application: Business logic (PackageAnalyzer, TestRunner, ScenarioRunner)
AI: AI providers and scenario generation
Infrastructure: Docker, file system (external concerns)
CLI: User interface (commander, chalk)

Requirements

Docker installed and running
Node.js >= 16
npm or yarn
(Optional) AI provider API key for intelligent test generation

Limitations

Only tests packages with CLI commands
Requires Docker (no Windows native support yet)
Can't test interactive prompts automatically
Network-dependent (pulls Docker images)
AI features require API credits

Roadmap

Contributing

We welcome contributions! Here's how you can help:

Ways to Contribute

🐛 Report Bugs: Create a bug report
💡 Suggest Features: Request a feature
📝 Improve Documentation: Fix typos, add examples, clarify instructions
🧪 Add Tests: Increase test coverage
🔧 Fix Issues: Check out good first issues

Development Setup

# Clone the repository
git clone https://github.com/ashishyd/npm-package-tester.git
cd npm-package-tester

# Install dependencies
npm install

# Build the project
npm run build

# Run tests
npm test

# Run linting
npm run lint

Guidelines

Follow the existing code style
Write tests for new features
Update documentation as needed
Ensure all tests pass before submitting PR
Keep PRs focused on a single feature or bug fix

Code of Conduct

Be respectful, inclusive, and constructive. We're all here to learn and build great tools together!

License

Related Projects

testcontainers - Integration testing with Docker
verdaccio - Private npm registry
np - Better npm publish

Support

Made with ❤️ for npm package authors

⭐ Star us on GitHub if you find this useful!

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
src		src
tests		tests
.eslintignore		.eslintignore
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.prettierrc.json		.prettierrc.json
README.md		README.md
jest.config.js		jest.config.js
package.json		package.json
tsconfig.eslint.json		tsconfig.eslint.json
tsconfig.json		tsconfig.json

ashishyd/npm-package-tester

Folders and files

Latest commit

History

Repository files navigation

npm-package-tester

Table of Contents

The Problem

The Solution

Features

Quick Start

Example Output

Default Testing

AI-Powered Testing

How It Works

1. Command Detection

2. Docker Environment

3. Smart Testing

Use Cases

1. Pre-Publish Validation

2. CI/CD Integration

3. Testing Published Packages

4. Debugging Installation Issues

Configuration

API Usage

CLI Options

AI-Powered Testing

Supported AI Providers

Testing Private Packages

Using npm Token

Using Custom Registry

How it Works

Supported Registries

Architecture

Clean Architecture

Requirements

Limitations

Roadmap

Contributing

Ways to Contribute

Development Setup

Guidelines

Code of Conduct

License

Related Projects

Support

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Packages