Fix prune, add tests to demonstrate brokenness in npm 4.1.0+ #16637

exogen · 2017-05-16T09:41:02Z

This adds two test cases to demonstrate how npm 4.1.0+ (including the latest version) fails to prune in certain cases (likely introduced in #15090). A fix is included, but will need to be reviewed carefully (I'm not sure whether cycle detection or checking userRequired is necessary, or if that's already taken care of by isExtraneous).

prune misbehaving is likely the cause of some of the funny business on display in issues #15727, #15669, #15646.

My interest in this stems from the fact that broken pruning causes postinstall-build to break on the latest versions of npm ever since 4.1.0.

The Tests

Call npm prune --production on a module with ONLY these dependencies:
```
{
  "devDependencies": {
    "test-package-with-one-dep": "0.0.0",
    "test-package": "0.0.0"
  }
}
```
...which looks like this before pruning:
```
├── [email protected]
└─┬ [email protected]
  └── [email protected]  deduped
```
npm prune --production SHOULD remove both packages, resulting in an empty node_modules tree, but it fails to remove test-package (because it's a prod dep of test-package-with-one-dep, but that shouldn't matter because test-package-with-one-dep itself is a dev dep).
Call prune --production with the same package in both dependencies and devDependencies. The package SHOULD NOT be removed.

The Fix

In --production mode, packages are now pruned if they are only reachable from dev dependencies (the actual logic is: NOT reachable from prod or opt dependencies).

/cc @iarna

exogen · 2017-05-16T10:15:38Z

    var isChildDev = function (parent) { return isDev(parent, childName) }
    return child.requiredBy.every(isChildDev)

This appears to be the logic at fault. If I understand correctly, it only prunes if child is a dev dependency in every package it's required by, but that's not right: it should prune if it's a dev dependency of the top-level package and/or only required by other dev dependencies. It's perfectly fine to prune if it's a prod dependency of a dev dependency.

…endency (npm#15669)

exogen · 2017-05-16T11:39:01Z

I came up with a fix and updated the PR description. There are now two test cases; CI shows that they both demonstrate the failure until this fix is applied. 😄

exogen · 2017-05-17T21:07:20Z

Some testing shows that this does indeed need to take cycles into account.

IMO it would be nice if this behavior was baked into isExtraneous in the first place (by adding a production flag). In production mode, devDeps should be considered extraneous. Then prune's job is to simply remove anything marked extraneous...no additional logic.

iarna

With and without your changes I'm seeing this test failure:

EDIT: never mind this, I misread my own edited output =D

iarna · 2017-06-30T07:58:13Z

test/tap/prune-dev-dep-cycle.js

+    t.is(code, 0, 'prune finished successfully')
+    t.equal(stderr,
+      '- [email protected] node_modules/a\n' +
+      '- [email protected] node_modules/b\n')


I wouldn't expect this to ever pass because that output is printed to stdout not stderr.

iarna · 2017-06-30T08:04:25Z

test/tap/prune-with-dev-dep-duplicate.js

+    '--loglevel', 'silent',
+    '--production', 'false'
+  ], EXEC_OPTS, function (err, code, stdout, stderr) {
+    t.ifErr(err, 'install finished successfully')


I know you likely just copied this from another test, but it was wrong there too. =D

The test label says 'install finished successfully' but that's not what not getting an err value means. err will ONLY be set if something catastrophic happened like we couldn't open a new process. This would be better written as:

if (err) throw err

(The story of how any tests started doing the ifError thing is an interesting bit of code archaeology, but the long and short of it is that this is never an appropriate check in this context. It was introduced due to either errors when converting between two types of test, or by someone who cargo culted it from another similar but quite different context.)

iarna · 2017-06-30T08:07:50Z

test/tap/prune-with-dev-dep-duplicate.js

+  ], EXEC_OPTS, function (err, code, stderr) {
+    t.ifErr(err, 'prune finished successfully')
+    t.notOk(code, 'exit ok')
+    t.notOk(stderr, 'Should not get data on stderr: ' + stderr)


This is not stderr that is being checked here, this is stdout.

I'm honestly not sure what the expected behavior for this one is? In my testing with the bleeding edge, it removes this:

# { # "action": "remove", # "name": "test-package", # "version": "0.0.0", # "path": "node_modules/test-package" # }

Is that correct, or wrong?

If I understand correctly, that is wrong: test-package should remain, because the result of npm prune --production should be "anything NOT required by production dependencies is removed", but test-package is in dependencies, so removing it would break the install.

iarna · 2017-06-30T08:09:01Z

test/tap/prune-with-only-dev-deps.js

+  ], EXEC_OPTS, function (err, code, stderr) {
+    t.ifErr(err, 'prune finished successfully')
+    t.notOk(code, 'exit ok')
+    t.equal(stderr,


This variable holds STDOUT, not STDERR.

iarna · 2017-06-30T08:11:03Z

test/tap/prune-with-only-dev-deps.js

+    '--loglevel', 'silent',
+    '--production', 'false'
+  ], EXEC_OPTS, function (err, code, stdout, stderr) {
+    t.ifErr(err, 'prune finished successfully')


Same comments about ifErr apply

iarna · 2017-06-30T08:43:15Z

Ok, so don't worry about making any changes to this, I wanted to give you the feedback, but I've gone and made them myself when I was making this merge clean w/ the branch that will be release-next tomorrow. (or released tomorrow, not sure =D)

This is what'll be merged: lock-missing-dep-fixes...iarna/pr/16637

exogen · 2017-06-30T18:02:11Z

@iarna Thanks for the feedback! I was indeed confused about the stderr/stdout discrepancies myself when making these changes – the code is copied from existing tests that had the same mistake. And yes, I left this with new test cases that were failing, never got around to solving it completely.

iarna · 2017-07-01T02:51:32Z

This was merged into release-next as 8a2d739 and 0712ac6.

@exogen

PR-URL: #16637 Credit: @exogen Reviewed-By: @iarna

@exogen

Fixes: #15669 PR-URL: #16637 Credit: @exogen Reviewed-By: @iarna

test: Add prune --production test with only devDependencies

5c089af

isaacs added the review label May 16, 2017

exogen changed the title ~~test: Add prune --production test with only devDependencies~~ Add prune test to demonstrate brokenness in npm 4.1.0+ May 16, 2017

exogen added 2 commits May 16, 2017 04:23

test: Add test for prune --production with duplicate prod and dev dep…

334d73d

…endency (npm#15669)

prune: Fix unpruned and incorrectly pruned dependencies

c29b62c

exogen changed the title ~~Add prune test to demonstrate brokenness in npm 4.1.0+~~ Fix prune, add tests to demonstrate brokenness in npm 4.1.0+ May 16, 2017

test: Add prune cycle test (copied from existing devDep cycle test)

c3afdfb

iarna suggested changes Jun 30, 2017

View reviewed changes

iarna approved these changes Jun 30, 2017

View reviewed changes

iarna self-assigned this Jun 30, 2017

iarna closed this Jul 1, 2017

zkat pushed a commit that referenced this pull request Jul 5, 2017

test: Add various tests for prune --production

656544c

PR-URL: #16637 Credit: @exogen Reviewed-By: @iarna

zkat pushed a commit that referenced this pull request Jul 5, 2017

prune: Fix unpruned and incorrectly pruned dependencies

d21ab57

Fixes: #15669 PR-URL: #16637 Credit: @exogen Reviewed-By: @iarna

zkat mentioned this pull request Jul 6, 2017

npm prune is messing the npm install and package-lock.json resulting in missing modules #17620

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix prune, add tests to demonstrate brokenness in npm 4.1.0+ #16637

Fix prune, add tests to demonstrate brokenness in npm 4.1.0+ #16637

Uh oh!

exogen commented May 16, 2017 •

edited

Loading

Uh oh!

exogen commented May 16, 2017 •

edited

Loading

Uh oh!

exogen commented May 16, 2017

Uh oh!

exogen commented May 17, 2017

Uh oh!

iarna left a comment •

edited

Loading

Uh oh!

iarna Jun 30, 2017

Uh oh!

iarna Jun 30, 2017

Uh oh!

iarna Jun 30, 2017

Uh oh!

exogen Jun 30, 2017 •

edited

Loading

Uh oh!

iarna Jun 30, 2017

Uh oh!

iarna Jun 30, 2017

Uh oh!

iarna commented Jun 30, 2017

Uh oh!

exogen commented Jun 30, 2017

Uh oh!

iarna commented Jul 1, 2017

Uh oh!

Uh oh!

Fix prune, add tests to demonstrate brokenness in npm 4.1.0+ #16637

Fix prune, add tests to demonstrate brokenness in npm 4.1.0+ #16637

Uh oh!

Conversation

exogen commented May 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The Tests

The Fix

Uh oh!

exogen commented May 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

exogen commented May 16, 2017

Uh oh!

exogen commented May 17, 2017

Uh oh!

iarna left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iarna Jun 30, 2017

Choose a reason for hiding this comment

Uh oh!

iarna Jun 30, 2017

Choose a reason for hiding this comment

Uh oh!

iarna Jun 30, 2017

Choose a reason for hiding this comment

Uh oh!

exogen Jun 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iarna Jun 30, 2017

Choose a reason for hiding this comment

Uh oh!

iarna Jun 30, 2017

Choose a reason for hiding this comment

Uh oh!

iarna commented Jun 30, 2017

Uh oh!

exogen commented Jun 30, 2017

Uh oh!

iarna commented Jul 1, 2017

Uh oh!

Uh oh!

exogen commented May 16, 2017 •

edited

Loading

exogen commented May 16, 2017 •

edited

Loading

iarna left a comment •

edited

Loading

exogen Jun 30, 2017 •

edited

Loading