3.1.5 breaks global install of dependents on Windows #720

cwalther · 2016-10-05T15:16:21Z

On Windows 10 (x86 and x64), installing a package that depends on sqlite3@3.1.5 (or something that resolves to 3.1.5, like ^3.1.4) globally (npm install -g) fails with error EBUSY: resource busy or locked, rmdir from npm install node-pre-gyp. The same thing works as expected with 3.1.4. It also works with 3.1.5 on OS X. Installing locally also works as expected.

I have not understood yet how exactly the error comes about, but it appears to be caused by 9a038b5.

Minimal reproducing example:

C:\Users\Walther\sqtest315>dir
 Volume in drive C has no label.
 Volume Serial Number is 7CE2-F96E

 Directory of C:\Users\Walther\sqtest315

05.10.2016  16:32    <DIR>          .
05.10.2016  16:32    <DIR>          ..
05.10.2016  16:28                85 package.json
               1 File(s)             85 bytes
               2 Dir(s)  26'892'546'048 bytes free

C:\Users\Walther\sqtest315>type package.json
{
"name": "sqtest315",
"version": "1.0.0",
"dependencies": {"sqlite3": "3.1.5"}
}
C:\Users\Walther\sqtest315>npm pack
sqtest315-1.0.0.tgz

C:\Users\Walther\sqtest315>npm install -g sqtest315-1.0.0.tgz

> sqlite3@3.1.5 preinstall C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773
> npm install node-pre-gyp

C:\Users\Walther\node
`-- node-pre-gyp@0.6.30
  +-- mkdirp@0.5.1
  | `-- minimist@0.0.8
  +-- nopt@3.0.6
  | `-- abbrev@1.0.9
  +-- npmlog@4.0.0
  | +-- are-we-there-yet@1.1.2
  | | `-- delegates@1.0.0
  | +-- console-control-strings@1.1.0
  | +-- gauge@2.6.0
  | | +-- aproba@1.0.4
  | | +-- has-color@0.1.7
  | | +-- has-unicode@2.0.1
  | | +-- object-assign@4.1.0
  | | +-- signal-exit@3.0.1
  | | +-- string-width@1.0.2
  | | | +-- code-point-at@1.0.1
  | | | | `-- number-is-nan@1.0.1
  | | | `-- is-fullwidth-code-point@1.0.0
  | | +-- strip-ansi@3.0.1
  | | | `-- ansi-regex@2.0.0
  | | `-- wide-align@1.1.0
  | `-- set-blocking@2.0.0
  +-- rc@1.1.6
  | +-- deep-extend@0.4.1
  | +-- ini@1.3.4
  | +-- minimist@1.2.0
  | `-- strip-json-comments@1.0.4
  +-- request@2.75.0
  | +-- aws-sign2@0.6.0
  | +-- aws4@1.4.1
  | +-- bl@1.1.2
  | | `-- readable-stream@2.0.6
  | +-- caseless@0.11.0
  | +-- combined-stream@1.0.5
  | | `-- delayed-stream@1.0.0
  | +-- extend@3.0.0
  | +-- forever-agent@0.6.1
  | +-- form-data@2.0.0
  | | `-- asynckit@0.4.0
  | +-- har-validator@2.0.6
  | | +-- chalk@1.1.3
  | | | +-- ansi-styles@2.2.1
  | | | +-- escape-string-regexp@1.0.5
  | | | +-- has-ansi@2.0.0
  | | | `-- supports-color@2.0.0
  | | +-- commander@2.9.0
  | | | `-- graceful-readlink@1.0.1
  | | +-- is-my-json-valid@2.15.0
  | | | +-- generate-function@2.0.0
  | | | +-- generate-object-property@1.2.0
  | | | | `-- is-property@1.0.2
  | | | +-- jsonpointer@4.0.0
  | | | `-- xtend@4.0.1
  | | `-- pinkie-promise@2.0.1
  | |   `-- pinkie@2.0.4
  | +-- hawk@3.1.3
  | | +-- boom@2.10.1
  | | +-- cryptiles@2.0.5
  | | +-- hoek@2.16.3
  | | `-- sntp@1.0.9
  | +-- http-signature@1.1.1
  | | +-- assert-plus@0.2.0
  | | +-- jsprim@1.3.1
  | | | +-- extsprintf@1.0.2
  | | | +-- json-schema@0.2.3
  | | | `-- verror@1.3.6
  | | `-- sshpk@1.10.1
  | |   +-- asn1@0.2.3
  | |   +-- assert-plus@1.0.0
  | |   +-- bcrypt-pbkdf@1.0.0
  | |   +-- dashdash@1.14.0
  | |   | `-- assert-plus@1.0.0
  | |   +-- ecc-jsbn@0.1.1
  | |   +-- getpass@0.1.6
  | |   | `-- assert-plus@1.0.0
  | |   +-- jodid25519@1.0.2
  | |   +-- jsbn@0.1.0
  | |   `-- tweetnacl@0.14.3
  | +-- is-typedarray@1.0.0
  | +-- isstream@0.1.2
  | +-- json-stringify-safe@5.0.1
  | +-- mime-types@2.1.12
  | | `-- mime-db@1.24.0
  | +-- node-uuid@1.4.7
  | +-- oauth-sign@0.8.2
  | +-- qs@6.2.1
  | +-- stringstream@0.0.5
  | +-- tough-cookie@2.3.1
  | `-- tunnel-agent@0.4.3
  +-- rimraf@2.5.4
  | `-- glob@7.1.0
  |   +-- fs.realpath@1.0.0
  |   +-- inflight@1.0.5
  |   +-- minimatch@3.0.3
  |   | `-- brace-expansion@1.1.6
  |   |   +-- balanced-match@0.4.2
  |   |   `-- concat-map@0.0.1
  |   +-- once@1.4.0
  |   `-- path-is-absolute@1.0.1
  +-- semver@5.3.0
  +-- tar@2.2.1
  | +-- block-stream@0.0.9
  | +-- fstream@1.0.10
  | | `-- graceful-fs@4.1.9
  | `-- inherits@2.0.3
  `-- tar-pack@3.1.4
    +-- debug@2.2.0
    | `-- ms@0.7.1
    +-- fstream-ignore@1.0.5
    +-- once@1.3.3
    | `-- wrappy@1.0.2
    +-- readable-stream@2.1.5
    | +-- buffer-shims@1.0.0
    | +-- core-util-is@1.0.2
    | +-- isarray@1.0.0
    | +-- process-nextick-args@1.0.7
    | +-- string_decoder@0.10.31
    | `-- util-deprecate@1.0.2
    `-- uid-number@0.0.6

npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "node-pre-gyp"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! path C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773
npm ERR! code EBUSY
npm ERR! errno -4082
npm ERR! syscall rmdir

npm ERR! EBUSY: resource busy or locked, rmdir 'C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\node\node_modules\.staging\sqlite3-a06df773\npm-debug.log
npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "sqtest315-1.0.0.tgz"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! code ELIFECYCLE

npm ERR! sqlite3@3.1.5 preinstall: `npm install node-pre-gyp`
npm ERR! Exit status 4294963214
npm ERR!
npm ERR! Failed at the sqlite3@3.1.5 preinstall script 'npm install node-pre-gyp'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sqlite3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm install node-pre-gyp
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sqlite3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sqlite3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\sqtest315\npm-debug.log

C:\Users\Walther\sqtest315>

npm-debug.log.zip

The text was updated successfully, but these errors were encountered:

sam-github · 2016-10-06T18:50:22Z

@springmeyer 9a038b5 is broken, you can't list node-pre-gyp as a dep, and install it in preinstall, I suspect what is happening is that sometimes npm is installing it at the same time as the preinstall script is trying.

The change was committed directly without a PR, so its hard to understand the justificatio.

I'm going to guess its because of the well known problem with npm 2.x that it does not guarantee a package's dependencies are installed before a package's install script runs, so its an attempt to work around this. The only robust fix is to bundle any deps that must be present at the time a package's install script runs. Which is what the code used to do... so why was it changed?

* Newer verions of mapbox/node-sqlite3 are breaking API Connect and LoopBack installations * Caused by 9a038b5 listing `node-pre-gyp` as a dep and installing it during `preinstall` * See issue at TryGhost/node-sqlite3#720 for * more details

* Newer versions of mapbox/node-sqlite3 are breaking API Connect and LoopBack installations * Caused by 9a038b5 listing `node-pre-gyp` as a dep and installing it during `preinstall` * See issue at TryGhost/node-sqlite3#720 for more details

springmeyer · 2016-10-06T22:02:28Z

you can't list node-pre-gyp as a dep, and install it in preinstall, I suspect what is happening is that sometimes npm is installing it at the same time as the preinstall script is trying.

@sam-github can you point me to npm documentation on this restriction? I've tried and been unable to replicate this problem.

The change was committed directly without a PR, so its hard to understand the justificatio.

Bundling is a headache because security issues in the bundled dependencies cannot be fixed without re-tagging node-pre-gyp, and every single module that depends on it upgrading. Across modules depending on node-pre-gyp this was resulting in a lot of unnecessary time lost and confusion from users that don't know why vulnerable modules are being included (since they are way downstream of the module that includes node-pre-gyp).

Secondarily the size of the package is then large and users were complaining. But this is going to be this way once installed of course, so this is less of a problem.

Some background at mapbox/node-pre-gyp#162 and mapbox/node-pre-gyp#163

springmeyer · 2016-10-06T22:21:05Z

@sam-github any idea what .sl-blip.js is. I see npm ERR! strong-arc@1.8.8 preinstall: node .sl-blip.js in many of the error messages at strongloop/strong-arc#1733.

In the meantime I'm working on a v3.1.6 release that bundles again to avoid the breakages.

rmg · 2016-10-06T22:51:17Z

The failing node .sl-blip.js is another example of annoying install script behaviour I think.

One thing you can try w.r.t. install scripts in the future like you did with node-pre-gyp is to do npm install --no-global node-pre-gyp. This will prevent it from being hoisted and installed globally when sqlite3 is being pulled in by a global module, but won't have any effect when not being pulled in via a global install. This is required because all npm CLI flags get turned into env vars, and the propagate to nested npm processes that way.

springmeyer · 2016-10-06T22:54:12Z

npm install --no-global node-pre-gyp

@rmg ah, that sounds really smart. Do you think that could potentially avoid this Windows-specific EBUSY error as well, or are you recommending as a good idea in general?

springmeyer · 2016-10-06T23:17:54Z

v3.1.6 is published. Can someone who can replicate confirm this is fixed and let me know here?

rmg · 2016-10-06T23:28:52Z

@springmeyer just as a general best practice. I'm not sure if it would help with the EBUSY or not, but it is possible it might.

I'll see if I can wrangle some people into testing 3.1.6.

superkhau · 2016-10-06T23:33:30Z

@springmeyer I just tested on node 5.11.1 and npm 3.8.6 and npm install -g apiconnect works again.

* Fixes installation issues with `npm install -g strongloop` and `npm install -g apiconnect * They now bundle `node-pre-gyp`, see https://github.com/mapbox/node-sqlite3/commits/de2055b * Initial bug discussion at TryGhost/node-sqlite3#720

* Fixes installation issues with `npm install -g strongloop` and `npm install -g apiconnect` * They now bundle `node-pre-gyp`, see https://github.com/mapbox/node-sqlite3/commits/de2055b * Initial bug discussion at TryGhost/node-sqlite3#720

cwalther · 2016-10-07T06:23:08Z

👍 Confirmed that things work again with 3.1.6 in both my minimal example (interesting that it didn’t reproduce the problem for you, btw – any differences in node/npm/os versions?) and my real case (a private package). Thanks!

I leave it to you whether to close this issue now or keep it open while looking for a better solution.

cwalther · 2016-10-07T07:50:04Z

I tried the npm install --no-global node-pre-gyp by hacking it into a copy of sqlite3-3.1.5.tgz. On the first attempt, my minimal example just failed with a different error (EPERM, see below), but subsequent attempts (deleting the installed package in between) were successful. I can’t make much sense of that, but I’ll just blame it on incomprehensible Windows behavior. So I’m unsure if that’s a viable solution.

Error output of first attempt

C:\Users\Walther\sqtest315cw>dir
 Volume in drive C has no label.
 Volume Serial Number is 7CE2-F96E

 Directory of C:\Users\Walther\sqtest315cw

07.10.2016  08:59    <DIR>          .
07.10.2016  08:59    <DIR>          ..
07.10.2016  09:00               132 package.json
               1 File(s)            132 bytes
               2 Dir(s)  26'830'405'632 bytes free

C:\Users\Walther\sqtest315cw>type package.json
{
"name": "sqtest315-cw",
"version": "1.0.0",
"dependencies": {"sqlite3": "C:\\Users\\Walther\\Desktop\\sqlite3-cw-3.1.5.tgz"}
}
C:\Users\Walther\sqtest315cw>npm pack
sqtest315-cw-1.0.0.tgz

C:\Users\Walther\sqtest315cw>npm install -g sqtest315-cw-1.0.0.tgz

> sqlite3@3.1.5 preinstall C:\Users\Walther\node\node_modules\.staging\sqlite3-87af97aa
> npm install --no-global node-pre-gyp

sqlite3@3.1.5 C:\Users\Walther\node\node_modules\.staging\sqlite3-87af97aa
`-- node-pre-gyp@0.6.30
  +-- mkdirp@0.5.1
  | `-- minimist@0.0.8
  +-- nopt@3.0.6
  | `-- abbrev@1.0.9
  +-- npmlog@4.0.0
  | +-- are-we-there-yet@1.1.2
  | | `-- delegates@1.0.0
  | +-- console-control-strings@1.1.0
  | +-- gauge@2.6.0
  | | +-- aproba@1.0.4
  | | +-- has-color@0.1.7
  | | +-- has-unicode@2.0.1
  | | +-- object-assign@4.1.0
  | | +-- signal-exit@3.0.1
  | | +-- string-width@1.0.2
  | | | +-- code-point-at@1.0.1
  | | | | `-- number-is-nan@1.0.1
  | | | `-- is-fullwidth-code-point@1.0.0
  | | +-- strip-ansi@3.0.1
  | | | `-- ansi-regex@2.0.0
  | | `-- wide-align@1.1.0
  | `-- set-blocking@2.0.0
  +-- rc@1.1.6
  | +-- deep-extend@0.4.1
  | +-- ini@1.3.4
  | +-- minimist@1.2.0
  | `-- strip-json-comments@1.0.4
  +-- request@2.75.0
  | +-- aws-sign2@0.6.0
  | +-- aws4@1.4.1
  | +-- bl@1.1.2
  | | `-- readable-stream@2.0.6
  | +-- caseless@0.11.0
  | +-- combined-stream@1.0.5
  | | `-- delayed-stream@1.0.0
  | +-- extend@3.0.0
  | +-- forever-agent@0.6.1
  | +-- form-data@2.0.0
  | | `-- asynckit@0.4.0
  | +-- har-validator@2.0.6
  | | +-- chalk@1.1.3
  | | | +-- ansi-styles@2.2.1
  | | | +-- escape-string-regexp@1.0.5
  | | | +-- has-ansi@2.0.0
  | | | `-- supports-color@2.0.0
  | | +-- commander@2.9.0
  | | | `-- graceful-readlink@1.0.1
  | | +-- is-my-json-valid@2.15.0
  | | | +-- generate-function@2.0.0
  | | | +-- generate-object-property@1.2.0
  | | | | `-- is-property@1.0.2
  | | | +-- jsonpointer@4.0.0
  | | | `-- xtend@4.0.1
  | | `-- pinkie-promise@2.0.1
  | |   `-- pinkie@2.0.4
  | +-- hawk@3.1.3
  | | +-- boom@2.10.1
  | | +-- cryptiles@2.0.5
  | | +-- hoek@2.16.3
  | | `-- sntp@1.0.9
  | +-- http-signature@1.1.1
  | | +-- assert-plus@0.2.0
  | | +-- jsprim@1.3.1
  | | | +-- extsprintf@1.0.2
  | | | +-- json-schema@0.2.3
  | | | `-- verror@1.3.6
  | | `-- sshpk@1.10.1
  | |   +-- asn1@0.2.3
  | |   +-- assert-plus@1.0.0
  | |   +-- bcrypt-pbkdf@1.0.0
  | |   +-- dashdash@1.14.0
  | |   | `-- assert-plus@1.0.0
  | |   +-- ecc-jsbn@0.1.1
  | |   +-- getpass@0.1.6
  | |   | `-- assert-plus@1.0.0
  | |   +-- jodid25519@1.0.2
  | |   +-- jsbn@0.1.0
  | |   `-- tweetnacl@0.14.3
  | +-- is-typedarray@1.0.0
  | +-- isstream@0.1.2
  | +-- json-stringify-safe@5.0.1
  | +-- mime-types@2.1.12
  | | `-- mime-db@1.24.0
  | +-- node-uuid@1.4.7
  | +-- oauth-sign@0.8.2
  | +-- qs@6.2.1
  | +-- stringstream@0.0.5
  | +-- tough-cookie@2.3.1
  | `-- tunnel-agent@0.4.3
  +-- rimraf@2.5.4
  | `-- glob@7.1.0
  |   +-- fs.realpath@1.0.0
  |   +-- inflight@1.0.5
  |   +-- minimatch@3.0.3
  |   | `-- brace-expansion@1.1.6
  |   |   +-- balanced-match@0.4.2
  |   |   `-- concat-map@0.0.1
  |   +-- once@1.4.0
  |   `-- path-is-absolute@1.0.1
  +-- semver@5.3.0
  +-- tar@2.2.1
  | +-- block-stream@0.0.9
  | +-- fstream@1.0.10
  | | `-- graceful-fs@4.1.9
  | `-- inherits@2.0.3
  `-- tar-pack@3.1.4
    +-- debug@2.2.0
    | `-- ms@0.7.1
    +-- fstream-ignore@1.0.5
    +-- once@1.3.3
    | `-- wrappy@1.0.2
    +-- readable-stream@2.1.5
    | +-- buffer-shims@1.0.0
    | +-- core-util-is@1.0.2
    | +-- isarray@1.0.0
    | +-- process-nextick-args@1.0.7
    | +-- string_decoder@0.10.31
    | `-- util-deprecate@1.0.2
    `-- uid-number@0.0.6

npm ERR! Windows_NT 10.0.10240
npm ERR! argv "C:\\Users\\Walther\\node\\node.exe" "C:\\Users\\Walther\\node\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "sqtest315-cw-1.0.0.tgz"
npm ERR! node v4.4.7
npm ERR! npm  v3.10.8
npm ERR! path C:\Users\Walther\node\node_modules\sqtest315-cw
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! syscall rename

npm ERR! Error: EPERM: operation not permitted, rename 'C:\Users\Walther\node\node_modules\sqtest315-cw' -> 'C:\Users\Walther\node\node_modules\.sqtest315-cw.DELETE'
npm ERR!     at moveAway (C:\Users\Walther\node\node_modules\npm\lib\install\action\finalize.js:38:5)
npm ERR!     at destStatted (C:\Users\Walther\node\node_modules\npm\lib\install\action\finalize.js:27:7)
npm ERR!     at FSReqWrap.oncomplete (fs.js:82:15)
npm ERR!
npm ERR! Error: EPERM: operation not permitted, rename 'C:\Users\Walther\node\node_modules\sqtest315-cw' -> 'C:\Users\Walther\node\node_modules\.sqtest315-cw.DELETE'
npm ERR!     at Error (native)
npm ERR!  [Error: EPERM: operation not permitted, rename 'C:\Users\Walther\node\node_modules\sqtest315-cw' -> 'C:\Users\Walther\node\node_modules\.sqtest315-cw.DELETE']
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Walther\sqtest315cw\npm-debug.log

C:\Users\Walther\sqtest315cw>

npm-debug-1.log.zip

This was referenced Oct 6, 2016

Heroku build fails strongloop/strong-arc#1732

Closed

Failed at the strong-arc@1.8.8 preinstall script strongloop/strong-arc#1733

Closed

Global installation is broken due to sqlite3 strongloop/strongloop#324

Closed

davidcheung mentioned this issue Oct 6, 2016

How to install in windows 10 i got error . my npm -v 3.10.8 & node -v 6.6.0 . I am install in three win system but got same error . strongloop/loopback#2831

Closed

sam-github mentioned this issue Oct 6, 2016

Lock sqlite version to fix global installation strongloop/strongloop#325

Closed

superkhau mentioned this issue Oct 6, 2016

Lock sqlite version to fix global installation strongloop/strongloop#326

Merged

springmeyer pushed a commit that referenced this issue Oct 6, 2016

rebundle node-pre-gyp - refs #720

de2055b

rmg mentioned this issue Oct 6, 2016

Unable to install strongloop npm/npm#14188

Closed

superkhau mentioned this issue Oct 6, 2016

Upgrade to sqlite3@^3.1.6 strongloop/strongloop#327

Merged

springmeyer closed this as completed Oct 21, 2016

springmeyer mentioned this issue Dec 7, 2016

Why is node-pre-gyp a bundled dependency? fsevents/fsevents#157

Closed

springmeyer mentioned this issue Dec 16, 2016

Bundling/preinstall woes mapbox/node-pre-gyp#260

Closed

sharkinsspatial mentioned this issue Apr 6, 2017

Preinstall with node-pre-gyp causes install failure on Windows with global flag. naturalatlas/node-gdal#195

Closed

danpat mentioned this issue Aug 15, 2017

Publicly expose binaries to Node.js bindings, resolves #4377 Project-OSRM/osrm-backend#4386

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.1.5 breaks global install of dependents on Windows #720

3.1.5 breaks global install of dependents on Windows #720

cwalther commented Oct 5, 2016

sam-github commented Oct 6, 2016

springmeyer commented Oct 6, 2016 •

edited

Loading

springmeyer commented Oct 6, 2016 •

edited

Loading

rmg commented Oct 6, 2016

springmeyer commented Oct 6, 2016

springmeyer commented Oct 6, 2016

rmg commented Oct 6, 2016

superkhau commented Oct 6, 2016

cwalther commented Oct 7, 2016

cwalther commented Oct 7, 2016

3.1.5 breaks global install of dependents on Windows #720

3.1.5 breaks global install of dependents on Windows #720

Comments

cwalther commented Oct 5, 2016

sam-github commented Oct 6, 2016

springmeyer commented Oct 6, 2016 • edited Loading

springmeyer commented Oct 6, 2016 • edited Loading

rmg commented Oct 6, 2016

springmeyer commented Oct 6, 2016

springmeyer commented Oct 6, 2016

rmg commented Oct 6, 2016

superkhau commented Oct 6, 2016

cwalther commented Oct 7, 2016

cwalther commented Oct 7, 2016

springmeyer commented Oct 6, 2016 •

edited

Loading

springmeyer commented Oct 6, 2016 •

edited

Loading