copyDependencyDir's cp '-RfL' needs to change 'f' to 'u' to support sloppy second copies #3360

paulpv · 2018-02-08T21:38:59Z

Let me start off by saying that today is Thursday and I just started using NativeScript on Monday, so I am most definitely a {N}ewb to {N} and this issue and my below comments might be making a lot of wrong assumptions.
Keep that in mind, and please kindly educate me where I am wrong.
(FWIW, I am a >25 year programmer competent in JavaScript, Java, C#, C/C++, Python... so I'm not a total newb)

I feel compelled, especially as a {N}ewb, to make this one soap-box statement: I have zero idea how this issue hasn't affected a non-trivial percentage of nativescript plugin developers out there. If you develop a nativescript plugin and keep it under [git] source control [who doesn't?] and add the repo'd plugin as a local dependency to an app, I don't see any possible way that the build could work. This issue [below] has been around for over 4 years, so I've got ta ask: How come this has not been fixed?

The issue is "simple", but I am having a very hard time articulating it simply or describing the "right" solution (which isn't obvious).

Here is the executive summary

What is happening is that during the build process the dependencies are actually being copied twice to platforms/android/app/src/main/assets/app/tns_modules.
The first copy of a dependency under source control is successful.
Any copied plugin that is under source control also has it's .git folder copied to platforms/android/app/src/main/assets/app/tns_modules/some-plugin/.git.
.git folders are well known to contain working/temp read-only files (ex: .git/objects/pack/pack-a98ffe92fb5a19a105f39a02942047edfc956aad.idx).
Those read-only .git files are now copied to tns_modules.
The second copy fails for the now obvious quite simple reason that cp cannot overwrite a read-only file.
The shelljs command attempts to use the -RfL flags, but the >4 year old shelljs/shelljs#98 issue says that shelljs cannot handle this.
So, the second copy fails, and the shelljs' error handling swallows the real error and generically report's the error as common.error("cannot create directory '" + dest + "': No such file or directory");.
https://github.com/shelljs/shelljs/blob/master/src/cp.js#L261

More Detail

This is directly related to #3028, and specifically the excellent detail in comment #3028 (comment).
That issue was reported 2017/08/02 and has had no resolution.
Issue #3028 refers to shelljs/shelljs#98.
That shelljs issue was reported 2013/12/16...over 4 years ago and has had no resolution!

Issue Symptom:
tns build android (or ios) fails with misleading error message:

Unable to apply changes on device: {deviceId}. Error is: Processing node_modules failed. Error: cp: cannot create directory '/Users/{user/project}/platforms/android/app/src/main/assets/app/tns_modules': No such file or directory.

NOTE: Sometimes I have also seen the following error message instead:

Cycle link found.
{~ one minute later}
Cycle link found.
{~ one minute later}
Cycle link found.
{~ one minute later}
Cycle link found.
...

I'm ignoring this second error for now [my theory is that fixing the first should prevent the second from ever happening].

The actual first error is...

copyFileSync: could not write to dest file (code=EACCES):/Users/{user/project}/platforms/android/app/src/main/assets/app/tns_modules/{some-plugin}/.git/objects/pack/pack-a98ffe92fb5a19a105f39a02942047edfc956aad.idx

https://github.com/shelljs/shelljs/blob/master/src/cp.js#L66
...caused by lib/tools/node-modules-dest-copy.js::copyDependencyDir(...)'s call to shelljs.cp("-RfL", dependency.directory, destinationPath);
https://github.com/NativeScript/nativescript-cli/blob/master/lib/tools/node-modules/node-modules-dest-copy.ts#L45

Repro

Repro # 1:

git clone https://github.com/EddyVerbruggen/nativescript-bluetooth.git
git clone https://github.com/EddyVerbruggen/nativescript-bluetooth-demo.git
cd nativescript-bluetooth-demo/Bluetooth
tns plugin remove nativescript-bluetooth
tns plugin add ../../nativescript-bluetooth
tns build android

Expected: Success

Result (includes some of my added logging; the outdated template is not relevant):

Copying template files...
Installing  tns-android
...
added 1 package in 6.199s
Error: sed: no such file or directory: /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/app/App_Resources/Android/app.gradle.
Check if you're using an outdated template and update it.
Project successfully created.
Executing before-prepare hook from /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/hooks/before-prepare/nativescript-dev-typescript.js
Found peer TypeScript 2.4.2
Preparing project...
preparePlatform yield +copyAppFiles
preparePlatform yield -copyAppFiles
preparePlatform yield +copyTnsModules

copyDependencyDir mkdir -p /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules/nativescript-bluetooth

copyDependencyDir +cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/nativescript-bluetooth /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules
copyDependencyDir -cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/nativescript-bluetooth /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules

copyDependencyDir mkdir -p /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules/tns-core-modules

copyDependencyDir +cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/tns-core-modules /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules
copyDependencyDir -cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/tns-core-modules /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules

copyDependencyDir mkdir -p /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules/tns-core-modules-widgets

copyDependencyDir +cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/tns-core-modules-widgets /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules
copyDependencyDir -cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/tns-core-modules-widgets /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules

Successfully prepared plugin nativescript-bluetooth for android.
Successfully prepared plugin tns-core-modules for android.
Successfully prepared plugin tns-core-modules-widgets for android.
preparePlatform yield -copyTnsModules

# NOTE THIS SECOND copyDependencyDir ON SINGLE BUILD RUN

copyDependencyDir mkdir -p /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules/nativescript-bluetooth

copyDependencyDir +cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/node_modules/nativescript-bluetooth /Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules

shelljs cp copyFileSync: could not write to dest file (code=EACCES):/Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/app/src/main/assets/app/tns_modules/nativescript-bluetooth/.git/objects/pack/pack-a98ffe92fb5a19a105f39a02942047edfc956aad.idx

shelljs cp Error: cp: cannot create directory '/Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules': No such file or directory
    at Object.error (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/node_modules/shelljs/src/common.js:112:27)
    at /Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/node_modules/shelljs/src/cp.js:246:18
    at Array.forEach (<anonymous>)
    at Object._cp (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/node_modules/shelljs/src/cp.js:225:11)
    at Object.cp (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/node_modules/shelljs/src/common.js:365:25)
    at TnsModulesCopy.copyDependencyDir (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/lib/tools/node-modules/node-modules-dest-copy.js:44:25)
    at TnsModulesCopy.copyModules (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/lib/tools/node-modules/node-modules-dest-copy.js:25:18)
    at NodeModulesBuilder.initialPrepareNodeModules (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/lib/tools/node-modules/node-modules-builder.js:45:28)
    at NodeModulesBuilder.<anonymous> (/Users/pv/.nvm/versions/node/v9.5.0/lib/node_modules/nativescript/lib/tools/node-modules/node-modules-builder.js:21:49)
    at Generator.next (<anonymous>)
cp: cannot create directory '/Users/pv/Development/GitHub/paulpv/temp2/nativescript-bluetooth-demo/Bluetooth/platforms/android/src/main/assets/app/tns_modules': No such file or directory
# build android
...

Repro # 2 (not 100% related, but semi-related to another issue w/ shelljs):

git clone https://github.com/bradmartin/nativescript-snackbar.git
cd nativescript-snackbar/demo
tns build android

> nativescript-dev-typescript@0.4.6 postinstall /Users/pv/Development/GitHub/paulpv/temp2/nativescript-snackbar/demo/node_modules/nativescript-dev-typescript
> node postinstall.js
Project already targets TypeScript ~2.6.0
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN demo No description
npm WARN demo No repository field.
npm WARN demo No license field.
added 45 packages in 8.23s
Copying template files...
Installing  tns-android
...
added 1 package in 3.745s
Error: sed: no such file or directory: /Users/pv/Development/GitHub/paulpv/temp2/nativescript-snackbar/demo/app/App_Resources/Android/app.gradle.
Check if you're using an outdated template and update it.
Project successfully created.
Executing before-prepare hook from /Users/pv/Development/GitHub/paulpv/temp2/nativescript-snackbar/demo/hooks/before-prepare/nativescript-dev-typescript.js
Found peer TypeScript 2.6.2
Preparing project...
preparePlatform
preparePlatform yield +copyAppFiles
preparePlatform yield -copyAppFiles
preparePlatform yield +copyTnsModules

copyDependencyDir mkdir -p /Users/pv/Development/GitHub/paulpv/temp2/nativescript-snackbar/demo/platforms/android/app/src/main/assets/app/tns_modules/nativescript-snackbar

copyDependencyDir +cp -RfL /Users/pv/Development/GitHub/paulpv/temp2/nativescript-snackbar/demo/node_modules/nativescript-snackbar /Users/pv/Development/GitHub/paulpv/temp2/nativescript-snackbar/demo/platforms/android/app/src/main/assets/app/tns_modules
Cycle link found.
{several seconds later...}
Cycle link found.
{several seconds later...}
Cycle link found.
{several seconds later...}
Cycle link found.
...

Solution

The non-obvious "right" solution is that the nativescript build process should probably not copy every file/folder to tns_modules.
It should probably ignore hidden/system files such as .git, .gitignore, etc.
The problem with this is, how to do it to be SCM agnostic?
My first instinct it to plagerize how npm install appears to ignore .git folder and copy only certain files/folders.
This change is beyond my current 4-day old experience w/ {N} to handle.

Another non-obvious solution may be to avoid the sloppy second copies.
I don't think this is relevant, because in the case of a legitimate livesync, multiple copy commands are inevitable.

This PR proposes a short term unblocker to replace the cp's force switch with a uupdate switch.
This way no unchanged files will be copied.
The problem will reproduce if the read-only file is modified and attempted to be copied a second time.
I don't foresee read-only files changing very often, so this weakness should hopefully not come up very often.
Thus the above "right" solution is the way things really need to be done.

ns-bot · 2018-02-08T21:39:03Z

Can one of the admins verify this patch?

rosen-vladimirov · 2018-02-13T00:54:00Z

run ci

rosen-vladimirov · 2018-02-13T08:13:32Z

run ci

Mitko-Kerezov · 2018-02-13T15:24:20Z

Hi @paulpv
Thank you for the detailed description of the issue as well as the summary. We really appreciate the pull request!

As far as the code is concerned I have run into some issues while testing your proposed changes.
I have tested on

Windows 8.1 machine with npm 5.6.0, nodejs v8.9.3
OS X El Capitan (10.11.6) with npm 5.6.0, nodejs v6.5.0
Ubuntu 14.04 with npm 5.3.0, nodejs v8.5.0

I have managed to reproduce the issue on all three platforms without your proposed changes, however with them I still get
cp: cannot create directory '<project>/platforms/android/src/main/assets/app/tns_modules': No such file or directory
error on all three platforms.

I would like to ask you for details about your environment in order to shed some further light on the matter.

rosen-vladimirov · 2018-05-02T09:52:19Z

Hey @paulpv ,
As we have not heard back from you for a while and as the suggested code does not fix the mentioned issue, I'm closing this PR. Feel free to reopen it in case you have any suggestions or concerns.

Fix NativeScript#3028

fcbaf51

paulpv mentioned this pull request Feb 8, 2018

Hang/Stall when using local nativescript-bluetooth dependency EddyVerbruggen/nativescript-bluetooth-demo#19

Open

rosen-vladimirov closed this May 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

copyDependencyDir's cp '-RfL' needs to change 'f' to 'u' to support sloppy second copies #3360

copyDependencyDir's cp '-RfL' needs to change 'f' to 'u' to support sloppy second copies #3360

paulpv commented Feb 8, 2018

ns-bot commented Feb 8, 2018

rosen-vladimirov commented Feb 13, 2018

rosen-vladimirov commented Feb 13, 2018

Mitko-Kerezov commented Feb 13, 2018

rosen-vladimirov commented May 2, 2018

copyDependencyDir's cp '-RfL' needs to change 'f' to 'u' to support sloppy second copies #3360

copyDependencyDir's cp '-RfL' needs to change 'f' to 'u' to support sloppy second copies #3360

Conversation

paulpv commented Feb 8, 2018

Here is the executive summary

More Detail

Repro

Solution

ns-bot commented Feb 8, 2018

rosen-vladimirov commented Feb 13, 2018

rosen-vladimirov commented Feb 13, 2018

Mitko-Kerezov commented Feb 13, 2018

rosen-vladimirov commented May 2, 2018