cf push fails when you have a read only js file bigger than 64KB

[bveenvliet]

When do a cf push to run.pivotal.io with a JS file that is 65KB or greater using the Windows 64 bit CF CLI:

• version 6.12.2 works (cf version 6.12.2-24abed3-2015-07-15)
• version 6.12.4 works (cf version 6.12.4-b4b6af1-2015-09-18)
• version 6.13 FAIL (cf version 6.13-e68ce0f-2015-10-15)
• version 6.14 FAIL (cf version 6.14.0+2654a47-2015-11-18)

easy to test, just add a huge comment block to any JS file, set the file to read only and cf push it.
Very strange thing is that the overall size of files does not include the >64KB file.

C:\Test>cf push dev-routetest
Using manifest file C:\Test\manifest.yml

Updating app XXXXXX in org XXXXXX-org / space development as me@person.com...
OK

Uploading test...
Uploading app files from: C\Test
Uploading 15.9K, 16 files <<------------ should be approx 85KB if it included the 70KB js file
Done uploading
FAILED
Error uploading application.
The resource file mode is invalid: File mode '0444' is invalid.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/108545872.

[bveenvliet]

and some additional CF_TRACE output....

REQUEST: [2015-11-19T10:45:46-05:00]
PUT /v2/resource_match HTTP/1.1
Host: api.run.pivotal.io
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.14.0+2654a47 / windows

[{"sha1":"5836ac31b71d3e0b2a33f169572d5f450cf9a830","size":2238},{"sha1":"8ff8685bc59a7b2beb02f668c00a1d7c1ca3a3b0","size":2700},{"sha1":"3bd06a3c74f2541727913432672eef7f8425fced","size":3384},{"sha1"
:"0","size":0},{"sha1":"20bca42e8a07b9c43c505c7abd5083f0768a7784","size":758},{"sha1":"c1ca55ff04c1b83f73dace7f0aad5be54e2a56e5","size":921},{"sha1":"1fea5a87bf4c7846f6677753b14b3a2de65891ea","size":7
1965},{"sha1":"cae1a9135cd82495054f792a9f10346c7f2d712f","size":1456},{"sha1":"0","size":0},{"sha1":"433d1bd9009730619383ec6ae78301bbe0ce4527","size":196},{"sha1":"d3d790f365b740cb70b217efa64394424783
8d01","size":288},{"sha1":"80ca1bd33edb63967d274ef3397169b04f59129c","size":792},{"sha1":"0","size":0},{"sha1":"60dec8f2f9d44b738b13dd4a28b661cbc558e612","size":100},{"sha1":"dd51ffc8937217898061004e7
7b2a9104a838554","size":476},{"sha1":"a80879a1a1967ce33796378b2b4fabc208a8151a","size":794},{"sha1":"be641dfd35a22db6f1b179cd9c836dab95bf2c75","size":210}]

RESPONSE: [2015-11-19T10:45:46-05:00]
HTTP/1.1 200 OK
Content-Length: 66
Connection: keep-alive
Content-Type: application/json;charset=utf-8
Date: Thu, 19 Nov 2015 15:45:41 GMT
Server: nginx
X-Cf-Requestid: a04ee878-bd3c-4728-64dd-d44b67af8320
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 207ace22-2ef5-42bd-4ac1-0249814193bb::e19916b1-6256-4042-a83f-953a3c286d89

[{"sha1":"1fea5a87bf4c7846f6677753b14b3a2de65891ea","size":71965}]
Uploading app files from: C:\Test
Uploading 15.9K, 16 files

[dkoper]

Hi Brad,

I cannot reproduce the error:

D:\projects\cf\node-bigfile>cf -v
cf version 6.14.0+2654a47-2015-11-18

D:\projects\cf\node-bigfile>dir
 Volume in drive D is DATA
 Volume Serial Number is 3ECB-5B5E

 Directory of D:\projects\cf\node-bigfile

20/11/2015  08:32    <DIR>          .
20/11/2015  08:32    <DIR>          ..
20/11/2015  08:32         7,254,130 app.js
11/06/2014  05:13               156 package.json
11/06/2014  05:13                17 Procfile
               3 File(s)      7,254,303 bytes
               2 Dir(s)  19,298,983,936 bytes free

D:\projects\cf\node-bigfile>cf push big-nodejs-file
<SNIP>
requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: big-nodejs-file.a1-app.cf-app.com
last uploaded: Thu Nov 19 21:34:07 UTC 2015
stack: cflinuxfs2
buildpack: node.js 1.5.3

     state     since                    cpu    memory          disk          details
#0   running   2015-11-20 08:34:34 AM   0.0%   49.3M of 256M   40.2M of 1G

D:\projects\cf\node-bigfile>

Even if I use cygwin chmod 0444 app.js to adjust the file's permissions, cf delete and push it works fine.
Is there anything other specific in your environment that could cause a permission error? What other files are in your directory?

About the 15.9K, note that CC caches any files >64K, so the CLI doesn't upload it on a subsequent push. The trace you posted shows that we upload a list of files (well, just size and checksum of each), which includes 71965 byte file, and CC sends back a response with this file (saying it matched a file in its cache, so no need for CLI to upload it again).

Cheers,
Dies Koper
CF CLI PM

[bveenvliet]

Strange. It happens for me every time.

C:\RouteTest>cf delete dev-routetest

Really delete the app dev-routetest?> y
Deleting app dev-routetest in org removed-org / space development as removed@domain.com...
OK

C:\RouteTest>cf push dev-routetest --no-start
Using manifest file C:\RouteTest\manifest.yml

Creating app dev-routetest in org removed-org / space development as removed@domain.com...
OK

Using route dev-routetest.cfapps.io
Binding dev-routetest.cfapps.io to dev-routetest...
OK

Uploading dev-routetest...
Uploading app files from: C:\RouteTest
Uploading 67.8K, 16 files
Done uploading
FAILED
Error uploading application.
The resource file mode is invalid: File mode '0444' is invalid.

C:\RouteTest>attrib *.js
A R C:\RouteTest\apiserver.js
A C:\RouteTest\config.js
A C:\RouteTest\logger.js
A C:\RouteTest\scratchpad.js
A C:\RouteTest\version.js

C:\RouteTest>attrib -r apiserver.js

C:\RouteTest>cf push dev-routetest --no-start
Using manifest file C:\RouteTest\manifest.yml

Updating app dev-routetest in org removed-org / space development as removed@domain.com...
OK

Uploading dev-routetest...
Uploading app files from: C:\RouteTest
Uploading 67.8K, 16 files
Done uploading
OK

C:\RouteTest>dir apiserver.js
Volume in drive C has no label.
Volume Serial Number is 5621-21A4

Directory of C:\RouteTest

2015-11-20 16:54 82,697 apiserver.js
1 File(s) 82,697 bytes
0 Dir(s) 7,168,012,288 bytes free

C:\RouteTest>cf --version
cf version 6.14.0+2654a47-2015-11-18

C:\RouteTest>

[bveenvliet]

and the changelog for version 6.13 includes 6 changes with the words 'file mode'

[dkoper]

Sorry, my bad - I had forgotten to make the file RO with attrib before I started my test.
With attrib +r, I get:

Uploading bog-nodejs-file...
FAILED
Error uploading application.
open d:\projects\cf\node-bigfile\app.js: Access is denied.

We'll investigate.

[krishicks]

Hi @bveenvliet:

This change was made as part of the switch to using the new Cloud Foundry runtime, Diego. The file mode must be at least 0600.

For completeness, the original story in the Cloud Foundry API backlog for the feature here, the code that (correctly) returns this error here, and get more information on the Apps API the CLI uses for uploading files here.

[krishicks]

I've asked CAPI to change the error message to be more helpful: rather than saying the filemode is invalid, it should describe the solution, which is setting your filemode to at least 0600. When they create a story I'll link it here.

[bveenvliet]

@krishicks @dkoper Our windows build machine gets latest code from the repository and does a cf push on the files (which have a +R attribute given to them from the source control). So now there is an extra step turning all of the files > 64KB to make them writable before deploying to Diego. Would be a cleaner solution if there was a command line arg to ignore file permissions during the cf push.

[krishicks]

@bveenvliet That's not something the CLI team can decide.

Perhaps @dieucao, the CF API PM, and/or @ematpl, the CF Runtime Diego PM, can elaborate on why the requirement exists.

[dkoper]

Hi Brad,

We've fixed the issue, feel free to download our Edge binary to try until we publish our next cli release.

Regards,
Dies Koper
CF CLI PM

[dkoper]

This fix is included in cf CLI release v6.14.1 published today.
https://github.com/cloudfoundry/cli#downloads

[bveenvliet]

Thanks!

[CorpusCallosum]

Hi I am getting a similar error:

FAILED
Error uploading application.
The resource file mode is invalid: File mode '0551' is invalid.

Read through the comments here, but I am unclear as to what the solution is? WHICH file needs to be in filemode 0600? OR, do I need to install CLI v6.14.1 to resolve? I'm currently on cf version 6.15.0+fa1bfe2-2016-01-13

Thanks.

[krishicks]

@CorpusCallosum What platform are you using cf push from? There was a fix done for Windows specifically but a Linux/OS X fix has not yet been released.

[CorpusCallosum]

Hey @krishicks , thanks for the quick reply. I'm on OSX, do you have any workaround available in the meantime for OSX users?

[krishicks]

The easiest thing to do is to chmod all your files, sadly. This is what used to happen automatically when pushing your files; the mode was set to 0744 for all files by default.

[CorpusCallosum]

Hi, I tried changing file permissions by running sudo chmod -R 744 .

Still getting the same error when I run cfpush however:

FAILED
Error uploading application.
The resource file mode is invalid: File mode '0551' is invalid.

Is there some way to find which file is causing the problem?

[krishicks]

Try find . -type f -perm 0551. Also, read the file manpage for how you can use -perm to search for files with other file modes; 0551 may not be your only problematic file.

[CorpusCallosum]

Hey Kris, I ran that command and this is the result:
sudo find . -type f -perm 0551 ./node_modules/ibm_db/installer/clidriver/adm/db2trc

So, as you can see, ironically the file that is giving me trouble is the clidriver itself. This is a problem, because although I can change permissions on the file, as soon as I npm run cfpush the driver get's downloaded again, with the wrong permissions, and causes the push to fail again.

It's a Catch 22! What to do?

Here is the log from the driver download

Downloading DB2 ODBC CLI Driver from http://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/macos64_odbc_cli.tar.gz...
Download and extraction of DB2 ODBC CLI Driver completed successfully ...
  CXX(target) Release/obj.target/odbc_bindings/src/odbc.o
  CXX(target) Release/obj.target/odbc_bindings/src/odbc_connection.o
  CXX(target) Release/obj.target/odbc_bindings/src/odbc_statement.o
  CXX(target) Release/obj.target/odbc_bindings/src/odbc_result.o
  CXX(target) Release/obj.target/odbc_bindings/src/dynodbc.o
  SOLINK_MODULE(target) Release/odbc_bindings.node

[krishicks]

That depends on what npm run cfpush is doing. You'd need to change it such that it does the chmod -R prior to running cf push.

[CorpusCallosum]

Ah yes as it turns out the npm run cfpush was also doing an update, which is why my perms were being overriden. Here is my updated npm script command that got it working for me, in case it help other poor souls:

"cfpush": "npm prune --production && npm update --production && sudo chmod -R 744 * && cf push cognitive-x-dev && npm update"

BTW, if this is indeed still a bug in the OSX version, perhaps the issue should be reopened? Thanks!

[krishicks]

This issue is specific to Windows, so it should not be opened. We're also already aware of the problem and have stories to fix it on non-Windows platforms.

[dmikusa]

@krishicks - Do you have a link to the non-Windows story in PT that you can share? I searched but couldn't find it.

[krishicks]

Set executable bit on all files that are missing it on non-Windows platforms by default: https://www.pivotaltracker.com/story/show/109770364

[matthewford]

Also seeing something similar with files in the .git folder and ruby bundler cache.

The resource file mode is invalid: File mode '0444' is invalid.

[dkoper]

@matthewford Feel free to submit a new issue with details and we'll take a look. This issue is closed; a new issue is easier to track for us and less likely to get lost.

[forbushbl]

This issue seems to be related to a problem I recently reported on the Cloud Controller. See cloudfoundry/cloud_controller_ng#1705