Do not parse config file if it doesn't exist #887

michaelruocco · 2019-11-16T16:32:22Z

Hi,

First off, thanks for producing such a useful plugin, I use it a lot and think it is great!

I have recently updated one of my projects to use version 6.0.0 and I have everything working fine, but I am finding that an exception is thrown and logged which is creating a bit of noise in my logs, so I was wondering if there might be a way around it?

It seems to be coming from the way the plugin is trying to lookup auth config, I wondered if it might just be the setup on my local mac, but it is happening on my travis CI builds too (the project is private at the moment but I was planning on making it public soon, I can do that earlier if seeing the whole logs would help?)

The exception that is being thrown is:

Failure when attempting to lookup auth config (docker repository: , configFile: /home/travis/.docker/config.json. Falling back to docker-java default behaviour
groovy.json.JsonException: Unable to process file: /home/travis/.docker/config.json
	at org.apache.groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:158)
	at groovy.json.JsonSlurper.parseFile(JsonSlurper.java:365)
	at groovy.json.JsonSlurper.parse(JsonSlurper.java:348)
	at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:109)
	at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:81)
	at com.bmuschko.gradle.docker.tasks.image.DockerPushImage.runRemoteCommand(DockerPushImage.groovy:55)
	at com.bmuschko.gradle.docker.tasks.AbstractDockerRemoteApiTask.start(AbstractDockerRemoteApiTask.groovy:73)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

When I run on my local mac it is similar but not exactly the same, although it does look to be the same sort of problem:

Failure when attempting to lookup auth config (docker repository: , configFile: /Users/michaelruocco/.docker/config.json. Falling back to docker-java default behaviour
groovy.json.JsonException: null not parse properly

The current character read is 'n' with an int value of 110
null not parse properly
line number 1
index number 1
no credentials server URL
.^
        at org.apache.groovy.json.internal.JsonParserCharArray.decodeNull(JsonParserCharArray.java:234)
        at org.apache.groovy.json.internal.JsonParserCharArray.decodeValueInternal(JsonParserCharArray.java:178)
        at org.apache.groovy.json.internal.JsonParserCharArray.decodeValue(JsonParserCharArray.java:157)
        at org.apache.groovy.json.internal.JsonParserCharArray.decodeFromChars(JsonParserCharArray.java:46)
        at org.apache.groovy.json.internal.JsonParserCharArray.parse(JsonParserCharArray.java:384)
        at org.apache.groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:112)
        at groovy.json.JsonSlurper.parseText(JsonSlurper.java:205)
        at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.runCredentialProvider(RegistryAuthLocator.groovy:252)
        at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.authConfigUsingStore(RegistryAuthLocator.groovy:271)
        at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:123)
        at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:81)
        at com.bmuschko.gradle.docker.tasks.image.DockerPushImage.runRemoteCommand(DockerPushImage.groovy:55)

In either case the plugin says that it will fall back to the default behaviour and then my image is still pushed as required so there is no functional problem. I tried to look at the docs and config to see if there was a way to turn off the auth config lookup (with the idea it would make the plugin just use the default behaviour and bypass the auth config lookup, but I wasn't able to find anything. Feel free to point me in the right direction if I have missed it!

For reference the relevant config that I have in my build.gradle file is:

plugins {
    id 'com.bmuschko.docker-remote-api' version '6.0.0'
}
...
import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage;
import com.bmuschko.gradle.docker.tasks.image.DockerPushImage;
...
docker {
    registryCredentials {
        username = System.getenv("DOCKER_USERNAME")
        password = System.getenv("DOCKER_PASSWORD")
    }
}

task buildImage(type: DockerBuildImage) {
    inputDir.set(file('.'))
    images.add('michaelruocco/verification-context-spring-app:latest')
}

task pushImage(type: DockerPushImage) {
    images.set(buildImage.images)
}

As I said feel free to point me in the right direction if I am just doing something stupid and missing some config, but any input you can offer would be much appreciated!

Thanks.

The text was updated successfully, but these errors were encountered:

bmuschko · 2019-11-16T16:43:24Z

Thanks for reporting.

Failure when attempting to lookup auth config (docker repository: , configFile: /home/travis/.docker/config.json. Falling back to docker-java default behaviour
groovy.json.JsonException: Unable to process file: /home/travis/.docker/config.json

Look like we are logging an error message in that situation. That's unnecessary. It should be a log message on a debug level.

The second issue seems to be something else. Looks like something's wrong with parsing your .docker/config.json file. Is your file empty by any chance? I think the code shouldn't even get there if there is no .docker/config.json file.

michaelruocco · 2019-11-16T22:51:59Z

Ah okay, yeah that sounds fair enough to me.

Regarding the second message that seems specific to my mac, I think I did see something about support for mac in the docs when I was reading earlier but I can't find it again now. I do have a file, but the contents looks a little odd:

{
  "HttpHeaders" : {
    "User-Agent" : "Docker-Client/19.03.4 (darwin)"
  },
  "credsStore" : "osxkeychain",
  "auths" : {
    "https://index.docker.io/v1/" : {

     }
     //there are further entries here but I don't want to show all the addresses / IPs as some are work related! but each one has a url and then an empty json block as above, maybe this is just something strange about the way docker is working on a mac maybe? I am not overly familiar with what the contents of this file should look like to be honest
  }
}

To be honest, my main concern was around the Travis CI build, changing the log level would solve that so I would be happy enough with that being included in the next release, although if you do have any ideas about my local mac issue I would be happy to hear them.

bmuschko · 2019-11-16T23:12:29Z

I tried to reproduce your issue with and without a config.json file on MacOSX (Docker for Desktop) but couldn't. I think there's something wrong with your JSON formatting.

Can you try to delete it and recreate it? What's your Docker distribution?

michaelruocco · 2019-11-16T23:26:55Z

It is version 2.1.0.4 (39773) I have the following component versions:

Engine: 19.03.4
Notary: 0.6.1
Compose: 1.24.1
Credential Helper: 0.6.3
Kubernetes: v1.14.7
Machine: 0.16.2

I just tried the last suggestion here on this thread: fsouza/go-dockerclient#677 i.e. delete the file, and untick the option for storing logins in osx keychain. After restarting and doing a docker login my file now contains:

{
	"auths": {
		"https://index.docker.io/v1/": {}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.4 (darwin)"
	},
	"credsStore": "desktop",
	"experimental": "disabled",
	"stackOrchestrator": "swarm"
}

but the error still persists, before I attempted a docker login the file did not exist, and then I just had the error being logged rather than a debug log. I am wondering if it is related to the use of keychain to store the passwords maybe?

bmuschko · 2019-11-16T23:30:42Z

No idea. My credsStore value is desktop. Would you mind setting up a test project that reproduces the issue + the Travis CI build?

michaelruocco · 2019-11-16T23:39:01Z

I will have a go and see if I can set something up tomorrow. Thanks for taking the time to look into it!

michaelruocco · 2019-11-17T07:26:26Z

Just thought, as well as looking into creating a project to reproduce the issue, would you mind sharing what your config.json file looks like? Then I will have an idea of a what the file should look like which might help identify the differences?

michaelruocco · 2019-11-17T07:56:14Z

I have added a repo here: https://github.com/michaelruocco/docker-gradle-plugin-bug. Let me know if any useful information is missing or needs to be added.

bmuschko · 2019-11-20T01:33:37Z

Thanks for providing the same project. I can reproduce the error. Looks like the issue only occurs when running the push command. Before I only tried out the build command. Looks like a bug to me.

Do you have any bandwidth to look into a fix for this? I may not be able to get to it anytime soon.

kgrech · 2019-11-20T05:03:29Z

@michaelruocco How does your config.json file looks like in case of the Travis CI build?

ghost · 2019-11-24T17:03:52Z

I am getting the same exception!!

michaelruocco · 2019-11-24T17:37:35Z

@kgrech sorry for the delayed response, I have had a busy week this week. I don't believe I have access to the file on the travis CI build. I guess I could try and add a cat command to print the file out when the build runs. I will try to do so shortly.

michaelruocco · 2019-11-24T18:17:40Z

@kgrech it turns out there is no config.json file present by the looks of things, the error being returned:

groovy.json.JsonException: Unable to process file: /home/travis/.docker/config.json

I should have checked more thoroughly before, the root exception when the travis build fails is:

Caused by: java.io.FileNotFoundException: /home/travis/.docker/config.json (No such file or directory)

However when it fails locally a file is present. I will try and clone / pull the repo shortly and investigate in more detail.

sebastianreloaded · 2019-11-25T08:40:38Z

in my opinion it would be the best to have an option to disable the .docker/config.json handling.
in my case it uses this file just fine for pulls from my private registry (aws), but pushes to the same registry always fail with "no basic auth credentials".
i switched to amazon-ecr-credential-helper for now, which works just fine.

michaelruocco · 2019-11-25T18:45:56Z

@sebastianreloaded yeah I would be happy with a flag or some config option to turn this handling off too. It would solve the issue well enough for me too.

lgniewkowski · 2019-11-26T10:13:04Z

@sebastianreloaded I have a similar problem. I have to delete docker.config to force the plugin to use the given credentials in build.gradle. Probably, in my situation problem is in url which is missing /v2 suffix.

bmuschko · 2019-12-02T20:00:03Z

@kgrech Did you have a chance to look into this issue more deeply?

kgrech · 2019-12-05T13:54:30Z

@bmuschko sorry for the delay, had some personal things to handle. I will definitely have a look this week

bmuschko · 2019-12-05T13:58:55Z

@kgrech Cool, thanks for getting back in it. Happy to help in any form. I might add some more test cases in the meantime that will help us identify the underlying issue.

danielefranceschi · 2019-12-05T14:40:25Z

FYI, same problem here in a gitlab-ci job:
(gradle 5.4.1 from dockerhub, plugin 6.0.0, docker-ce 18.09.6 on Centos7)

BUILD SUCCESSFUL in 18s
7 actionable tasks: 7 executed
Failure when attempting to lookup auth config (docker repository: REDACTED, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour
groovy.json.JsonException: Unable to process file: /root/.docker/config.json
	at org.apache.groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:158)
	at groovy.json.JsonSlurper.parseFile(JsonSlurper.java:365)
	at groovy.json.JsonSlurper.parse(JsonSlurper.java:348)
	at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:109)
	at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:81)
	at com.bmuschko.gradle.docker.tasks.image.DockerPushImage.runRemoteCommand(DockerPushImage.groovy:55)
	at com.bmuschko.gradle.docker.tasks.AbstractDockerRemoteApiTask.start(AbstractDockerRemoteApiTask.groovy:73)

Images (2 tags) are correctly pushed to the registry.

xiaoyao9184 · 2019-12-06T09:34:15Z

Same problem here.
But if use docker login command to log in, config.json will be created with 'auth' chapter.
you will get "Could not push image: no basic auth credentials" error.

Test on Linux, On windows docker desktop 'auth' chapter always empty, no error.

tmohme-hypoport · 2019-12-06T14:06:25Z

Same CI "problem": Missing "~/.docker/config.json" leads to irritating exception while DockerBuildImage still works fine.

Local DockerPushImage produces the exact same exception as for @michaelruocco (image is still pushed successfully).
Using

docker desktop for Mac 2.1.0.5
Engine: 19.03.5

Content of local (never manually edited, did a "docker login" on CL) "~/.docker/config.json":

{
  "stackOrchestrator" : "swarm",
  "credsStore" : "desktop",
  "HttpHeaders" : {
    "User-Agent" : "Docker-Client/19.03.2 (darwin)"
  },
  "auths" : {

  }
}

mikeykoelbl · 2019-12-06T15:03:47Z

I get a similar a different problem when trying to push an image, and the RegistryAuthLocator throws an error $HOME/.docker/config.json file does not exist if the file does not exist (Jenkins on Linux).

As @bmuschko says, at this should not be logged as an error, and I think it should be warn or below. Having no config.json file is not an error condition, and the best thing would be to return the defaultAuthConfig without an error if the configFile does not exist. Having a config.json that cannot be parsed, however, should be logged as a warn.

09:38:59  Failure when attempting to lookup auth config (docker repository: reflex-docker-snapshot.hannover-re.com, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour
09:38:59  groovy.json.JsonException: Unable to process file: /home/jenkins/.docker/config.json
09:38:59  	at org.apache.groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:158)
09:38:59  	at groovy.json.JsonSlurper.parseFile(JsonSlurper.java:365)
09:38:59  	at groovy.json.JsonSlurper.parse(JsonSlurper.java:348)
09:38:59  	at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:109)
09:38:59  	at com.bmuschko.gradle.docker.internal.RegistryAuthLocator.lookupAuthConfig(RegistryAuthLocator.groovy:81)
09:38:59  	at com.bmuschko.gradle.docker.tasks.image.DockerPushImage.runRemoteCommand(DockerPushImage.groovy:55)
...
09:38:59  Caused by: java.io.FileNotFoundException: /home/jenkins/.docker/config.json (No such file or directory)
09:38:59  	at java.base/java.io.FileInputStream.open0(Native Method)
09:38:59  	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
09:38:59  	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
09:38:59  	at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:79)
09:38:59  	at org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:1776)
09:38:59  	at org.apache.groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:152)
09:38:59  	... 90 more

cdancy · 2019-12-06T15:18:00Z

Anyone care to send in a PR? Seems like an easy enough fix and believe we all agree upon the solution.

mikeykoelbl · 2019-12-06T15:27:20Z

I am working on a PR for at least part of this issue.

mikeykoelbl · 2019-12-06T15:33:10Z

#894

bmuschko · 2019-12-10T19:31:48Z

@sebastianreloaded Can you provide a sample project of some sort or explain the behavior in more detail?

bmuschko · 2019-12-10T19:32:25Z

@lgniewkowski Do you have a sample of the config file that was failing for you?

sebastianreloaded · 2019-12-11T08:21:35Z

@sebastianreloaded Can you provide a sample project of some sort or explain the behavior in more detail?

i will try to put something together over the weekend.

kgrech · 2019-12-11T19:55:16Z

@bmuschko I see you have already fixed missing config case, thanks. Is there anything required to be done on the missing file case?

@michaelruocco Thanks for providing a sample project. I have checked, the issue is not in the format of config.json.
The issue is that docker-credential-desktop beeing called and it returns some response. The response is supposed to be a valid JSON and the plugin tries to parse it into a Map<String,String> here

However, it looks like docker-credential-desktop returns not a json or json in an unexpected format, so it is not able to parse. From your project message groovy.json.JsonException: null not parse properly I suspect it returns null value for one of the fields

Could you please run docker-credential-desktop tool manually in the CLI, provide it the url of your registery and paste me what does it return back (you can replace tokens with dummy text)? I do not have Mac to reproduce it

@bmuschko if you are a mac user as well, can you paste the output of docker-credential-desktop (I recall you have said you are able to reproduce as well)

kgrech · 2019-12-11T20:04:09Z

@danielefranceschi Could you paste the content of your /root/.docker/config.json file please?

bmuschko · 2019-12-11T20:54:13Z

@kgrech Please find my answers below.

I see you have already fixed missing config case, thanks. Is there anything required to be done on the missing file case?

I think that use case is covered now. I will likely release a new version of the plugin soon to unblocks users that ran into that specific use case.

if you are a mac user as well, can you paste the output of docker-credential-desktop (I recall you have said you are able to reproduce as well)

The usage of the docker-credential-desktop tool is a little weird. I am not quite sure how to get the credentials of the URL https://index.docker.io/v1/. The usage of the command docker-credential-desktop get https://index.docker.io/v1/ seems incorrect though no additional help is offered by the output.

michaelruocco · 2019-12-11T21:28:21Z

@kgrech I am not 100% sure which parts of the command you want me to run but:

Michaels-MBP-2:git michaelruocco$ docker-credential-desktop version
0.6.0

Michaels-MBP-2:git michaelruocco$ docker-credential-desktop list
{"https://XXX:443":"docker-pull","https://XXX.dkr.ecr.XXX.amazonaws.com":"AWS","https://XXX:443":"docker-pull","https://index.docker.io/v1/":"XXX","https://registry.heroku.com":"_"}

Looks like valid JSON to me, not sure if that helps much. I am not sure if I can do the same on travis CI, but I could give it a try if needed. Let me know if thats the type of thing you were after or you need me to run something else?

kgrech · 2019-12-12T05:50:15Z

@bmuschko @michaelruocco
so you need to run docker-credential-desktop get command. It will not print anything as it waits for std id. Type the URL of your registry there and press enter and CTRL+D (or Command+D probably in mac). It will print the access token JSON. Below is how it works in mine for the different helper

:

michaelruocco · 2019-12-12T09:34:13Z

So after doing that and using the standard docker hub registry which is the one I am using where I am getting this error I see the following:

Michaels-MacBook-Pro-2:git michaelruocco$ docker-credential-desktop get
https://index.docker.io/v1/
{"ServerURL":"https://index.docker.io/v1/","Username":"michaelruocco","Secret":"XXX"}

Again it looks to be valid json and the type of thing I would probably expect. It works in exactly the same fashion for each of the different registries I have got in terms of the json format always being the same. Hope this helps!

kgrech · 2019-12-12T09:46:33Z

@michaelruocco Thanks, this is helpful, I will check it in the evening!

danielefranceschi · 2019-12-12T11:56:23Z

@danielefranceschi Could you paste the content of your /root/.docker/config.json file please?

@kgrech That file does not exist, as I am in a CI pipeline and no docker login has been executed (and it will never be): credentials are used exclusively by the docker-java client library.

bmuschko · 2019-12-12T15:27:50Z

I released a new version that addressed the issue of a non-existent config file. Most users in this issue reported that very use case. Please open other, more specific issues for other use cases. Thanks!

danielefranceschi · 2019-12-12T15:49:05Z

what about the com.bmuschko.docker-spring-boot-application plugin?

bmuschko · 2019-12-12T15:52:21Z

@danielefranceschi Can you be more specific? The convention plugins use the same underlying mechanism for resolving the Docker config file.

danielefranceschi · 2019-12-12T16:08:55Z

plugins {
    id 'com.bmuschko.docker-spring-boot-application' version '6.1.0'
}

does not find the 6.1.0 version on repositories gradle-plugins, maven-central, jcenter
(6.0.0 pom was on gradle-plugins repo)

danielefranceschi · 2019-12-12T16:26:19Z

Now it works perfectly in CI, thanks! (and thanks for the great work too)

(don't know if it was a local issue, a nexus proxy issue, a repo issue, whatever)

kgrech · 2019-12-12T16:39:40Z

@michaelruocco I've tested you json and the plugin indeed can parse it with no issue. Maybe in your case, it returns something else.

@bmuschko I have raised a PR: #898
What I did is that I slightly improved your test case to work with a JSON provided by @michaelruocco
But most important is that I've added a debug line to log the response of the credential-helper so we would be able to ask users to run the plugin with a debug logging instead of asking to run credential helper manually for future issues. Would you consider merging it?

bmuschko self-assigned this Nov 16, 2019

bmuschko added the bug label Nov 16, 2019

bmuschko added this to the v6.0.1 milestone Nov 16, 2019

bmuschko mentioned this issue Nov 20, 2019

Plugin resolves and uses Docker credential helper #865

Merged

bmuschko changed the title ~~DockerPushImage is working but throwing exception and creating noise in logs~~ Do not parse config file if it doesn't exist Dec 12, 2019

bmuschko closed this as completed Dec 12, 2019

tmohme-hypoport mentioned this issue Jan 27, 2020

Unable to push to AWS ECR: HttpHostConnectException on port 80 #901

Closed

Do not parse config file if it doesn't exist #887

Do not parse config file if it doesn't exist #887

Comments

michaelruocco commented Nov 16, 2019 • edited Loading

bmuschko commented Nov 16, 2019

michaelruocco commented Nov 16, 2019 • edited Loading

bmuschko commented Nov 16, 2019

michaelruocco commented Nov 16, 2019

bmuschko commented Nov 16, 2019

michaelruocco commented Nov 16, 2019

michaelruocco commented Nov 17, 2019

michaelruocco commented Nov 17, 2019

bmuschko commented Nov 20, 2019

kgrech commented Nov 20, 2019

ghost commented Nov 24, 2019

michaelruocco commented Nov 24, 2019

michaelruocco commented Nov 24, 2019

sebastianreloaded commented Nov 25, 2019 • edited Loading

michaelruocco commented Nov 25, 2019

lgniewkowski commented Nov 26, 2019

bmuschko commented Dec 2, 2019

kgrech commented Dec 5, 2019

bmuschko commented Dec 5, 2019

danielefranceschi commented Dec 5, 2019 • edited Loading

xiaoyao9184 commented Dec 6, 2019

tmohme-hypoport commented Dec 6, 2019

mikeykoelbl commented Dec 6, 2019

cdancy commented Dec 6, 2019

mikeykoelbl commented Dec 6, 2019

mikeykoelbl commented Dec 6, 2019

bmuschko commented Dec 10, 2019

bmuschko commented Dec 10, 2019

sebastianreloaded commented Dec 11, 2019

kgrech commented Dec 11, 2019 • edited Loading

kgrech commented Dec 11, 2019

bmuschko commented Dec 11, 2019

michaelruocco commented Dec 11, 2019 • edited Loading

kgrech commented Dec 12, 2019

michaelruocco commented Dec 12, 2019

kgrech commented Dec 12, 2019

danielefranceschi commented Dec 12, 2019 • edited Loading

bmuschko commented Dec 12, 2019

danielefranceschi commented Dec 12, 2019

bmuschko commented Dec 12, 2019

danielefranceschi commented Dec 12, 2019 • edited Loading

danielefranceschi commented Dec 12, 2019 • edited Loading

kgrech commented Dec 12, 2019

michaelruocco commented Nov 16, 2019 •

edited

Loading

michaelruocco commented Nov 16, 2019 •

edited

Loading

sebastianreloaded commented Nov 25, 2019 •

edited

Loading

danielefranceschi commented Dec 5, 2019 •

edited

Loading

kgrech commented Dec 11, 2019 •

edited

Loading

michaelruocco commented Dec 11, 2019 •

edited

Loading

danielefranceschi commented Dec 12, 2019 •

edited

Loading

danielefranceschi commented Dec 12, 2019 •

edited

Loading

danielefranceschi commented Dec 12, 2019 •

edited

Loading