In fresh container : /etc/chef/client.pem: no such file or directory

[jp]

Hey all,

I'm using the unedited web_server example in the README.md of this repo.

The chef-client is ending with the following error :

[2015-04-09T11:33:50-04:00] ERROR: machine_image[web_server] (@recipe_files::/home/jpellet/dev/ci/chef-zero/example.rb line 3) had an error: Docker::Error::NotFoundError: Expected([200, 
201, 202, 203, 204, 304]) <=> Actual(404 Not Found)

The docker logs are showing that the container is properly created, the chef-client is trying to create the key in /etc/chef/client.pem but the file is apparently not properly created as you can see in the following docker logs:

INFO[5556] GET /v1.16/images/chef:web_server/json       
INFO[5556] +job image_inspect(chef:web_server)          
INFO[5556] -job image_inspect(chef:web_server) = OK (0) 
INFO[5556] POST /v1.16/containers/create                
INFO[5556] +job create()                                
INFO[5556] +job log(create, 9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b, chef:mario) 
INFO[5556] -job log(create, 9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b, chef:mario) = OK (0) 
INFO[5556] -job create() = OK (0)                       
INFO[5556] POST /v1.16/containers/9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b/copy 
INFO[5556] +job container_copy(9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b, etc/chef/client.pem) 
stat /var/lib/docker/aufs/mnt/9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b/etc/chef/client.pem: no such file or directory
INFO[5556] -job container_copy(9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b, etc/chef/client.pem) = ERR (1) 
ERRO[5556] stat /var/lib/docker/aufs/mnt/9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b/etc/chef/client.pem: no such file or directory 
ERRO[5556] Handler for POST /containers/{name:.*}/copy returned error: Could not find the file /etc/chef/client.pem in container 9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f4
16dff1b 
ERRO[5556] HTTP Error: statusCode=500 Could not find the file /etc/chef/client.pem in container 9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b 
INFO[5556] DELETE /v1.16/containers/9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b 
INFO[5556] +job rm(9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b) 
INFO[5556] +job log(destroy, 9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b, chef:mario) 
INFO[5556] -job log(destroy, 9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b, chef:mario) = OK (0) 
INFO[5556] -job rm(9804928d1b06ae1044f77af55a4abd4f2a7740e898e3995a1d54a73f416dff1b) = OK (0) 
INFO[5557] POST /v1.16/containers/web_server/stop?t=0?  
INFO[5557] +job stop(web_server)                        
Container already stopped

chef-client output :

jpellet@box:~/dev/ci/chef-zero$ CHEF_DRIVER=docker chef-client -z example.rb 
[2015-04-09T11:33:47-04:00] WARN: No config file found or specified on command line, using command line options.
[2015-04-09T11:33:47-04:00] WARN: No cookbooks directory found at or above current directory.  Assuming /home/jpellet/dev/ci/chef-zero.
Starting Chef Client, version 12.2.1
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Compiling Cookbooks...
[2015-04-09T11:33:48-04:00] WARN: Node box has an empty run list.
Converging 2 resources
Recipe: @recipe_files::/home/jpellet/dev/ci/chef-zero/example.rb
  * machine_image[web_server] action create
    - update node web_server at http://localhost:8889
    -   update normal.chef_provisioning.reference.allocated_at from "2015-04-09 15:18:44 UTC" to "2015-04-09 15:33:48 UTC"
    -   remove normal.chef_provisioning.reference.container_id
    - update node web_server at http://localhost:8889
    -   add normal.chef_provisioning.reference.container_id = "a63ce0dfeed12b8cb9eb0caae0310b6812b76fec6e618dfed82ce67b6771bb6f"
    - generate private key (2048 bits)
    ================================================================================
    Error executing action `create` on resource 'machine_image[web_server]'
    ================================================================================

    Docker::Error::NotFoundError
    ----------------------------
    Expected([200, 201, 202, 203, 204, 304]) <=> Actual(404 Not Found)

    Resource Declaration:
    ---------------------
    # In /home/jpellet/dev/ci/chef-zero/example.rb

      3: machine_image 'web_server' do
      4:   recipe 'apache'
      5: 
      6:   machine_options :docker_options => {
      7:       :base_image => {
      8:           :name => 'ubuntu',
      9:           :repository => 'ubuntu',
     10:           :tag => '14.04'
     11:       }
     12:   }
     13: end
     14: 

    Compiled Resource:
    ------------------
    # Declared in /home/jpellet/dev/ci/chef-zero/example.rb:3:in `from_file'

    machine_image("web_server") do
      action :create
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      chef_server {:chef_server_url=>"http://localhost:8889", :options=>{:client_name=>"box", :signing_key_filename=>nil}}
      driver "docker"
      machine_options {:docker_options=>{:base_image=>{:name=>"ubuntu", :repository=>"ubuntu", :tag=>"14.04"}}}
      declared_type :machine_image
      cookbook_name "@recipe_files"
      recipe_name "/home/jpellet/dev/ci/chef-zero/example.rb"
      run_list_modifiers [#<Chef::RunList::RunListItem:0x007fc949cbbf00 @version=nil, @type=:recipe, @name="apache">]
    end


Running handlers:
[2015-04-09T11:33:49-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-04-09T11:33:49-04:00] ERROR: Exception handlers complete
[2015-04-09T11:33:49-04:00] FATAL: Stacktrace dumped to /home/jpellet/.chef/local-mode-cache/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 2.872126338 seconds
[2015-04-09T11:33:50-04:00] ERROR: machine_image[web_server] (@recipe_files::/home/jpellet/dev/ci/chef-zero/example.rb line 3) had an error: Docker::Error::NotFoundError: Expected([200, 201, 202, 203, 204, 304]) <=> Actual(404 Not Found)

[2015-04-09T11:33:50-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[marc-]

Check your docker service API version (docker --version). It might be incompatible with docker-api minimum required version .

[jp]

I'm currently using this version :

$ docker --version
Docker version 1.5.0, build a8a31ef

The calls on the docker-api are done on the 1.16 version of the API and it seems to be working properly ...

[marc-]

I've just tried to run it myself. Works fine for me. Though my chef-client version is 11.18.6 (had pain with >=12.0.0) and I also use chef-provisioning v0.19 (anything >=v0.20 is not supported yet).

[jp]

What is surprising to me is that the docker log is showing an attempt to write the file etc/chef/client.pem without the first /

Here are the debug log of the chef client

    - update node pup_backend at http://localhost:8889
    -   add normal.chef_provisioning.reference.container_id = "9d4c4a4a2506e68cbca48c90e6a3595a39dcbd73b347755348470cd1bcbe9376"[2015-04-09T12:51:16-04:00] INFO: Processing private_key[in_memory] action create (basic_chef_client::block line 78)
[2015-04-09T12:51:16-04:00] DEBUG: providers for generic private_key resource enabled on node include: []
[2015-04-09T12:51:16-04:00] DEBUG: providers that refused resource private_key[in_memory] were: []
[2015-04-09T12:51:16-04:00] DEBUG: providers that support resource private_key[in_memory] include: []
[2015-04-09T12:51:16-04:00] DEBUG: no providers supported the resource, falling back to enabled handlers
[2015-04-09T12:51:16-04:00] DEBUG: providers that survived replacement include: []
[2015-04-09T12:51:16-04:00] DEBUG: dynamic provider resolver FAILED to resolve a provider
[2015-04-09T12:51:16-04:00] DEBUG: Platform basic_chef_client not found, using all defaults. (Unsupported platform?)

    - generate private key (2048 bits)[2015-04-09T12:51:16-04:00] DEBUG: execute 'ls -d /etc/chef/client.pem' with options {:read_only=>true}
[2015-04-09T12:51:16-04:00] DEBUG: stopped /containers/web_server

Is the issue is the "Unsupported platform" ?

[jp]

Ok I got it here is the super hack I had to do:

unset http_proxy

Well, that makes me a bit sad as now it move forward, but I'm stuck at this point:

- write file /etc/chef/client.rb on mario4
- write file /tmp/detect.sh on web_server
[2015-04-09T13:17:45-04:00] ERROR: Error connecting to https://www.chef.io/chef/metadata?v=&prerelease=false&p=ubuntu&pv=14.04&m=x86_64, retry 1/5

I can't have a proxy set to connect to docker but I need to proxy to connect to chef.io.

[jp]

It looks very similar to this issue: tmatilai/vagrant-proxyconf#109

[jp]

As stated in the following documentation, Docker cli will honour the proxy set in the environment which is also true for the NO_PROXY variable : icecrime/docker@3a1dbef

I'm not able to run the chef-client as expected with the following:

export NO_PROXY=/var/run/docker.sock,localhost,127.0.0.1,.sock

I guess that's more of a Docker issue then, I'm closing this.