1.1.2 does not work correctly with directory environments on Puppetserver 1.1.3

[elconas]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 3.8.7 (Puppetserver 1.1.3-1.el7)
• Ruby: jruby (with puppetserver)
• Distribution: Centos 7
• Module version: 1.1.2

How to reproduce (e.g Puppet code you use)

Use environment name "myenvironment".

Puppet run failes with:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not find parent provider ruby of curl on node NODE

In this case puppet/archive is consumed by rtyler/jenkins 1.7.0 (https://github.com/jenkinsci/puppet-jenkins/releases/tag/v1.7.0).

What are you seeing

Class loading failes, caused by https://tickets.puppetlabs.com/browse/SERVER-973 on Puppetserver.

What behaviour did you expect instead

Should compile.

[elconas]

Maybe you can also do require File.dirname(FILE) + ' /../../path/to/helper/code' as mentioned here

[gregswift]

In the same software environment and same module setup (archive 1.1.2 or 1.1.3-rc0 with jenkinsci-puppet 1.7.0) but with an environment named "jenkins" I saw the same autoload error. When i researched it i found #105 and made an environment called production. Autoload error stopped and was replaced with

Error: Could not autoload puppet/type/archive: /var/lib/puppet/lib/puppet/type/archive.rb:53: syntax error, unexpected ':', expecting ')'
  newparam(:path, namevar: true) do
                          ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: syntax error, unexpected ':', expecting ')'
...aram(:allow_insecure, boolean: true, parent: Puppet::Paramet...
                              ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: Can't assign to true
...allow_insecure, boolean: true, parent: Puppet::Parameter::Bo...
                              ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: syntax error, unexpected ':', expecting '='
...secure, boolean: true, parent: Puppet::Parameter::Boolean) d...
                              ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: syntax error, unexpected ')', expecting $end
...t: Puppet::Parameter::Boolean) do
                              ^
Error: Could not retrieve catalog from remote server: Could not intern from text/pson: Could not autoload puppet/type/archive: /var/lib/puppet/lib/puppet/type/archive.rb:53: syntax error, unexpected ':', expecting ')'
  newparam(:path, namevar: true) do
                          ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: syntax error, unexpected ':', expecting ')'
...aram(:allow_insecure, boolean: true, parent: Puppet::Paramet...
                              ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: Can't assign to true
...allow_insecure, boolean: true, parent: Puppet::Parameter::Bo...
                              ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: syntax error, unexpected ':', expecting '='
...secure, boolean: true, parent: Puppet::Parameter::Boolean) d...
                              ^
/var/lib/puppet/lib/puppet/type/archive.rb:223: syntax error, unexpected ')', expecting $end
...t: Puppet::Parameter::Boolean) do
                              ^
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

So i then changed to elconas' branch for #226 and its giving me that same set of syntax errors.

[elconas]

What is your environment ? (Puppetwise).
In our puppetserver 1.1.3 setup, the fix in #226 works. It also works with puppet apply. I can not test passenger or any other puppet server implementation.

[alexjfisher]

Could not find parent provider ruby of curl

@elconas The ruby parent provider is in lib/puppet/provider/archive, not under lib/puppet_x. Are you 100% sure you're not hitting https://tickets.puppetlabs.com/browse/SERVER-94 instead of https://tickets.puppetlabs.com/browse/SERVER-973?

ie Do all of your environments (including the one your puppet server is in) use puppet-archive 1.1.2?

[alexjfisher]

From #105

I get this error randomly on my clients and if I re-run puppet sometimes it goes away and works without any problems. Any idea why it randomly occurs ?

@elconas I think there's a good chance that #226 appears to fix the problem, but you just got lucky when running puppet a second time.

@gregswift I'm guessing you're running on a ruby 1.8 system? (CentOS/RHEL 6 usually).

[gregswift]

@alexjfisher yes, its CentOS 6. otherwise the rest is like @elconas

Puppet: 3.8.7 (Puppetserver 1.1.3-1.el7)
Ruby: jruby (with puppetserver)
Distribution: Centos 6
Module version: 1.1.2

[alexjfisher]

@gregswift Since version 1.0.0 of this module, ruby 1.8 is no longer supported. If you need a puppet 3 based puppetserver, you may have to think about migrating to Centos 7. It's not just puppet-archive, there are now lots of modules that don't support ruby 1.8 in their most recent releases.

[elconas]

I tested again on the Puppetmaster with the code from pull #226. I created different environments and changed the version of puppet/archive in the branches (only the metadata.json).

• Puppetmaster environment: "production".
• Agent environments
  • devvm_development (with puppet/archive 1.1.3-development)
  • devvm_master (with puppet/archive 1.1.3-master)
  • devvm_emptybranch (NO puppet/archive installed)

[r10k@puppet ~]$ grep "version\":" /etc/puppet/environments/devvm_*/modules/archive/metadata.json
/etc/puppet/environments/devvm_development/modules/archive/metadata.json:  "version": "1.1.3-development",
/etc/puppet/environments/devvm_master/modules/archive/metadata.json:  "version": "1.1.3-master",

Then I added code to the environments to print out the module version and create an dummy archive.

[r10k@puppet ~]$ grep -v -e ^# -e "^$" /etc/puppet/environments/devvm_*/manifests/site.pp
/etc/puppet/environments/devvm_development/manifests/site.pp:$metadata = load_module_metadata('archive')
/etc/puppet/environments/devvm_development/manifests/site.pp:notify { $metadata['version']: }
/etc/puppet/environments/devvm_development/manifests/site.pp:archive { "/tmp/destfile":
/etc/puppet/environments/devvm_development/manifests/site.pp:  source  => "file:///tmp/sourcefile",
/etc/puppet/environments/devvm_development/manifests/site.pp:}

/etc/puppet/environments/devvm_emptybranch/manifests/site.pp:$metadata = load_module_metadata('archive')
/etc/puppet/environments/devvm_emptybranch/manifests/site.pp:notify { $metadata['version']: }

/etc/puppet/environments/devvm_master/manifests/site.pp:$metadata = load_module_metadata('archive')
/etc/puppet/environments/devvm_master/manifests/site.pp:notify { $metadata['version']: }
/etc/puppet/environments/devvm_master/manifests/site.pp:archive { "/tmp/destfile":
/etc/puppet/environments/devvm_master/manifests/site.pp:  source  => "file:///tmp/sourcefile",
/etc/puppet/environments/devvm_master/manifests/site.pp:}

Then I cleared the environment cache (curl -X DELETE https://localhost:8140/puppet-admin-api/v1/environment-cache)

Then I ran the agent. It seems it ALWAYS failes the first time:

[vagrant@testvm ~]$ sudo puppet agent --verbose --ignorecache --no-daemonize --no-usecacheonfailure --no-splay --onetime --server puppet --certname $(facter -p fqdn) --environment devvm_devel opment
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not find parent provider ruby of curl on node testvm.vagrant.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

However it WORKS the second time and so one:

[vagrant@testvm ~]$ sudo puppet agent --verbose --ignorecache --no-daemonize --no-usecacheonfailure --no-splay --onetime --server puppet --certname $(facter -p fqdn) --environment devvm_devel opment
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for testvm.vagrant.local
Info: Applying configuration version '1475359223'
Notice: 1.1.3-development
Notice: /Stage[main]/Main/Notify[1.1.3-development]/message: defined 'message' as '1.1.3-development'
Notice: Finished catalog run in 0.15 seconds

Also module version is different in the environments:

[vagrant@testvm ~]$ sudo puppet agent --verbose --ignorecache --no-daemonize --no-usecacheonfailure --no-splay --onetime --server puppet --certname $(facter -p fqdn) --environment devvm_development
...
Notice: /Stage[main]/Main/Notify[1.1.3-development]/message: defined 'message' as '1.1.3-development'
....
[vagrant@testvm ~]$ sudo puppet agent --verbose --ignorecache --no-daemonize --no-usecacheonfailure --no-splay --onetime --server puppet --certname $(facter -p fqdn) --environment devvm_master
...
Notice: /Stage[main]/Main/Notify[1.1.3-master]/message: defined 'message' as '1.1.3-master'
....

Only devvm_emptybranch faiels all the time as expected:

[vagrant@testvm ~]$ sudo puppet agent --verbose --ignorecache --no-daemonize --no-usecacheonfailure --no-splay --onetime --server puppet --certname $(facter -p fqdn) --environment devvm_emptybranch
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find module archive in environment devvm_emptybranch at /etc/puppet/environments/devvm_emptybranch/manifests/site.pp:4 on node testvm.vagrant.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

If I clean the environment cache again, the first run failes again.

@alexjfisher Can you explain ????

[elconas]

One more note.

With the original code (puppet/archive 1.1.3-rc0 aka master), the test failes occasionally (1 out of 10) with:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not find parent provider ruby of curl on node testvm.vagrant.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Any it seems to ALWAYS fail after plugin sync (e.g. when running agains devvm_emptybranch before):

Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet_x]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet_x/bodeco]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet_x/bodeco/util.rb]/ensure: defined content as '{md5}f9455d96862232411f74b1a0f45e3db7'
Notice: /File[/var/lib/puppet/lib/puppet_x/bodeco/archive.rb]/ensure: defined content as '{md5}72c33f95411dcdfabfa6f402e267cedd'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/assemble_nexus_url.rb]/ensure: defined content as '{md5}d37ee34c8b63728eefeca30ee3965509'
Notice: /File[/var/lib/puppet/lib/puppet/provider/archive]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/archive/curl.rb]/ensure: defined content as '{md5}0e994a1a78ac64ca4ae51f6d63873cc7'
Notice: /File[/var/lib/puppet/lib/puppet/provider/archive/ruby.rb]/ensure: defined content as '{md5}4192e1bb337ce7b51cc5f3bff534f286'
Notice: /File[/var/lib/puppet/lib/puppet/provider/archive/wget.rb]/ensure: defined content as '{md5}bc5b55f585bad751f988c03a09cea71e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/artifactory_sha1.rb]/ensure: defined content as '{md5}4838e5244aa4c24c3a3581faa7863ada'
Notice: /File[/var/lib/puppet/lib/puppet/type/archive.rb]/ensure: defined content as '{md5}c24db21052716bead179495d8159b276'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/go_md5.rb]/ensure: defined content as '{md5}845b2d433175a617d9a28cd33c25c3c5'
Notice: /File[/var/lib/puppet/lib/facter/archive_windir.rb]/ensure: defined content as '{md5}a646f8234e87041223c97dd919801ad8'
Notice: /File[/var/lib/puppet/lib/puppet_x/bodeco/cacert.pem]/ensure: defined content as '{md5}d7f7dd7e3ede3e323fc0e09381f16caf'
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not autoload puppet/type/archive: Could not autoload puppet/provider/archive/curl: Could not find parent provider ruby of curl on node testvm.vagrant.local
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

So it seems both somehow work "strange" :)

[gregswift]

thanks. We've got an el7 upgrade in our plans.

[elconas]

@alexjfisher .. im not sure about SERVER-94. Now that I have read this:

• https://tickets.puppetlabs.com/browse/PUP-731
• https://docs.puppet.com/puppet/latest/reference/environments_limitations.html
• https://tickets.puppetlabs.com/browse/SERVER-94

I am totally confused. I can not answer anymore if environments work, if my code in #226 fixes something and what the problem is. Is 4.6.2 the solution ? What about https://docs.puppet.com/puppet/latest/reference/man/generate.html ? .. Maximum confusion for everyone using environments.

My (internal) conclusion is: Puppet Environments are risky and do not not provide production level benefit / isolation (this is what they are used for typically) until SERVER-94 is fixed, which might never be fixed (as sandboxing is difficult).

However the Problem is: rtyler/jenkins switched from camptocamp/archive to puppet/archive from 1.6.1 to 1.7.0 which seem incompatible. However rtyler/jenkins 1.7.0 is needed for Jenkins2 which needed, as Jenkins 1.6 is EOL. Due to the environment bug, I can not stage the rollout by testing on dev and then merge to master. ... very disappointing for a tool with an enterprise behind (puppetlabs) ...

[alexjfisher]

I agree https://tickets.puppetlabs.com/browse/SERVER-94 is incredibly frustrating. Perhaps you can deploy a separate puppet master for dev/testing?