Packer no longer checks PACKER_HOME_DIR/plugins for plugin location

[mschuchard]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

As recently as version 1.7.10 of Packer, Packer would check the location PACKER_HOME_DIR/plugins for the location of a plugin and correctly utilize it. According to the plugin location documentation, this is priority three in reverse lookup order. With version 1.8.3, Packer will no longer check that location for plugins. I have tested and verified it will still check locations one and two for the location of a plugin. This makes plugin acceptance testing more difficult since we should not conflict with the proper installation namespaces maintained by packer init for a full release version.

If this is an intended change, then please communicate it so that plugin maintainers can update our acceptance tests to locally build the plugin in the same directory as path.root, but not locally install the built plugin into the PACKER_HOME_DIR/plugins directory. This would ensure Packer would accurately find the plugin since it would be at location two in the documentation. Thank you.

Reproduction Steps

Place a required plugin in the location PACKER_HOME_DIR/plugins and build a Packer template requiring the plugin.

Packer version

1.8.3

Simplified Packer Template

n/a

Operating system and Environment details

Linux home directory structure e.g. ~/.packer.d/plugins

Log Fragments and crash.log files

Output of logs from go test with PACKER_ACC=1:

2022/09/07 15:41:04 Current exe path: /usr/local/bin/packer
2022/09/07 15:41:04 [TRACE] Starting external plugin /usr/local/bin/NONEXISTENT
2022/09/07 15:41:04 Starting plugin: /usr/local/bin/NONEXISTENT []string{"/usr/local/bin/NONEXISTENT"}
2022/09/07 15:41:04 ui error: Error: failed loading PLUGIN
fork/exec /usr/local/bin/NONEXISTENT: no such file or directory

[nywilken]

Hi @mschuchard thanks for bubbling this up. This should still work as documented. We use the same mechanism for acceptance testing HashiCorp maintained plugins. Looking at the provided logs it appears that you may also have a plugin in /usr/local/bin/, which is in the same directory as the packer binary. This is priority number 1 in the lookup order.

If it is indeed true that there is a plugin /usr/local/bin/NONEXISTENT and that plugin fails to load I believe Packer will fail and exit entirely; thus stopping the process of looking for other plugins.

Have you tired removing /usr/local/bin/NONEXISTENT?

If possible could provide the full debug logs to help us better understand the the paths Packer is checking against. I look forward to hearing back.

Below is the output from my local test using the Packer nightly build, which has identical plugin loading code as 1.8.3

~>  tree ~/.packer.d/plugins
/Users/dev/.packer.d/plugins
└── packer-plugin-amazon

0 directories, 1 file
~>  PACKER_LOG=1 PACKER_LOG_PATH=out.log packer build amazon-ebs_amznlinxu_shell-local.pkr.hcl

2022/09/07 16:02:57 [INFO] Packer version: 1.8.4-dev [go1.18.5 darwin arm64]
2022/09/07 16:02:57 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:02:57 [TRACE] discovering plugins in /Users/dev/Development/go/bin
2022/09/07 16:02:57 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:02:57 [TRACE] discovering plugins in /Users/dev/.packer.d/plugins
2022/09/07 16:02:57 [DEBUG] Discovered plugin: amazon = /Users/dev/.packer.d/plugins/packer-plugin-amazon
2022/09/07 16:02:57 [INFO] found external [chroot ebs ebssurrogate ebsvolume instance] builders from amazon plugin
2022/09/07 16:02:57 [INFO] found external [import] post-processors from amazon plugin
2022/09/07 16:02:57 found external [ami parameterstore secretsmanager] datasource from amazon plugin
2022/09/07 16:02:57 [TRACE] discovering plugins in .
2022/09/07 16:02:57 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/09/07 16:02:57 [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/dev/.packerconfig
2022/09/07 16:02:57 [WARN] Config file doesn't exist: /Users/dev/.packerconfig
2022/09/07 16:02:57 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:02:57 [INFO] Setting cache directory: /Users/dev/.cache/packer
2022/09/07 16:02:57 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:02:57 [TRACE] validateValue: not active for vms_to_build, so skipping
2022/09/07 16:02:57 [TRACE] Starting external plugin /Users/dev/.packer.d/plugins/packer-plugin-amazon start builder ebs

I also tested with 1.8.3 to be sure

2022/09/07 16:14:23 [INFO] Packer version: 1.8.3 [go1.17.11 darwin arm64]
2022/09/07 16:14:23 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:14:23 [TRACE] discovering plugins in /opt/homebrew/bin
2022/09/07 16:14:23 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:14:23 [TRACE] discovering plugins in /Users/dev/.packer.d/plugins
2022/09/07 16:14:23 [DEBUG] Discovered plugin: amazon = /Users/dev/.packer.d/plugins/packer-plugin-amazon
2022/09/07 16:14:23 [INFO] found external [chroot ebs ebssurrogate ebsvolume instance] builders from amazon plugin
2022/09/07 16:14:23 [INFO] found external [import] post-processors from amazon plugin
2022/09/07 16:14:23 found external [ami parameterstore secretsmanager] datasource from amazon plugin
2022/09/07 16:14:23 [TRACE] discovering plugins in .
2022/09/07 16:14:23 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/09/07 16:14:23 [INFO] PACKER_CONFIG env var set; attempting to open config file: /Users/dev/.packerconfig
2022/09/07 16:14:23 [WARN] Config file doesn't exist: /Users/dev/.packerconfig
2022/09/07 16:14:23 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:14:23 [INFO] Setting cache directory: /Users/dev/.cache/packer
2022/09/07 16:14:23 Old default config directory found: /Users/dev/.packer.d
2022/09/07 16:14:23 [TRACE] validateValue: not active for vms_to_build, so skipping
2022/09/07 16:14:23 [TRACE] Starting external plugin /Users/dev/.packer.d/plugins/packer-plugin-amazon start builder ebs
2022/09/07 16:14:23 Starting plugin: /Users/dev/.packer.d/plugins/packer-plugin-amazon []string{"/Users/dev/.packer.d/plugins/packer-plugin-amazon", "start", "builder", "ebs"}

[mschuchard]

There are no Packer plugins installed in /usr/local/bin. This is why the go fork/exec returns the no such file or directory when it cannot find the plugin in the other three locations and then attempts to load a nonexistent plugin.

I can try to find some more information from the logs, but for loading a plugin from PACKER_HOME_DIR/plugins the literal difference is:

1.7.10 - loaded and works completely correct
1.8.3 - insists on attempting to load a nonexistent plugin in the same path as Packer and fails

I am not sure what exactly from the logs would be helpful to see. Please let me know.
My acceptance test framework is also very similar to packer-plugin-ansible/provisioner/ansible if that is helpful.

[mschuchard]

Ok I debugged a little more and have discovered that Packer 1.8.3 independently does locate plugins in PACKER_HOME_DIR/plugins. Packer 1.7.10 with Packer SDK 0.3.1 will also locate plugins in PACKER_HOME_DIR/plugins. Packer 1.8.3 with Packer SDK 0.3.1 does NOT locate plugins in PACKER_HOME_DIR/plugins.

So this is actually an incompatibility between a recent update to Packer and possibly a necessary corresponding update to the Packer SDK.

For now I can workaround by building the plugin and not locally installing it since it can still be located in cwd.

[mschuchard]

Should I migrate this issue to the packer-sdk issue tracker?

[mschuchard]

Additional information is that Packer also cannot find the plugins locally installed in ~/.config/packer/plugins. This seems to be the preferred path beginning in 1.8.0 according to Packer logging and default init behavior, but I also notice the documentation still references the old path.

[github-actions]

This issue has been synced to JIRA for planning.

JIRA ID: HPR-941

[nywilken]

@mschuchard thank you for all your work in tracking down the inconsistent behavior with the Plugin loading for Packer. TBH the change to the SDK to use the XDG directory spec was a bit of a surprise to me as Packer uses the older directory style ($HOME/.packer.d/). We've done some work to clean up the documentation in #12485 and we've fixed a bit of the plugin loading logic in #12414 that should fix the plugin loading order issues.

The changes are available within the latest nightly builds if you wish to pull it down and give it a test run. Once I merge #12485 this issue will be closed. But if you test and find issues please feel free to drop a comment on the thread and I will gladly reopen.

[mschuchard]

Oh sorry for only thumbs up and not replying with a thanks. I have been conditioned into believing all my issues in the Hashicorp repos are locked by the bot, and so I forgot I could reply. I know it is tough triaging these Packer issues with the current staffing, and so I appreciate the resolution.

I am still rolling with 1.8.7 and 0.4 at the moment, but will update the deps and take advantage of these changes once I return to Packer plugin development in my cycle of plugin development (re: Hashi stack recently transitioned from a Vault plugin to a Terraform plugin, and so will probably be able to revisit Packer in September).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.