Pulling plugins from domains not github.com

[johnypony3]

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

Allow pulling plugins from other vcs sites, in my case gitlab.

Use Case(s)

github is external to my build server and is therefore against company standards for use as an external dependency location

Potential configuration

packer {
  required_plugins {
    windows-update = {
      version = "0.12.0"
      source = "gitlab.company.com/mercedes/windows-update"
    }
  }
}

Potential References

This is documented as an eventual here: plugins source-addresses

[SwampDragons]

Hi, thanks for reaching out. This is definitely on our roadmap for something we want to do in the future, hence the "eventual" comment.

As a workaround, you can manually download plugins and place them into the plugins directory on your build server in order to make them discoverable by Packer without being downloaded. Full details can be found here: https://www.packer.io/docs/plugins (click the "manual (multi-component plugin)" tab)

[tomsiewert]

For us, we sometimes hit the GitHub rate limit on our CI runners which made us require to create mirrors of the used plugins. Those mirrors are located in Sub-Groups in our internal GitLab, but Packer force-validates / requires the source in a hostname/namespace/name format.

Error: Invalid plugin source string

  on debian/config.pkr.hcl line 7, in packer:
   7:     ansible = {
   8:       version = ">= 1.0.0"
   9:       source  = "git.company.com/mirror/github.com/hashicorp/ansible"
  10:     }

The "source" attribute must be in the format "hostname/namespace/name"

[azr]

Hello there, thanks for opening, Packer is meant to support other plugin getters than GitHub ! We just did not have the time to do it due to other priorities. If you would like to do that, I will try to give code pointers though, we would be more than happy to review PRs on this !

First, you will have to implement a new plugin getter, the GitHub one is being used here:

packer/command/init.go

Lines 85 to 86 in 0e3fcb5

	getters := []plugingetter.Getter{
	&github.Getter{

and being implemented in here:

packer/packer/plugin-getter/github/getter.go

Lines 29 to 32 in f48583c

	type Getter struct {
	Client *github.Client
	UserAgent string
	}

The getter interface is defined here, I'd recommend reading it first :) :

packer/packer/plugin-getter/plugins.go

Lines 274 to 323 in 54a4f59

	// A Getter helps get the appropriate files to download a binary.
	type Getter interface {
	// Get allows Packer to know more information about releases of a plugin in
	// order to decide which version to install. Get behaves similarly to an
	// HTTP server. Packer will stream responses from get in order to do what's
	// needed. In order to minimize the amount of requests done, Packer is
	// strict on filenames and we highly recommend on automating releases.
	// In the future, Packer will make it possible to ship plugin getters as
	// binaries this is why Packer streams from the output of get, which will
	// then be a command.
	//
	// * 'releases', get 'releases' should return the complete list of Releases
	// in JSON format following the format of the Release struct. It is also
	// possible to read GetOptions to filter for a smaller response. Some
	// getters don't. Packer will then decide the highest compatible
	// version of the plugin to install by using the sha256 function.
	//
	// * 'sha256', get 'sha256' should return a SHA256SUMS txt file. It will be
	// called with the highest possible & user allowed version from get
	// 'releases'. Packer will check if the release has a binary matching what
	// Packer can install and use. If so, get 'binary' will be called;
	// otherwise, lower versions will be checked.
	// For version 1.0.0 of the 'hashicorp/amazon' builder, the GitHub getter
	// will fetch the following URL:
	// https://github.com/hashicorp/packer-plugin-amazon/releases/download/v1.0.0/packer-plugin-amazon_v1.0.0_SHA256SUMS
	// This URL can be parameterized to the following one:
	// https://github.com/{plugin.path}/releases/download/{plugin.version}/packer-plugin-{plugin.name}_{plugin.version}_SHA256SUMS
	// If Packer is running on Linux AMD 64, then Packer will check for the
	// existence of a packer-plugin-amazon_v1.0.0_x5.0_linux_amd64 checksum in
	// that file. This filename can be parameterized to the following one:
	// packer-plugin-{plugin.name}_{plugin.version}_x{proto_ver.major}.{proto_ver._minor}_{os}_{arch}
	//
	// See
	// https://github.com/hashicorp/packer-plugin-scaffolding/blob/main/.goreleaser.yml
	// and
	// https://www.packer.io/docs/plugins/creation#plugin-development-basics
	// to learn how to create and automate your releases and for docs on
	// plugin development basics.
	//
	// * get 'zip' is called once we know what version we want and that it is
	// compatible with the OS and Packer. Zip expects an io stream of a zip
	// file containing a binary. For version 1.0.0 of the 'hashicorp/amazon'
	// builder and on darwin_amd64, the GitHub getter will fetch the
	// following ZIP:
	// https://github.com/hashicorp/packer-plugin-amazon/releases/download/v1.0.0/packer-plugin-amazon_v1.0.0_x5.0_darwin_amd64.zip
	// this zip is expected to contain a
	// packer-plugin-amazon_v1.0.0_x5.0_linux_amd64 file that will be checksum
	// verified then copied to the correct plugin location.
	Get(what string, opts GetOptions) (io.ReadCloser, error)
	}

[ianblackshere]

Hello, I am interested in working this issue. My team has a use-case for this where we would like to build on our private networks without having to copy the plugins to each host as we are spread across multiple sites.

[johnypony3]

@ianblackshere we got around this by pulling the packages in and pulling them from local.