Add support for protobuf serialization for plugin communication #13120
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
cf5cf79
to
1fb5b20
Compare
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
2 times, most recently
from
e3a2f63
to
c567587
Compare
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
f668d07
to
9ff6fe7
Compare
As we're trying to move away from gob for serialising data over the wire, this commit adds the capability for Packer to pick dynamically between gob or protobuf for the serialisation format to communicate with plugins. As it stands, if all the plugins discovered are compatible with protobuf, and we have not forced gob usage, protobuf will be the serialisation format picked. If any plugin is not compatible with protobuf, gob will be used for communicating with all the plugins that will be used over the course of a command.
When building a plugin, we may want some customisation capabilities beyond changing the version/pre-release/metadata, and instead run commands or change files on the filesystem. To do so, we introduce functions under the BuildCustomisation type, which have two responsabilities: changing the current state of the plugin's directory, and cleaning up afterwards. These customisations are passed as parameters to the BuildSimplePlugin function, and are called one-by-one, deferring their cleanup after the build process is finished. A first implementation of such a customisation is added with this commit, in order to change the version of a module that the plugin depends on, which we'll use to change the version of the plugin SDK in order to test how Packer behaves with different versions of the SDK for a single plugin.
Compiling plugins was originally intended to be an idempotent operation. This however starts to change as we introduce build customisations, which have the unfortunate side-effect of changing the state of the plugin directory, leading to conflicts between concurrent compilation jobs. Therefore to mitigate this problem, this commit changes how compilation jobs are processed, by introducing a global compilation queue, and processing plugins' compilation one-by-one from this queue. This however makes such requests asynchronous, so test suites that require plugins to be compiled will now have to wait on their completion before they can start their tests. To this effect, we introduce one more convenience function that processes those errors, and automatically fails the test should one compilation job fail for any reason.
With the draft to support both gob and protobuf as serialisation formats for Packer, along with the SDK changes that propel them, we add a series of tests that make sure the logic that picks which protocol is solid and functional. These tests rely on building several versions of the tester plugin, with and without protobuf support, to then install them in the tests as needed to test the logic of Packer using packer build with them, and templates that require multiple plugins.
lbajolet-hashicorp
force-pushed
the
feature/protobuf-serialization
branch
from
57739d9
to
7bb4c19
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In HashiCorp Packer 1.12, we are working to remove Packer's reliance on gob encoding for its plugin system architecture. We will introduce protos and custom HCL ObjectSpec wrappers for sending ObjectSpecs types over the wire using protobufs.
This work will look similar to the work already done for Nomad. The first implementation will use an environment variable to support the protocol switching server for the existing net/RPC implementation. Users of a plugin via the environment can toggle net/rpc with gob or net/rpc with protobuf communication for any supported Platform. Since Packer can not mix and match serialization formats at runtime, Packer will determine the supported formats by each plugin and use the appropriate wire protocol. If all plugins support the new serialization, protobuf/msgpack will be selected as the wire protocol. If the most compatible format is gob then the old serialization protocol will be used. Users wishing to force the old serialization for compatibility reasons can specify the PACKER_FORCE_GOB environment variable to turn off protobuf and msgpack serialization detection.