Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HPR-1260] Support Project Level Service Principals Auth with HCP Packer #12520

Merged
merged 2 commits into from
Jul 25, 2023

Conversation

nywilken
Copy link
Contributor

@nywilken nywilken commented Jul 19, 2023

HCP supports two types of service principles: Organization-level and project-level.
When a user tries to publish to an active HCP Packer registry using a plsp the client
fails when configuring the client due to a API permission error; namely plsp do not have
the permissions to query an org for a list of projects. Setting the HCP_PROJECT_ID does
not resolve the issue because the call to ListProjects is still executed.

This changes updates the client configuration params to obtain both the HCP Organization and
Project IDs that will be used for connecting to the HCP Packer registry. With this change
if a user provides a project Id via the HCP_PROJECT_ID environment variable no call to ListProjects will
be made. Instead the value will be take as is and used to create the connection. A user connecting with
a project level service principle must provide a valid HCP_PROJECT_ID in order to connect.


Example Outputs:

  1. Using a Project-level service principle (plsp) without setting HCP_PROJECT_ID
~>  HCP_PACKER_BUILD_FINGERPRINT=01H5QGVQ1CWKPQY3HHKE2DSMKF go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5QGVQ1CWKPQY3HHKE2DSMKF"
Error: HCP: populating iteration failed

Failed to create client connection to artifact registry: status 1: err unable to
fetch project

If the provided credentials are tied to a specific project trying setting the
HCP_PROJECT_ID environment variable to one you want to use.


exit status 1
  1. Using a Project-level service principle (plsp) and setting HCP_PROJECT_ID
~>  HCP_PACKER_BUILD_FINGERPRINT=01H5QGVQ1CWKPQY3HHKE2DSMKF HCP_PROJECT_ID=a847a3fd-4b02-4924-a895-e8fbf4eb4815 go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5QGVQ1CWKPQY3HHKE2DSMKF"
amazon-ebs.basic-example-east: output will be in this color.
amazon-ebs.basic-example-west: output will be in this color.

==> amazon-ebs.basic-example-east: Prevalidating any provided VPC information
==> amazon-ebs.basic-example-east: Prevalidating AMI Name: packer_AWS_1689785670_v1.0.0
==> amazon-ebs.basic-example-west: Prevalidating any provided VPC information
==> amazon-ebs.basic-example-west: Prevalidating AMI Name: packer_AWS_1689785670_v1.0.0
  1. Using plsp and HCP_PROJECT_ID but using a project that is not scoped to the plsp should fail.
~>  HCP_PROJECT_ID=10afc2bd-6fbc-4dbd-965f-24fc640622da go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5QHVSF0M7K8H28H5494KGJ6"
Error: HCP: populating iteration failed

failed to initialize bucket "learn-packer-ubuntu": [PUT
/packer/2021-04-30/organizations/{location.organization_id}/projects/{location.project_id}/images][403]
PackerService_CreateBucket default  &{Code:7 Details:[] Message:Not Authorized}
  1. Using Org-level service principle and setting HCP_PACKER_ID
~>  HCP_PACKER_BUILD_FINGERPRINT=01H5QGVQ1CWKPQY3HHKE2DSMKF HCP_PROJECT_ID=a847a3fd-4b02-4924-a895-e8fbf4eb4815 go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5QGVQ1CWKPQY3HHKE2DSMKF"
amazon-ebs.basic-example-east: output will be in this color.
amazon-ebs.basic-example-west: output will be in this color.

==> amazon-ebs.basic-example-east: Prevalidating any provided VPC information
==> amazon-ebs.basic-example-east: Prevalidating AMI Name: packer_AWS_1689785884_v1.0.0
==> amazon-ebs.basic-example-west: Prevalidating any provided VPC information
==> amazon-ebs.basic-example-west: Prevalidating AMI Name: packer_AWS_1689785884_v1.0.0
  1. Using Org-level service principle with no extra environment variables (default use case)
~>  go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5QHJ5DP483TDP86ZY73HHEE"
amazon-ebs.basic-example-east: output will be in this color.
amazon-ebs.basic-example-west: output will be in this color.
  1. Setting HCP_PROJECT_ID to an invalid UUID
~> HCP_PROJECT_ID=1847a3fd-4b02-4924-a895-e8fbf4eb4815 go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5QGVQ1CWKPQY3HHKE2DSMKF"
Error: HCP: populating iteration failed

Failed to create client connection to artifact registry: status 1: err project
validation for id "1847a3fd-4b02-4924-a895-e8fbf4eb4815" responded in error:
[GET
/packer/2021-04-30/organizations/{location.organization_id}/projects/{location.project_id}/registry][403]
PackerService_GetRegistry default  &{Code:7 Details:[] Message:Not Authorized}


exit status 1

~>  HCP_PROJECT_ID=1847a-4b02-4924-a895-e8fbf4eb4815 go run . build ../learn-hcp-packer-get-started
Tracking build on HCP Packer with fingerprint "01H5TECWG18QGAWAZ41YZ77T6K"
Error: HCP: populating iteration failed

Failed to create client connection to artifact registry: status 1: err project
validation for id "1847a-4b02-4924-a895-e8fbf4eb4815" responded in error: [GET
/packer/2021-04-30/organizations/{location.organization_id}/projects/{location.project_id}/registry][400]
PackerService_GetRegistry default  &{Code:3 Details:[0x14000576cc0]
Message:location: (project_id: must be a valid UUID.).}


exit status 1

@nywilken nywilken requested a review from a team as a code owner July 19, 2023 16:50
@nywilken nywilken changed the title nywilken/add hcp plsp support [HPR-1260] Support Project Level Service Principles Auth with HCP Packer Jul 19, 2023
Copy link
Contributor

@lbajolet-hashicorp lbajolet-hashicorp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I left a few comments on the code, but the logic seems sound, thanks for taking care of this!

internal/hcp/api/client.go Outdated Show resolved Hide resolved
internal/hcp/api/client.go Outdated Show resolved Hide resolved
internal/hcp/api/client.go Show resolved Hide resolved
internal/hcp/api/client.go Outdated Show resolved Hide resolved
internal/hcp/api/client.go Outdated Show resolved Hide resolved
@nywilken nywilken force-pushed the nywilken/add-hcp-plsp-support branch from 0728e49 to 15aef8d Compare July 20, 2023 20:07
@nywilken nywilken changed the title [HPR-1260] Support Project Level Service Principles Auth with HCP Packer [HPR-1260] Support Project Level Service Principals Auth with HCP Packer Jul 20, 2023
@nywilken nywilken added bug hcp backport/1.9.x Backport PR changes to `release/1.9.x` labels Jul 21, 2023
internal/hcp/api/client.go Outdated Show resolved Hide resolved
}
// ValidateRegistryForProject validates that there is an active registry associated to the configured organization and project ids.
// A successful validation will result in a nil response. All other response represent an invalid registry error request or a registry not found error.
func (client *Client) ValidateRegistryForProject() error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting and good idea!

Copy link
Contributor

@sylviamoss sylviamoss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼 Looks good to me!

Copy link
Contributor

@lbajolet-hashicorp lbajolet-hashicorp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a final question/nit, other than that the code looks good to me!

}
}

return client, nil
}

func (c *Client) loadOrganizationID() error {
if env.HasOrganizationID() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding this check, doesn't that conflict with the comment left on line 71?

if both HCP_ORGANIZATION_ID and HCP_PROJECT_ID are set via env variables the hcpConfig may have all we need already.

If the client's Profile has them set already via the environment variable, is there a reason why we check for this here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the initialization code done in Canonicalize, I don't see where the org/project ID are loaded from the environment, is it possible that the Profile.OrganizationID or Profile.ProjectID are never set beforehand? If this is so, the check on line 72 seems to be always false, is my assumption correct?

Copy link
Contributor Author

@nywilken nywilken Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey great question! This took me for a little ride at first, as well, but if you take a deep dive into the Canonicalize call there is logic that builds the default UserProfile (Profile()) using the FromEnv configuration options. FromEnv will check the the local environment for a number of environment variables and set each one if the criteria for loading the env. variable is met. For some variables the rule is "If set use it" for others like HCP_ORGANIZATION_ID and HCP_PROJECT_ID they both must be set in order for the SDK to load them from the env. variable.

The SDK does not validate the values like we are doing for the Project ID but if the SDK logic changes in the future to do some validation and our Packer user sets both values we would benefit from the SDK doing the lifting. This is why I added the logic to check if set use. I'm checking the loading of the SDK because it may help us in the future when it comes to handling these vars consistently between products.

If the client's Profile has them set already via the environment variable, is there a reason why we check for this here?

As for this question the check is a guard clause if they are both set then we return and we don't check if they are set again. If only one is set then we have to check which is set and validate both accordingly.

Packer does not need the Organization ID set via a env. variable at this time. But I don't see any harm in adding it. In fact when I first wrote this code Packer loaded both but it was removed in favor of ListProjects and ListOrganization. Given the recent changes in service principles and how the platform is evolving my thinking is we should support the four main env variables until we can fully rely on the SDK for them.

Please let me know if that clears things up for you.

I will update the comment in the code to call out that they are only set by the SDK if both Project ID and Organization ID are set.

HCP supports two types of service principals: Organization-level and project-level.
When a user tries to publish to an active HCP Packer registry using a plsp the client
fails when configuring the client due to a API permission error; namely plsp do not have
the permissions to query an org for a list of projects. Setting the HCP_PROJECT_ID does
not resolve the issue because the call to ListProjects is still executed.

This changes updates the client configuration params to obtain both the HCP Organization and
Project IDs that will be used for connecting to the HCP Packer registry. With this change
if a user provides a project Id via the HCP_PROJECT_ID environment variable no call to ListProjects will
be made. Instead the value will be take as is and used to create the connection. A user connecting with
a project level service principals must provide a valid HCP_PROJECT_ID in order to connect.
When setting a project id via the HCP_PROJECT_ID env the client will try to validate the project
by checking that it has an associated registry. If the project is invalid or not a valid UUID an error
will be displayed to the user

* Add comment to clarify usage of SDK loaded env. variables
@nywilken nywilken force-pushed the nywilken/add-hcp-plsp-support branch from 3d30330 to 777c816 Compare July 24, 2023 18:19
@nywilken nywilken merged commit 60c66fd into main Jul 25, 2023
9 checks passed
@nywilken nywilken deleted the nywilken/add-hcp-plsp-support branch July 25, 2023 15:31
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/1.9.x Backport PR changes to `release/1.9.x` bug hcp
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants