Can't use third party providers with a backend?

[Krenair]

Hi,

I was trying to use this with https://github.com/atlassian/terraform-provider-artifactory/ - I built it myself and placed it in a directory under my terraform_source dir called terraform.d/plugins/linux_amd64
I'm using the S3 backend with a role_arn
I was able to plan/apply with the terraform-resource, but it fails at the implicit get stage when trying to terraform init:

fetching ljfranklin/terraform-resource@sha256:d63ba8bb3262392adfa3e68dad9b94584bcb4ec74095392f51a3a71543afa33a

c9b1b535fdd9 [==========================================] 2.7MiB/2.7MiB

45bf92bbfa9b [========================================] 12.1MiB/12.1MiB

cb7d27f29af6 [==============================================] 115b/115b

eb90ce17dcdf [==============================================] 185b/185b

8cd304396fa9 [==============================================] 200b/200b

0b9bd7b7539f [==============================================] 200b/200b

1f4bb64ae182 [========================================] 11.4MiB/11.4MiB

ef1f64844071 [========================================] 15.9MiB/15.9MiB

e95857abebe9 [========================================] 16.3MiB/16.3MiB

2020/03/06 11:40:45 terraform init command failed.

Error: exit status 1

Output: 

Initializing the backend...


Successfully configured the backend "s3"! Terraform will automatically

use this backend unless the backend configuration changes.


Initializing provider plugins...

- Checking for available provider plugins...


Provider "artifactory" not available for installation.


A provider named "artifactory" could not be found in the Terraform Registry.


This may result from mistyping the provider name, or the given provider may

be a third-party provider that cannot be installed automatically.


In the latter case, the plugin must be installed manually by locating and

downloading a suitable distribution package and placing the plugin's executable

file in the following directory:

    terraform.d/plugins/linux_amd64


Terraform detects necessary plugins by inspecting the configuration and state.

To view the provider versions requested by each module, run

"terraform providers".



Error: no provider exists with the given name

This can't work as it won't have my terraform_source with the provider binary.

[Krenair]

I have resorted to building support for the provider into the resource image:

FROM golang:1.14 as builder
WORKDIR /workspace
ENV CGO_ENABLED="0"
RUN git clone https://github.com/atlassian/terraform-provider-artifactory.git \
    && cd terraform-provider-artifactory \
    && go build

# Terraform 0.12.22
FROM ljfranklin/terraform-resource@sha256:d63ba8bb3262392adfa3e68dad9b94584bcb4ec74095392f51a3a71543afa33a
COPY --from=builder /workspace/terraform-provider-artifactory/terraform-provider-artifactory /usr/local/bin/terraform-provider-artifactory

[ljfranklin]

@Krenair I have a test here which exercises the custom plugin flow. I just updated that test to use the new backend syntax and it seems to be working as expected.

I remember Terraform being very picky about the directory and file names in the plugin directory. Was your filename exactly terraform.d/plugins/linux_amd64/terraform-provider-artifactory_v1.6.3? To further debug the issue I would try replicating your setup locally and use terraform init --plugin-dir your-plugin-dir to figure out whether Terraform is able to recognize your plugins correctly.

[Krenair]

No, the file's name was terraform.d/plugins/linux_amd64/terraform-provider-artifactory - but in any case, this was inside my terraform_source directory, how would a concourse get (which doesn't have a parameter to tell it where to find the source) be able to find that?

…

On Sat, 7 Mar 2020, 12:56 Lyle Franklin, ***@***.***> wrote: @Krenair <https://github.com/Krenair> I have a test here <https://github.com/ljfranklin/terraform-resource/blob/39f3825b2713ab4f63382967699674275ca5c72c/src/terraform-resource/out/out_custom_plugins_test.go#L234> which exercises the custom plugin flow. I just updated that test to use the new backend syntax and it seems to be working as expected. I remember Terraform being very picky about the directory and file names in the plugin directory. Was your filename exactly terraform.d/plugins/linux_amd64/terraform-provider-artifactory_v1.6.3? To further debug the issue I would try replicating your setup locally and use terraform init --plugin-dir your-plugin-dir to figure out whether Terraform is able to recognize your plugins correctly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#107?email_source=notifications&email_token=AAGJ2JQPLWNXKVRFFQZFYELRGI77DA5CNFSM4LC6S6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEODYWOQ#issuecomment-596085562>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGJ2JX6GEOTYGWCQZ5OCH3RGI77DANCNFSM4LC6S6UQ> .

[ljfranklin]

@Krenair oh, I missed the part about this being on the get step. I didn't realize Terraform had this behavior, but it looks when you run terraform init with backends it also tries to look up plugins to download from the default workspace. So I'm guessing at some point you've ran an apply against the default workspace and Terraform is trying to be overly helpful by downloading the artifactory plugin that the default workspace uses.

I'll do some thinking about how to fix this going forward. In the meantime you can work around the issue by baking the plugin into a custom docker image as you've done or by deleting or moving the statefile out of the default workspace.

[ljfranklin]

I think the fix for the resource will be to add -get-plugins=false to the init call for the get and check steps. This wouldn't work if you had a backend that was declared in a plugin, but it looks like support for plugable backends isn't happening soon so I'm fine with this fix for now.

[Krenair]

Yeah that sounds okay, alternatively we could make it possible to provide a source directory for plugins when getting?

…

On Sat, 7 Mar 2020, 14:30 Lyle Franklin, ***@***.***> wrote: I think the fix for the resource will be to add -get-plugins=false to the init call for the get and check steps. This wouldn't work if you had a backend that was declared in a plugin, but it looks like support for plugable backends <hashicorp/terraform#5877> isn't happening soon so I'm fine with this fix for now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#107?email_source=notifications&email_token=AAGJ2JQR4VJQ4MUOVLFFRQTRGJLARA5CNFSM4LC6S6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOD2ZVI#issuecomment-596094165>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGJ2JRIKXRJVUGGMQI46O3RGJLARANCNFSM4LC6S6UQ> .

[ljfranklin]

Afraid not, the Concourse design doesn't allow get steps to receive any directories as input: https://concourse-ci.org/implementing-resource-types.html#resource-in

[ljfranklin]

@Krenair I think this change should fix the issue you were having. That commit is now present on the ljfranklin/terraform-resource:latest and ljfranklin/terraform-resource:0.12.23 images, when you get a chance can you check if that fixes the error you were seeing?

[Krenair]

Seems to have done the trick, thanks.