Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ec2): 'encoded list token' error using Vpc imported from deploy-time lists #12040

Merged
merged 4 commits into from
Dec 14, 2020

Conversation

rix0rrr
Copy link
Contributor

@rix0rrr rix0rrr commented Dec 12, 2020

Even though using Vpc.fromVpcAttributes() using deploy-time lists like
from Fn.importValue()s and CfnParameters was never really supposed
to work, it accidentally did.

The reason for that is:

// Encoded list token
const subnetIds = Token.asList(Fn.importValue('someValue'))
// [ '#{Token[1234]}' ]

// Gets parsed to a singleton `Subnet` list:
const subnets = subnetIds.map(s => Subnet.fromSubnetAttributes({ subnetId: s }));
// [ Subnet({ subnetId: '#{Token[1234]}' }) ]

// This 'subnetId' is illegal by itself, and if yould try to use it for,
// say, an ec2.Instance it would fail. However, if you treat this single
// subnet as a GROUP of subnets:
new CfnAutoScalingGroup({ subnetIds: subnets.map(s => s.subnetId) })
// [ '#{Token[1234]}' ]

// And this resolves back to:
resolve(cfnSubnetIds)
// SubnetIds: { Fn::ImportValue: 'someValue' }

We introduced an additional check in #11899 to make sure that the
list-element token that represents an encoded list ('#{Token[1234]}')
never occurs in a non-list context, because it's illegal there.

However, because:

  • Subnet.fromSubnetAttributes() logs the subnetId as a warning
    to its own metadata (which will log a string like "there's something wrong with '#{Token[1234]}' ...").
  • All metadata is resolved just the same as the template expressions
    are.

The resolve() function encounters that orphaned list token in the
metadata and throws.


The proper solution would be to handle unparseable list tokens
specially to never get into this situation, but doing that requires
introducing classes and new concepts that will be a large effort and
not be backwards compatible. Tracking in #4118.

Another possible solution is to stop resolving metadata. I don't
know if we usefully use this feature; I think we don't. However,
we have tests enforcing that it is being done, and I'm scared
to break something there.

The quick solution around this for now is to have
Subnet.fromSubnetAttributes() recognize when it's about to log
a problematic identifier to metadata, and don't do it.

Fixes #11945.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…ime lists

Even though using `Vpc.fromVpcAttributes()` using deploy-time lists like
from `Fn.importValue()`s and `CfnParameters` was never really supposed
to work, it accidentally did.

The reason for that is:

```ts

// Encoded list token
const subnetIds = Token.asList(Fn.importValue('someValue'))
// [ '#{Token[1234]}' ]

// Gets parsed to a singleton `Subnet` list:
const subnets = subnetIds.map(s => Subnet.fromSubnetAttributes({ subnetId: s }));
// [ Subnet({ subnetId: '#{Token[1234]}' }) ]

// This 'subnetId' is illegal by itself, and if yould try to use it for,
// say, an ec2.Instance it would fail. However, if you treat this single
// subnet as a GROUP of subnets:
new CfnAutoScalingGroup({ subnetIds: subnets.map(s => s.subnetId) })
// [ '#{Token[1234]}' ]

// And this resolves back to:
resolve(cfnSubnetIds)
// SubnetIds: { Fn::ImportValue: 'someValue' }
```

--------

We introduced an additional check in #11899 to make sure that the
list-element token that represents an encoded list (`'#{Token[1234]}'`)
never occurs in a non-list context, because it's illegal there.

However, because:

* `Subnet.fromSubnetAttributes()` logs the subnetId as a *warning*
  to its own metadata (which will log a string like `"there's something
  wrong with '#{Token[1234]}' ..."`).
* All metadata is resolved just the same as the template expressions
  are.

The `resolve()` function encounters that orphaned list token in the
metadata and throws.

--------

The *proper* solution would be to handle unparseable list tokens
specially to never get into this situation, but doing that requires
introducing classes and new concepts that will be a large effort and
not be backwards compatible. Tracking in #4118.

Another possible solution is to stop resolving metadata. I don't
know if we usefully use this feature; I think we don't. However,
we have tests enforcing that it is being done, and I'm scared
to break something there.

The quick solution around this for now is to have
`Subnet.fromSubnetAttributes()` recognize when it's about to log
a problematic identifier to metadata, and don't do it.

Fixes #11945.
@rix0rrr rix0rrr requested a review from a team December 12, 2020 10:19
@rix0rrr rix0rrr self-assigned this Dec 12, 2020
@gitpod-io
Copy link

gitpod-io bot commented Dec 12, 2020

@github-actions github-actions bot added the @aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud label Dec 12, 2020
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Dec 12, 2020
@rix0rrr rix0rrr requested a review from eladb December 12, 2020 10:20
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: e1b27b4
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify
Copy link
Contributor

mergify bot commented Dec 14, 2020

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 0690da9 into master Dec 14, 2020
@mergify mergify bot deleted the huijbers/vpc-token-import branch December 14, 2020 09:07
flochaz pushed a commit to flochaz/aws-cdk that referenced this pull request Jan 5, 2021
…ime lists (aws#12040)

Even though using `Vpc.fromVpcAttributes()` using deploy-time lists like
from `Fn.importValue()`s and `CfnParameters` was never really supposed
to work, it accidentally did.

The reason for that is:

```ts

// Encoded list token
const subnetIds = Token.asList(Fn.importValue('someValue'))
// [ '#{Token[1234]}' ]

// Gets parsed to a singleton `Subnet` list:
const subnets = subnetIds.map(s => Subnet.fromSubnetAttributes({ subnetId: s }));
// [ Subnet({ subnetId: '#{Token[1234]}' }) ]

// This 'subnetId' is illegal by itself, and if yould try to use it for,
// say, an ec2.Instance it would fail. However, if you treat this single
// subnet as a GROUP of subnets:
new CfnAutoScalingGroup({ subnetIds: subnets.map(s => s.subnetId) })
// [ '#{Token[1234]}' ]

// And this resolves back to:
resolve(cfnSubnetIds)
// SubnetIds: { Fn::ImportValue: 'someValue' }
```

--------

We introduced an additional check in aws#11899 to make sure that the
list-element token that represents an encoded list (`'#{Token[1234]}'`)
never occurs in a non-list context, because it's illegal there.

However, because:

* `Subnet.fromSubnetAttributes()` logs the subnetId as a *warning*
  to its own metadata (which will log a string like `"there's something
  wrong with '#{Token[1234]}' ..."`).
* All metadata is resolved just the same as the template expressions
  are.

The `resolve()` function encounters that orphaned list token in the
metadata and throws.

--------

The *proper* solution would be to handle unparseable list tokens
specially to never get into this situation, but doing that requires
introducing classes and new concepts that will be a large effort and
not be backwards compatible. Tracking in aws#4118.

Another possible solution is to stop resolving metadata. I don't
know if we usefully use this feature; I think we don't. However,
we have tests enforcing that it is being done, and I'm scared
to break something there.

The quick solution around this for now is to have
`Subnet.fromSubnetAttributes()` recognize when it's about to log
a problematic identifier to metadata, and don't do it.

Fixes aws#11945.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
rix0rrr added a commit that referenced this pull request Jan 18, 2021
This PR has a fix similar to #12040, and introduces more explanations
and safeguards around the mechanism of importing a VPC using
`fromVpcAttributes()`. This is not the recommended way of importing
VPCs, but many users really don't want to use lookups so we'd better
make it a little safer.

This PR contains:

* A fix in the EKS library to have it stop logging subnet IDs to
  metadata if it looks like the subnet ID will lead to the
  aforementioned synthesis error (similar to the fix in the linked
  PR).
* A validation in the EKS library to stop a similar-but-different
  error from occurring if people select multiple VPC subnet groups
  from a VPC imported from token lists; this can never work and we
  might as well tell them directly.
* A metadata warning added to VPCs imported using
  `Vpc.fromVpcAttributes()`, to inform users that their VPC imported
  in this way has a good chance of not working in all cases they
  expect it to.
* A mechanism to specify the length of deploy-time lists at synthesis
  time, by passing an `assumedLength` parameter to `Fn.split()`. This
  will produce a list that is safe for manipulation.
* A note on how to use `Vpc.fromVpcAttributes()` in the `ec2` README.

Fixes #12160.
mergify bot pushed a commit that referenced this pull request Jan 19, 2021
This PR has a fix similar to #12040, and introduces more explanations
and safeguards around the mechanism of importing a VPC using
`fromVpcAttributes()`. This is not the recommended way of importing
VPCs, but many users really don't want to use lookups so we'd better
make it a little safer.

This PR contains:

* A fix in the EKS library to have it stop logging subnet IDs to
  metadata if it looks like the subnet ID will lead to the
  aforementioned synthesis error (similar to the fix in the linked
  PR).
* A validation in the EKS library to stop a similar-but-different
  error from occurring if people select multiple VPC subnet groups
  from a VPC imported from token lists; this can never work and we
  might as well tell them directly.
* A metadata warning added to VPCs imported using
  `Vpc.fromVpcAttributes()`, to inform users that their VPC imported
  in this way has a good chance of not working in all cases they
  expect it to.
* A mechanism to specify the length of deploy-time lists at synthesis
  time, by passing an `assumedLength` parameter to `Fn.split()`. This
  will produce a list that is safe for manipulation, and removes the limitations
  addressed by the previous bullets.
* A note on how to use `Vpc.fromVpcAttributes()` in the `ec2` README.

Fixes #12160.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
mohanrajendran pushed a commit to mohanrajendran/aws-cdk that referenced this pull request Jan 24, 2021
This PR has a fix similar to aws#12040, and introduces more explanations
and safeguards around the mechanism of importing a VPC using
`fromVpcAttributes()`. This is not the recommended way of importing
VPCs, but many users really don't want to use lookups so we'd better
make it a little safer.

This PR contains:

* A fix in the EKS library to have it stop logging subnet IDs to
  metadata if it looks like the subnet ID will lead to the
  aforementioned synthesis error (similar to the fix in the linked
  PR).
* A validation in the EKS library to stop a similar-but-different
  error from occurring if people select multiple VPC subnet groups
  from a VPC imported from token lists; this can never work and we
  might as well tell them directly.
* A metadata warning added to VPCs imported using
  `Vpc.fromVpcAttributes()`, to inform users that their VPC imported
  in this way has a good chance of not working in all cases they
  expect it to.
* A mechanism to specify the length of deploy-time lists at synthesis
  time, by passing an `assumedLength` parameter to `Fn.split()`. This
  will produce a list that is safe for manipulation, and removes the limitations
  addressed by the previous bullets.
* A note on how to use `Vpc.fromVpcAttributes()` in the `ec2` README.

Fixes aws#12160.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(core): Vpc.fromVpcAttributes using list tokens now throws errors
3 participants