[BUG] There seems to be an issue in how Runway evalutes changes.

[BradBloomingdale]

Bug Description

I've run into two separate incidents this week with Runway (with different customers) where the plan output makes no sense.

Scenario 1 - despite reconfiguring multiple stacks from using blueprints to CFN templates, Runway detects no changes.

I switched from class_path to template_path but Runway thinks nothing changed. In one update, that was the only change. In another, we changed the parameters as well and got the same basic output -- nochange.

The workaround was to arbitrarily change values in .env files to be able to make updates.

[runway] Oregon (us-west-2):processing deployment (in progress)
[runway] Oregon (us-west-2):processing regions sequentially...
[runway]
[runway] Oregon (us-west-2).ecr:processing module in us-west-2 (in progress)
[runway] found environment file: /Users/brad/sinfrastructure/ecr.cfn/common-us-west-2.env
[runway] ecr.yaml:init (in progress)
[runway] cfngin_bucket stacker-tmdx-common already exists
[runway] ecr.yaml:init (complete)
[runway] ecr.yaml:deploy (in progress)
[runway] mdmecr:skipped (nochange)
[runway] cmsecr:skipped (nochange)
[runway] ecr.yaml:deploy (complete)
[runway] Oregon (us-west-2).ecr:processing module in us-west-2 (complete)
[runway] Oregon (us-west-2):processing deployment (complete)

Scenario 2 - plan output suggests nochange, but then the deploy triggers updates.

The plan output:

[runway] diffing stacks: GatewayASG
[runway] ${rxref vpc::VPC}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref gateway-alb::targetgroupARN}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref EC2SecurityGroup::EC2SecurityGroupID}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref vpc::PriSubnetAz1}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref vpc::PriSubnetAz2}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref vpc::PriSubnetAz3}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] dev-GatewayASG:no changes
[runway] GatewayASG:complete

The subsequent deployment:

[runway] 02-ec2-asg.yaml:deploy (in progress)
[runway] ${rxref vpc::VPC}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref gateway-alb::targetgroupARN}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref EC2SecurityGroup::EC2SecurityGroupID}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref vpc::PriSubnetAz1}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref vpc::PriSubnetAz2}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] ${rxref vpc::PriSubnetAz3}: lookup query syntax "<relative-stack-name>::<OutputName>" has been deprecated; to learn how to use the new lookup query syntax visit https://runway.readthedocs.io/page/cfngin/lookups/rxref.html
[runway] GatewayASG:submitted (updating existing stack)
[runway] GatewayASG:complete (updating existing stack)
[runway] 02-ec2-asg.yaml:deploy (complete)

Expected Behavior

I would expect "nochange" to accurately indicate no changes have been detected and will not occur.

Steps To Reproduce

Scenario 1:
Migrating from blueprints to YAML CFN templates

Scenario 2:
Not quite sure. The intended update wasn't for the resource that was updated.

Runway version

2.x

Installation Type

pypi (pip, pipenv, poetry, etc)

OS / Environment

Multiple:

Scenario 1 is being run locally on MacOS Sonoma with Python3.10. - 2.6.14
Scenario 2 is running in a CodeBuild project orchestrated by CodePipeline: Amazon Linux 2 with Python 3.9 - 2.x (version unknown).

Anything else?

No response

[ITProKyle]

Scenario 1

I switched from class_path to template_path but Runway thinks nothing changed.

Runway's CFNgin relies on CloudFormation to detect change by submitting the CloudFormation Template to CloudFormation. When making the change from a Blueprint to a Template (or vice versa), if <stack>:skipped (nochange) was logged, the two are written identically and CloudFormation determined that the stack has no change.

If you don't believe this is the case, can you try setting environment variable DEBUG=2 (e.g. DEBUG=2 runway <action>) when running Runway? This will display all of the botocore calls being made and should show if/when calls are being made to CloudFormation to determine if there was change.

If those calls are missing, can you please provide a repo with minimal code required to reproduce the issue?

Scenario 2

Similar to CFNgin deploys detecting change with CloudFormation, using plan with CFNgin also uses CloudFormation to detect change. This is done via CloudFormation Change Set. If the Change Set does not detect change, CFNgin will report the same. There are things that a Change Set won't show but that will still put a Stack into an "Updating" state. Primarily this is Stack metadata (tags on the Stack itself, Stack description, Stack parameter descriptions, etc) which have no real impact to the infrastructure managed by the Stack. This metadata is updated with the same CloudFormation.Client.update_stack call that is used to update the Stack's Template.