sam deploy doesn't update params in the stack when modified in the CF template

[insane-dreamer]

Description:

Making a change to a default parameter in a CF template, and/or to a parameter that is passed to a child stack described in a CF template, does not register when running sam build/deploy. Since it doesn't create/modify a resource, sam seems to think there has been no change. This makes it extremely difficult to keep a stack in sync with its CF template, which is especially problematic when you have multiple stacks (in the same region or different regions) who have to stay in sync with the latest changes (which is achieved by sharing the same platform.yaml).

Steps to reproduce:

1. Deploy stack with parameters. Deploy.
2. Make a change to the parameter default value in platform.yaml.
3. Deploy.

Observed result:

Check the value of the parameter in the Stack via the Console: value has not changed
(If the parameter was passed to a child stack, the child stack's value will also remain unchanged.)

Expected result:

The param value in the Stack should be updated to match the updated value in platform.yaml. If the param was passed to a child stack defined in platform.yaml, that child stack should also be updated with the new value.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 22.04
2. sam --version: 1.45.0
3. AWS region: any

Add --debug flag to command you are running

[moelasmar]

Thanks @insane-dreamer for raising this issue. I want to double check couple of points:

• after you change the default value of the parameter, how do you execute the sam deploy command. Do you still pass the parameter values in the command?
• Could you please check if the parameter values are stored in the sam config toml file or not?

[insane-dreamer]

@moelasmar Thanks. We do not pass in the new values as --parameter-overrides or in the samconfig.toml file when redeploying the stack. We could do that of course, and it would resolve the issue, but it somewhat defeats the purpose of CF templates:

• a --parameter-overrides is particularly problematic as that would leave no record of the latest pushed state of the template is.
• adding it to the samconfig.toml file, and committing that to a repo, is not as bad, but we over time wind up with two files, one of which overrides the other, again makes it difficult to clearly ascertain the state of any stack and to ensure that other stacks match. The way we use samconfig.toml is to only include settings that may differ between stacks (i.e., region, cluster size, etc.)

The beauty (and promise) of cloudformation is the ability to have versioned templates that allow us to ensure that stacks remain in sync and to know the values of a stack at a given time. (Stack drift in the console is ok for adhoc verification, but not particularly useful for this case). If one however has to have additional settings outside the template, then that defeats the purpose of a versioned template providing "ground truth". As we've moved away from OpsWorks to use more serverless services we've become more reliant on Cloudformation in this regard.

A side issue but related to this discussion is there is no way to diff a template with a stacks' actual state, and therefore no way to know with 100% certainty, that a particular template, or even template + samconfig.toml, matches the current state of a deployed stack. A sam diff command that does this would be extremely useful.

[moelasmar]

Thanks @insane-dreamer for your detailed response. I just want to confirm that you do not save the parameters values in the toml file. sam deploy command will save these values in the toml file, and reuse it during the template deployment, and so the parameters default values will not be used

[insane-dreamer]

@moelasmar That's correct. We understand that the .toml file will override the default values in the template (i.e., platform.yaml) and we do use the .toml for select parameters that are expected to change from time to time (i.e., an ASG). But we don't want the .toml to become cluttered with settings that affect all stacks and change infrequently (i.e., a new semi-permanent change to a stack, say a new CodeStar Connection) and which would require careful syncing between all stacks (in addition to the platform.yaml template).

There is a workaround workflow, and that is, if desiring to change a default value in a stack, to change it both in the CF template AND add it to the .toml file, then sam deploy, then remove it from the .toml file (since it's no longer needed as an override once it's changed in the stack, and any new stack would use the default value in the template). But this is both clunky and cumbersome to replicate across many stacks.

[qingchm]

@insane-dreamer Hi there since updates on parameters on stacks will create an empty changeset, and deploy is based on changeset execution logic, I don't think any parameter-only updates are possible. Same for CFN console, if you edit a deployed stack to only edit the pararmeters and try to re-deploy, it would exit saying that there's nothing to be deployed. If I'm understanding correctly you want us to support a feature that updates stacks with parameter updates only. Any suggestions to how this can be achieved is welcomed as inputs to this discussion!

[dgard1981]

@insane-dreamer, @qingchm, I opened a similar issue last year - #4180 - and the conclusion there was that (to paraphrase) SAM makes the use of CloudFormation defaults basically poiontless.

As an example, when I make the following changes, the new CloudFormation default value is ignored. Note that the changes are not parameter-only.

• Make a code change that resulted in replacement of an AWS::Lambda::LayerVersion resource.
• Update the default value of a parameter in my SAM template.yaml file.

  Parameters:
    LogLevel:
      Description: The log level to apply to the application.
      Type: String
      AllowedValues: [ DEBUG, INFO, WARNING, ERROR, CRITICAL ]
  -   Default: WARNING
  +   Default: INFO

Now when I run sam deploy and inspect the change set that is created, I can see exactly what I want to happen is planned.

• A new Lambda Layer will be created due to my code change.
  
• The new default value has been added to the template.

  Parameters:
    LogLevel:
      Description: The log level to apply to the application.
      Type: String
      AllowedValues:
      - DEBUG
      - INFO
      - WARNING
      - ERROR
      - CRITICAL
      Default: INFO

However, when I check the change set Input, the parameter value is still the same as the original default value (WARNING).

Note that there are no parameter_overrides defined in samconfig.yaml, meaning the input of WARNING is being set not by the user, but by some opaque SAM operation, meaning it is neigh on impossible to predict input values when using a SAM template in conjunction with CloudFormation parameters that have default values.

Therefore I don't believe that this issue is to do with CloudFormation limitations, but rather opaque SAM operations, and it's very frustrating that it exists. While it is possible to add every single parameter to samconfig.yaml, it's very time consuming to manage (particularly with multiple environments), prone to human error, and really negates the point of default values.

[insane-dreamer]

If I'm understanding correctly you want us to support a feature that updates stacks with parameter updates only.

@qingchm That's correct.

Ultimately, the problems that we're hoping to solve with SAM/CF are the following:

1. Be certain that a local CF template, which may be combined with a few .toml overrides, 100% match the state of an AWS Stack.
2. If we make a change to a default setting in a CF template, and that CF template is used on multiple Stacks (i.e., we have the same stack deployed in different regions and/or for different stages (dev/staging/prod, etc.), we can update all stacks by simply running sam deploy of the modified .yaml template to those Stacks.

Currently, neither of these are possible, because as you pointed out, some changes (like parameters) don't generate a Changeset, and therefore nothing happens when you deploy them. This leaves us with a degree of uncertainty about the exact state of our stacks and whether they are all exactly in sync (except for those settings in which they explicitly differ, i.e., the region, which would be listed in separate .toml files).

So perhaps the fix is to ensure that any change to a CF template, including a parameter change, generates a CF Changeset.

[Deanfei]

Got the same issue. Very frustrating. Same thing happens for the api gateway. Becoming less and less confidence of SAM for the certainty of deployment.

[qingchm]

@insane-dreamer Thanks for the detailed response! I understand that this is frustrating, and I believe that this is a limitation of the CFN changeset itself, a similar issue is described on here: https://repost.aws/questions/QUSvocJcK-SA6rNu7QOydbIA/questions/QUSvocJcK-SA6rNu7QOydbIA/aws-cloudformation-deploy-doesn-t-recognize-change-in-default-value?, so in order to be smart enough for SAM to update stack on parameter changes, a possible solution is to add the use of update_stack calls when we detect parameter changes. To verify the statement you can try to use a sam sync command and see if you get parameter updates as expected (we don't recommend on using this to deploy production stacks, but this command uses update_stack calls instead of create_change_set calls). Let me know if you like the solution, if this is feasible I can discuss with team on improving this limitation. @Deanfei Also would like to hear from you if sam sync suits your development needs!

[LorneCurrie]

I would like to add to the ticket, that we have also picked up on this. I am using parameters to record the time stamp that the template was built at so that we can force a Custom Resource Lambda to trigger with each build so that we can run some tasks in the background that the updated deployment requires. I admit that I should just use an Event Bridge to trigger the lambda, but this would be expected behaviour that when we deploy a template with a new default parameter, it will update it.

[joshuamkite]

I have just lost an afternoon because of this, just as I was getting used to SAM. This happens to me regardless as to whether I update my stack parameters as a default or as an override- no changes are detected. I can see even that .aws-sam/build/template.yaml is updated but still no change is pushed up. I even tried deleting the entire .aws-sam directory to trigger a local rebuild but again my template was not updated. So the change evaluation logic appears broken at the AWS end. Frankly this is a nightmare in development - I've had exactly the issue described above. Surely AWS can either fix this logic or create some additional call that can be used to force a refresh. Sidestepping it with sam sync didn't work out for me regardless of the stern warnings to use only in development environments.

[cswilliams]

I also just got bit by this unexpected behavior. Sure hoping this can get addressed.