Use absolute path of template to decide if we need to remove it

[vaxvms]

What this PR does

When alertmanager.data_dir is set to a relative path (as in default configuration), templates are deleted and recreated on a regular basis.
This PR fix the issue.

Which issue(s) this PR fixes or relates to

Fixes #3591

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[aknuds1]

Could this fix be accompanied by a test? I think that would be helpful in order to avoid a potential future regression :)

[vaxvms]

I'll need some guidance on how to write a test for this case.

I can trigger it in multitenant_test.go:TestMultitenantAlertmanager_loadAndSyncConfigs but only if user3Dir is a relative path. And user3Dir is an absolute path because of multitenant_test.go:mockAlertmanagerConfig using t.TempDir()

[pracucci]

When alertmanager.data_dir is set to a relative path (as in default configuration), templates are deleted and recreated on a regular basis.

Why are they? I think I need more guidance on what the root cause is, to better understand the fix.

[aknuds1]

Why are they? I think I need more guidance on what the root cause is, to better understand the fix.

I was also hoping a test would make it clearer how this bug is happening.

[vaxvms]

When alertmanager.data_dir is set to a relative path (as in default configuration), templates are deleted and recreated on a regular basis.

Why are they? I think I need more guidance on what the root cause is, to better understand the fix.

A list of all current file templates is built at https://github.com/grafana/mimir/blob/main/pkg/alertmanager/multitenant.go#L638

Then, at https://github.com/grafana/mimir/blob/main/pkg/alertmanager/multitenant.go#L649 they are removed from the list if they are still relevant (ie, in the received configuration).

But at theses two steps, they aren't built in the same way :

• by joining configuration parameter with filename at the first step. If configuration is a relative path, we end up with a relative path
• by using safeTemplateFilepath function at the second step. This function always return an absolute path (see https://github.com/grafana/mimir/blob/main/pkg/alertmanager/multitenant.go#L1141 )

My PR built the two filenames the same way, fixing the bug.

[vaxvms]

Why are they? I think I need more guidance on what the root cause is, to better understand the fix.

I was also hoping a test would make it clearer how this bug is happening.

I've added a testcase in the PR

[56quarters]

Test looks good to me! I'll wait for some other maintainers to take look before merging.

[aknuds1]

Thanks, this looks good as I can understand why you would need to make this fix (to mirror line 643) in light of your test.

Please review my language related suggestions :)

[vaxvms]

Thanks for your suggestions :) Here they are.

[aknuds1]

LGTM, nice fix!