Manually ASIM Deployment - Failed to validate, Conflict

[rcegan]

Describe the bug
Upon trying to deploy ASIM parsers via the 'Deploy to Azure' button and filling out the required parameters, a 'Conflict' error appears with the following error: (workspace, location and resource group have been omitted)

The resource '<SENTINELWORKSPACE> already exists in location <LOCATION> in resource group <RESOURCEGROUP>. A resource with the same name cannot be created in location <LOCATION>. Please select a new resource name.

Upon giving it a different name, it creates a new LAW in the designated resource group.

To Reproduce
Steps to reproduce the behavior:

1. Go to list of parsers https://github.com/Azure/Azure-Sentinel/tree/master/ASIM
2. Deploy any of them
3. Fill out the workspace name in the 'Workspace' field and pick the correct resource group containing the LAW
4. Deploy

Expected behavior
The parsers should deploy into the targeted workspace and become available for query.

Screenshots

Desktop (please complete the following information):

• OS: Windows 11
• Browser: Edge
• Version: 115

Additional context
I have all required permissions to deploy this content.

[github-actions]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[github-actions]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-rbajaj]

Hi @rcegan, we are checking on this and we will provide an update soon.

[v-rbajaj]

Hi @rcegan, waiting for update from concerned team, once we receive an update will let you know.

[v-rbajaj]

Hi @rcegan, waiting for update from concerned team, once we receive an update will let you know.

[v-rbajaj]

Hi @rcegan, we are still waiting for update from concerned team, we will get back to you once there is an update.

[v-rbajaj]

Hi @rcegan, we are still waiting for update from concerned team, we will get back to you once there is an update.

[rcegan]

I was able to work around the problem by deploying the resources to a different location than the log analytics workspace itself. Solves my issue but I suspect that's probably not the intended behaviour

[v-rbajaj]

I was able to work around the problem by deploying the resources to a different location than the log analytics workspace itself. Solves my issue but I suspect that's probably not the intended behaviour

According to the error message, it is working as expected.

Try deleting the previous one and installing in the same location, it should work fine.

[vakohl]

@rcegan thanks for raising this issue. Can you please share the location details where it failed and where it worked for you?

[rcegan]

I was working with Australia Southeast and Australia East. Deploying to Australia Southeast, with the log analytics workspace in the same location, the deployment failed (due to a resource already existing in that location - the workspace itself). Picking a different location (Australia East) it appeared to work fine.

[azurekid]

Hi, This behavior is as expected due to the following reason.
When deploying the ASIM parsers using the ARM templates a POST request is done at the background.

Because the ASIM parsers already exists, this will result in an error as described above.

There are 2 workarounds to remediate this issue:

• Remove the existing ASIM parsers for the workspace using the Microsoft PowerShell script
• Deploy the updated parsers using the API using the UPDATE method.

It is more an issue related to Log Analytics than Microsoft Sentinel or ASIM as they are dependent on Log Analytics.

What I can do is create a PowerShell script that updates the ASIM parsers is they already exists and use the ASIM yaml files as input. Or create and ARM template that uses a Script Extension to run the PowerShell script in a different context.

This might need some extra permissions in the Azure Subscription / Resource Group but need to validate this first.

@rcegan @vakohl

[vakohl]

@rcegan I created a new workspace, deployed Process Event parsers multiple times on same workspace. I didn't got that error. Am I missing anything that you tried? Which schema parser you were trying to deploy. Can you please retry on a new and on the same workspace you previously tried, see if you are still getting that error?

[v-rbajaj]

Hi @rcegan,

Gentle Reminder: We are awaiting for your response on this issue. If you still need to keep this issue active please respond on it in the next 2 days. If we don't receive response, we will be close this issue.

[jusso-dev]

Sorry to add to the noise, but I'm having the exact same problem. Tried deploying all ASIM parsers and then just one-by-one, same error message "Conflict"

I've tried putting the LAW in Australia Southeast and the LAW in Australia East, and also putting them in the same region, no difference to the output.

Drilling into the error message isn't any more help, just states eTAG for specific error was outdated

newer data exists. If you are using eTag please use the latest one and try again in a few minutes. Operation Id: '4918a525df68776dd0a7063aaa6db6dd' (Code: NewerDataExists)

Trying to redeploy multiple times to get other deployments to succeed does not help either.

[vakohl]

@rcegan @jusso-dev FYI, this issue is being investigated.

[vakohl]

@rcegan @jusso-dev @azurekid
This issue is still being looked into. This seems more of Log Analytics issue than Sentinel, we are working with concerned teams to get this fixed.

[vakohl]

tagging related issue, raised by another user - #10481

[frendsick]

The root cause of the issue is that each parser deployment script deploys the Log Analytics workspace before deploying the saved search. The conflict happens when tens of parser deployment ARM templates are trying to deploy the Log Analytics workspace at the same time.

The current version of ASIM parser deployment could create a new Log Analytics workspace, which could be unwanted behavior. I would expect the ASIM parser deployment ARM template only to create parsers for the already existing Log Analytics workspace and not create another one if I, for example, type the workspace name wrong into the template.

The issue does not relate to the existing versions of the ASIM parsers, as I have seen this issue occur when deploying the ASIM parsers for the first time to a given Log Analytics workspace. The error messages always refer to the workspace, which is constantly redeploying itself by the referenced ARM templates. Also, the referenced parser templates always seem to define "etag": "*", meaning they always overwrite the existing resource.

I made an example commit on how the issue could be solved by only verifying the Log Analytics workspace exists by declaring the workspace as an existing resource: 0986d7f

@vakohl, what do you think of this solution? Would you like me to implement it for the other parsers as well, or would you prefer to tackle the issue in-house?

[hitem]

Any news on this?
Ive deployed a few sentinels the last few months and all of them had these issues. I started deploying each entity by itself and i have about 70% success and 30% fails with "Newer data exists".

Here is when running the DNS asim by itself as an example:

"Failed to modify resource '/subscriptions/xxxxxxxxxxxxxxx' Microsoft.OperationalInsights/workspaces/xxxxxxxxx', newer data exists. If you are using eTag please use the latest one and try again in a few minutes"

[frendsick]

I found a way to deploy the saved search without deploying the Log Analytics workspace and without using the languageVersion 2.0. I believe this should be the least possible change for each parser ARM template file to fix the issue of unnecessarily redeploying the Log Analytics workspace.

Example: 5aa3b75

I will prepare the required changes for the parser ARM templates and create a PR.