PRs from openapi-sdkautomation should not be marked as customer-reported

[ramya-rao-a]

These are the PRs from automation that end up getting the customer-reported label. See https://github.com/Azure/azure-sdk-for-js/pulls?q=is%3Apr+author%3Aapp%2Fopenapi-sdkautomation+is%3Aclosed+

This messes up my count for actual community contributions :(

[ramya-rao-a]

cc @AlexGhiondea, @mitchdenny

[kurtzeborn]

Assigning this to @lillyjma for follow up. Pretty sure the change is to FabricBot rules to not apply that label when the issues are opened by a bot. Not sure if FabricBot is able to distinguish between a bot account and a real account (note that in GitHub UI it's labeled as a "bot" account). If it can't, a workaround for this specific issue is to not apply the label when this specific bot account ("openapi-sdkautomation") opened the issue.

[lilyjma]

@AlexGhiondea: This bot is called openapi-sdkautomation, not the usual msftbot that I'm familiar with. Not sure who owns this bot or when it's added to our repos.

[mitchdenny]

The openapi-sdkautomation bot is something that creates PRs against our repository to validate incoming swagger changes. However the intent behind this issue is really to make sure our fabric bot knows not to set those PRs/issues as customer reported.

We don't know for sure, but the rule you need to set is probably something like this:

You'll need to experiment to make sure.

[lilyjma]

@mitchdenny Yeah, I figured that might be the problem. Wes said the openapi bot isn't even a user account, so I'm not sure if this will help. He suspects that the bot is a GH app, and we're waiting for Phoenix to confirm that. I looked through all the conditions available, if the bot is an app, maybe we could do: Condition="Title contains..." Pattern="ReleasePR". This is not ideal, but I can't think of anything else now. @AlexGhiondea thoughts?

[ramya-rao-a]

I would not recommend using any pattern in the PR title which is subject to change at any point.

[mitchdenny]

I'm pretty sure the GitHub events that FabricBot is responding to will have the bot name where the user identity is.

[lilyjma]

@mitchdenny I didn't know that's the case. I've added the rule that you proposed. @ramya-rao-a let me know if the issue still comes up.

[PhoenixHe-NV]

@zhenglaizhang @ruowan to confirm the config. I don't know much about this config line.

[ruowan]

We didn't own this config. Maybe it could work by adding a new check for activity sender.

[lilyjma]

@mitchdenny I don't think it's working. I added the rule 2 days ago, and for a recent PR created by the api bot, the msftbot is still adding customer-reported. The api bot is actually an app, https://github.com/apps/openapi-sdkautomation. Maybe that's why adding the condition didn't work. cc @AlexGhiondea

[ramya-rao-a]

Any updates here?

[ramya-rao-a]

Ping!

[weshaggard]

I'm not sure if this will help or not but I noticed in the github issues query it marked the user as "app/openapi-sdkautomation" so I tried using that as the user.

@ramya-rao-a please let use know if you see any new issues.

[ramya-rao-a]

@weshaggard This PR is from 4 days ago and it did get the customer-reported label: Azure/azure-sdk-for-js#17566

[weshaggard]

I suspect we don't have many other options here given the tools we have. While looking at the rule again I did notice there was a space at the end of the user name, not sure if that is any issue or not but I removed the space and saved the rule just in case.

[AlexGhiondea]

@weshaggard that might have done it... let's wait and see. If that doesn't work, I am not sure what else we can do... maybe have a separate rule that removes that label?

[ramya-rao-a]

Nope, did not work. Here is another PR from 4 days ago: Azure/azure-sdk-for-js#17738

[xiangyan99]

We are seeing the same issue in python repo. e.g. Azure/azure-sdk-for-python#21251.

It was created by azclibot and we labeled it "customer-reported".

[weshaggard]

For regular github user accounts we can probably add them to a list to exclude, but for the app accounts we still don't have any good way to exclude them.

@jsquire you have been using the FabricBot a while do you know of any other potential ways to exclude this customer-focused label for our bots?

[jsquire]

Unfortunately, I don't. The loosest part of the current ruleset is the check for Member; my understanding is that will match any account that belongs to the Azure organization - whether it has permissions to one of our specific repositories or not.

If we're able to have partner bots join the Azure organization, that should be sufficient to avoid getting tagged as customer-reported; if not, the manually managed list is probably the only other option we have that doesn't involve permission grants. For my two cents, I'd really like to avoid that list if we can, I can see that being unpleasant to maintain.

[EDIT]
... and I've just caught up on reading the rest of the thread. If the bot is not a GH account, how in the world is it able to work with the GH API to submit PRs?

[weshaggard]

There are some bot accounts that are github users, but there are also github applications that have their own set of permissions and can create issues/PRs in the repo and they are app accounts.

[jsquire]

I just reached out to the team that owns the Fabric bot for advice; hopefully, they'll have some insight that we can take advantage of.

[jsquire]

In the meantime, I'm going to try tuning the rule in the JS repo to filter against openapi-sdkautomation[bot] which is what the responder comment used for the author in #17566.

@ramya-rao-a: Can you keep an eye and let me know how that works out?