What should curators check before they merge a new proteobot PR (new public workflow submission)

[mlocardpaulet]

How it is today:
Currently, when users submit a workflow output for public submission, we receive a new PR in Proteobot. There, it is easy to check the new new point in the results json. We can also find the hash that allows us to retrieve the uploaded files and ProteoBench intermediate file in the server.
So in theory, we could check a few things before we accept the PR. But we have no instruction and what to check, how, and what criteria constitute an "acceptable" submission.

Points to answer in this discussion:
please add more if you think of somehing

1. who should be curator? How many do we need (this is module-specific)
2. what should curators check for?
3. what can we do to automate part of the process?

[mlocardpaulet]

I personally think that we should check for (if it is easy enough):

1. do the modifications in the parameters correspond to the ones we find in the intermediate file?
2. was the fasta file used the correct one?
3. are there duplicated peptidoform (this should not happen, but can if we mess up parsing)

[rodvrees]

I think all three points are good, question for point 1:
Do you suggest we look in the parameters and then make sure each of these modification is found in the intermediate file? Because if so I have the following comment:
I think it's necessary to approach it in the opposite direction, i.e. are all modifications found in the intermediate file found in the parameters. Because if we see it as necessary that all modifications set in params are present in the intermediate file, we would not allow any run where a modification is set in params, but is not identified by the search engine. Examples:

Since we leave it up to the user to choose whatever PTMs they might like to add to the search space, it might occur that an outlandish modification is searched for but not identified, and I don't feel this run should immediately be considered invalid.
Search engines might be buggy and fail in their PTM identification/reporting. I.e. MaxDIA, to the best of my knowledge, fails to identify ANY modification that is set as fixed or variable. There's communication on this in their help forum (https://groups.google.com/g/maxquant-list/c/8EX0VgnrOMA/m/eiCToGenAwAJ), but no response from their support team. Obviously this is a bad thing and it can be discussed whether or not MaxDIA runs should be submitted at all, but this is a problem from the search engine site, not ours or the user who submits this point.
Conclusion, TLDR:

Instead of checking if every parameter modification is in the intermediate file, we should check if every modification in the intermediate file is in the parameters.

[rodvrees]

Altough the more I read your sentence, the more I think you were already looking at it from that direction 😅

[mlocardpaulet]

I was, we are on the same page. But I don't think I thought about it as much. Thanks a lot for clarifying.

[mlocardpaulet]

4. We can also check the min and max peptide length in the result files.