Review Reports
Reviewer reports are feedback given to the authors by experts in the
manuscript subject matter. There are two types in Aperta:
FrontMatterReviewerReports and ReviewerReports. They only differ in
the types of material under review.
In Figure rrerd , we see the relevant models
interacting with the reviewer reporting system. Starting with
Invitation, the system adds reviewers to a manuscript with a
ReviewerReportTask assigned to them. After submitting a
ReviewerReport, the administrators can take over and make a Decision
on the paper.
A reviewer needs an invitation before any they can submit a review of a manuscript, Invitations are the mechanism for adding reviewers. Because an invitation has complexity around it status, a state machine realizes the implementation.
In Figure ism , we find the state machine for
Invitation. Although it isn't called out specifically, accepted is
the desired end state.
Table invitation shows a summary of the states and their meaning. Remember that each state is per invitation sent.
State |
Meaning |
|---|---|
Pending |
Invitation created, unsent |
Invited |
Invitation sent |
Accepted |
Reviewer accepted |
Declined |
Reviewer declined |
Rescinded |
Invitation taken back (Reviewer removed) |
The screenshot in Figure ilist1 shows the states of invitees in the Invite Reviewers card. The following sections touch on each of the states represented.
The pending state is the initial state for an invitation that is not yet sent. The Invite Reviewers Task lets a staff user create, organize, edit and send invitations.
Figure ilist1 shows a pending invitation. Because the staff can accept invitations on a user's behalf, it has the accepted by information in the row.
After sending an invitation to a reviewer, the invitation moves on to
the invited state. From there, it can go to Accepted, Declined or
Rescinded.
The status of Invited shows up on the invitations in Figure ilist1 .
This is the happy path. At this point, the reviewer can fill out a
ReviewerReport. The reviewers also have access to the manuscript in
this state. The access remains until the staff make a decision on the
paper.
When a reviewer declines the invitation, we end up in this final state. There is nothing more to do other than track that this user has declined for this round.
If for any reason the staff need to take back (rescind) a reviewer invitation, they only need to hit the rescind button. This is also a final state and removes any outstanding invitations or permissions the potential reviewer had.
The main goal of the reporting flow is to capture reviewer reports. These reports are the owners of the questions that the respective reviewers answer.
There is one reviewer report per reviewer per decision round. It is possible, therefore, to have a single reviewer provide feedback on multiple rounds of a manuscript's revision cycle. This also means for any given manuscript, there will be many reports over the lifetime.
Figure rraasm shows how a ReviewerReport moves
through its states. Some of these details stem from the mechanics of
creating reports (Section Creation of Reviewer Reports).
Basically, there are the three states for most reports. The first is when an invitation remains unaccepted. The invitation can be in any of the pre-acceptance states in Figure ism .
Once the users accepts a corresponding invitation, the reviewer report
moves to the review_pending state. It's here that the reviewer has
control of the next state transition. He or she fills out the feedback
forms and then has the opportunity to submit the review. Figure
feedback_form shows and example feedback form.
The submission of a review happens only once. This locks down a review and signals the staff that the review is complete. No further edits are permissible to the feedback after submitting.
Naturally, after submitting, the report moves to the submitted state.
From here, it shows up for use next by the Decision model in Section
=Decision= to generate emails to authors.
The completion of a review also shows up in the workflow state with a green check-mark as shown in Figure workflow_completed .
Besides the current state of a review, there is also the possiblility to have a dash instead of a checkmark. This indicates that the reviewer has not accepted an invitation. These only show up after a user has accepted a review in a prior decision. Figure workflow_status shows these states.
For each reviewer, the ReviewerReportTask holds the status of the
current round. And a manuscript will have several rounds before a final
decision of reject or accept. Each reviewer can have a review per round.
Or maybe he or she wasn't invited at all. In either case, there will be
a current status available in existing reviewer tasks.
For past rounds, there is a Review History section that has the reviewer involvement captured for earlier manuscript versions.
Figure status_and_history shows the current status
and a brief history list. Note that the history of a reviewer reports
begins with the first report created (invitation accepted). This means
the user's history could start with a later manuscript version than the
initial v0.0.
This task helps to organize a given user's collection of reviews on a manuscript. The system assigns the reviewer to this task and the reports that it contains.
There is only one ReviewerReportTask per Paper. This means it lives
across rounds. It shows the status of the reviews across rounds.
A Decision model is the single root linking all of the other items
together: Paper, Invitation, ReviewerReport. Because the
Decision is representative of the revision rounds on a manuscript, it
holds the history and current state for these items necessary on a
per-round basis.
When making a decision on a paper, the staff will have to select an email template to send to the authors. Some of the available templates will insert the reviewer reports in the email body. Figure decision illustrates an example email with reviewer feedback.
There are special ways that reviewer reports interact with the Aperta system.
The invitation workflow creates ReviewerReportTask in two different
ways because they report on reviewer status in the (long-running)
reviewer report tasks. The difference being whether or not a review has
been already accepted. Or, perhaps better stated, if a
ReviewerReportTask exists.
Let's walk through an example with Reviewer A. An administrator invites Reviewer A to provide comments on a manuscript. Because this is the first invitation, no task exists for Reviewer A at this time. Once the reviewer accepts the invitation, the task comes into existence.
Figure confirmation shows the confirmation dialog presented when completing a review. After this point, the answers given become permanent. A reviewer must be sure to make the review complete.
After the staff submits a decision, the reviewers all lose permission to view the manuscript. This is to ensure that they don't see any intermediate work and are expressly invited to future rounds.
Note that any pending invitations are also rescinded at this time. This avoids any possible acceptance after a completed decision.
The reviewer retains access to any ReviewerReport that he or she may
own. There is a permission related to just these reports for possible
future use.
To anonymize the reviewers' names, we assign reviewer numbers. These are
per reviewer per paper. This means that the numbers for any given
reviewer persist between decisions. Figure
numberingmodels shows the relationship to the
ReviewerNumber model. It basically holds the number for each paper and
has some convience methods for assigning the numbers.
At the time of writing, this is settable via feature flags.
It's possible for the staff to make a decision with outstanding
(incomplete) ReviewerReports and Invitations. For the latter, Aperta
simply rescinds the unaccepted invitations. For the former, the state
remains as-is.
When the author subits the paper again for the next decision, Aperta creates a new reviewer report and the old one stays in its final state.
This means that the historical record could have reports in any of the
available ReviewerReport states displayed in Figure
rraasm .
Currently behind a feature flag, automated chasing helps staff personel follow up on outstanding invitations and reports. I won't deal with the details here, but the overall idea is that by leveraging due-dates, Aperta can send out automatic emails reminding a reviewer to complete the feedback form.
The due-date is present on the ReviewerReportTask in Figure
chasingstatus . This is an administrator view so we
also see the Change due date button as well as the toggles for the
automated reminders.
The minor revisions of a paper version come from changes requested in a technical check. The typical workflow is to evaluate the paper against a set of prescribed requirements (tech checks). If the author fails to meet any of these requirements, the paper goes back for fixing.
When sending the manuscript back for this type of failure, the versioning increments by 0.1. So, a v1.0 paper becomes v1.1. This does not go through the full decision flow, so we do not create new reviewer reports.
ReviewerReports are separate from tasks for two main reasons. First,
this separation allows for the reports to keep their own status and
history. The task remains the main card on the paper for that reviewer
and collect the multiple reviews supplied over the manuscript lifecycle.
The second reason is permissions. By separating the reviews out into a separate-from-the-paper entity, we can better control access and open the door for reviewers to access their feedback outside of the context of a manuscript.
The ReviewerReportTask is for most manuscripts. If a given submission
is a non-research paper (frontmatter), then it will use the
FrontMatterReviewerReportTask instead.
They are the same in the sense that reviewers need to provide feedback. They differ in the questions they ask and models they use.
Reviewer reports still require a bit more attention. I present two areas of future work below.
Because the permissions system assigns the ReviewerReports to a user
(reviewer), then it's possible to have a list of completed reviews
outside of the context of a manuscript.
This may be useful for a reviewer to have a list of her own reviews provided. This could be relevant to staff to identify reviewers that provide the most helpful feedback.
On the initial move to ReviewerReports, a Boolean field indicated the
migrated reports in case the migration failed. This field is no longer
necessary. Removing it would be a good option to eliminate cruft (and
possible confusion). At the time of writing, we are over six months out
from data migration.
There is a current limitation of when the staff can invite reviewers. This may not be desireable and create an artifical dependency that lengthens the time to receive reviews.
Right now, the paper must be in a submitted state in order to enable reviewer invitations. Any changes to this behavior needs undergo discussion with the staff and consider business needs.
Besides the screenshots, I used a couple of gems to generate diagrams for this report. These details follow.
To generate the ERDs, I used the
rails-erd gem. The
installation is easy. Add it to the Gemfile and run. The command line
I used for the diagrams above is in Listing erdrun .
rake erd only="Invitation,ReviewerReport,Task,Decision,Answer" filename="reviewerreporterd" filetype="pdf" title=falseThe other ERD highlights how ReviewerNumber plays with the other
classes. The command line for this is in Listing
numberingerd .
rake erd only="User,Paper,ReviewerReport,Task,ReviewerNumber" filename="reviewernumbererd" filetype="pdf" title=false inheritance=trueThese commands generate PDFs of the diagrams you see earlier.
To produce the AASM diagrams, I installed the graphviz_aasm gem. Then, on the command line, you can just type in a command line specifying the models you need as shown in Listing aasmrun .
rake aasm:draw CLASS=ReviewerReport,Invitationreviewerreporterd.pdf (application/pdf)
invitation_aasm.png (image/png)
invitation_list.png (image/png)
invitation_history.png (image/png)
reviewerreport_aasm.png (image/png)
completed_review.png (image/png)
workflow_completed_review.png (image/png)
workflow_review_states.png (image/png)
review_history_and_status.png (image/png)
decision_and_reviewer_reports.png (image/png)
submission_confirmation.png (image/png)
reviewernumbererd.pdf (application/pdf)
review_status_with_due_date.png (image/png)
invitation_list.png (image/png)
invitation_history.png (image/png)
completed_review.png (image/png)
workflow_completed_review.png (image/png)
workflow_review_states.png (image/png)
review_history_and_status.png (image/png)
decision_and_reviewer_reports.png (image/png)
submission_confirmation.png (image/png)
review_status_with_due_date.png (image/png)
review_questions.png (image/png)
blank_review.png (image/png)
reviewerreporterd.pdf (application/pdf)
invitation_aasm.png (image/png)
reviewerreport_aasm.png (image/png)
reviewernumbererd.pdf (application/pdf)
reviewerreport.svg (image/svg+xml)
reviewerreport.png (image/png)
