Review ratings

PC members and, optionally, external reviewers can rate one another’s reviews. We hope this feedback will help reviewers improve the quality of their reviews. The interface appears above each visible review:

How helpful is this review?  

When rating a review, please consider its value for both the program committee and the authors. Helpful reviews are specific, clear, technically focused, and, when possible, provide direction for the authors’ future work. The rating options are:

The review has acceptable quality. This is the default, and should be used for most reviews.
Very helpful
Great review. Thorough, clear, constructive, and gives good ideas for next steps.
Too short
The review is incomplete or too terse.
Too vague
The review’s arguments are weak, mushy, or otherwise technically unconvincing.
Too narrow
The review’s perspective seems limited; for instance, it might overly privilege the reviewer’s own work.
Not constructive
The review’s tone is unnecessarily aggressive or gives little useful direction.
Not correct
The review misunderstands the paper.

HotCRP reports the numbers of non-average ratings for each review. It does not report who gave the ratings, and it never shows rating counts to authors.

To find which of your reviews might need work, simply search for “rate:−”. To find all reviews with positive ratings, search for “re:any rate:+”. You may also search for reviews with specific ratings; for instance, search for “rate:helpful”.


Chairs set how ratings work on the review settings page.


A review’s ratings are visible to any unconflicted PC members who can see the review, but HotCRP tries to hide ratings from review authors if they could figure out who assigned the rating: if only one PC member could rate a review, then that PC member’s rating is hidden from the review author.