Paper ranking is a way to extract the PC’s preference order for submitted papers. Each PC member ranks the submitted papers, and a voting algorithm, the Schulze method by default, combines these rankings into a global preference order.
HotCRP supports ranking through tags. The chair chooses a tag for ranking—“rank” is a good default—and enters it on the settings page. PC members then rank papers using their private versions of this tag, tagging their first preference with “~rank#1”, their second preference with “~rank#2”, and so forth. To combine PC rankings into a global preference order, the PC chair selects all papers on the search page and chooses Tags > Calculate rank, entering “rank” for the tag. At that point, the global rank can be viewed by searching for “order:rank”.
PC members can enter rankings by reordering rows in a paper list. For example, for rank tag “rank”, PC members should search for “editsort:#~rank”. Ranks can be entered directly in the text fields, or the rows can be dragged into position using the dotted areas on the right-hand side of the list.
Alternately, PC members can use an offline ranking form. Download a ranking file, rearrange the lines to create a rank, and upload the form again. For example, here is an initial ranking file:
# Edit the rank order by rearranging this file's lines. # The first line has the highest rank. # Lines that start with "#" are ignored. Unranked papers appear at the end # in lines starting with "X", sorted by overall merit. Create a rank by # removing the "X"s and rearranging the lines. A line that starts with "=" # marks a paper with the same rank as the preceding paper. A line that starts # with ">>", ">>>", and so forth indicates a rank gap between the preceding # paper and the current paper. When you are done, upload the file at # http://your.site.here.com/offline Tag: ~rank X 1 Write-Back Caches Considered Harmful X 2 Deconstructing Suffix Trees X 4 Deploying Congestion Control Using Homogeneous Modalities X 5 The Effect of Collaborative Epistemologies on Theory X 6 The Influence of Probabilistic Methodologies on Networking X 8 Rooter: A Methodology for the Typical Unification of Access Points and Redundancy X 10 Decoupling Lambda Calculus from 802.11 Mesh Networks in Moore's Law X 11 Analyzing Scatter/Gather I/O Using Encrypted Epistemologies
The user might edit the file as follows:
8 Rooter: A Methodology for the Typical Unification of Access Points and Redundancy 5 The Effect of Collaborative Epistemologies on Theory = 1 Write-Back Caches Considered Harmful 2 Deconstructing Suffix Trees >> 4 Deploying Congestion Control Using Homogeneous Modalities X 6 The Influence of Probabilistic Methodologies on Networking X 10 Decoupling Lambda Calculus from 802.11 Mesh Networks in Moore's Law X 11 Analyzing Scatter/Gather I/O Using Encrypted Epistemologies
Uploading this file produces the following ranking:
|#8||Rooter: A Methodology for the Typical Unification of Access Points and Redundancy||~rank#1|
|#5||The Effect of Collaborative Epistemologies on Theory||~rank#2|
|#1||Write-Back Caches Considered Harmful||~rank#2|
|#2||Deconstructing Suffix Trees||~rank#3|
|#4||Deploying Congestion Control Using Homogeneous Modalities||~rank#5|
Since #6, #10, and #11 still had X prefixes, they were not assigned a rank. Searching for “order:~rank” returns the user’s personal ranking; administrators can search for “order:pcname~rank” to see a PC member’s ranking. Once a global ranking is assigned, “order:rank” will show it.