Proposal for a Persistent Conference Paper Review System
NOTE: This proposal has been shelved, since the steering committees of ISCA, ASPLOS, and HPCA have informed us that they do not wish to participate in a persistent review system. Without the participation of these key conferences, there is no point in pursuing this idea further.
Initially proposed for MICRO-XL, co-program chairs Mikko Lipasti and Gokhan Memik
Executive Summary
The computer architecture research community has grown substantially in recent years. As a result, the number of papers submitted to conferences has increased to the point where the conventional approach for reviewing papers creates an untenable workload for program committee members. This paper proposes a persistent paper review system that will enable reviews for resubmitted papers to persist from one conference to the next, reducing the review workload for subsequent program committees and providing better continuity in review content. The proposed system should not only reduce reviewer workload but also improve the quality of accepted papers.
Background
Currently, top tier computer architecture conferences typically receive between 150 and 200 paper submissions, out of which 25-35 are typically accepted. Given a typical program committee with 30-35 members, and an expectation of five reviews per paper, one of which is externally solicited, each program committee member would have to personally review between 17 and 27 papers, and solicit external reviews for four to seven additional papers. Hence, agreeing to serve on a PC represents a substantial commitment of time and effort.
There are several alternatives for reducing the workload per PC member. One is to increase the size of the program committee. However, this is only a local optimization, since it implies that each member of the community must be willing to serve more frequently on program committees, since each committee now demands a greater number of participants. Furthermore, very large program committees are difficult to manage effectively, leading to potential degradation in the quality and efficacy of PC meetings. A second alternative would be to rely more heavily on external reviews. Again, this only appears to help, since the number of reviews borne by the subset of the community that is both willing and able to review papers does not decrease, but simply shifts from reviews performed while serving on a PC to reviews performed in response to external solicitations. In addition, external reviews are far less valuable in the context of a PC meeting, also contributing to a potential degradation in the quality of the paper selection process. A third potential solution is to require fewer reviews per paper, either through a blanket reduction in reviews for all papers, or by filtering out “easy rejects” with a small number of early reviews for each paper, followed by a second round of additional reviews for the papers that pass the first-round filter. Due to inherent and seemingly random variation in the quality and relevance of reviews, this approach (which has been employed in the past) would also contribute to a potential degradation in the paper selection process.
Instead, the proposed system exploits redundancy in the reviews that are generated across conferences. Since the majority of submitted papers are rejected, many of the papers submitted to subsequent conferences are in fact more-or-less revised versions of papers that have been rejected by earlier conferences. This system will allow the earlier reviews to persist in the review database, will allow authors to identify their papers as revised versions of earlier submissions, and will give the program chair the discretion to utilize the prior reviews, as well as the demonstrated expertise of the associated reviewers, to reduce the workload for the current program committee. The mechanics of this process would be closely modeled after the submit-review-revise-review-accept cycle followed by most journals.
The two main advantages of the proposed persistent review system are: (1) reduced workload for program committees following an initial startup transient (clearly, the first few PCs to utilize this system will have to do more work than usual, since they may be asked to re-review resubmitted papers for subsequent conferences, even though they may not be serving on the PCs for those conferences); and (2) greater continuity and consistency in paper reviews.
Implementation Details
There are several key implementation issues that are addressed in this section. These include creation of a persistent web-based review repository; additional paper submission functionality; additional functionality when reviewing papers; additional functionality for the program chair to view, assign, and track reviews; and the preservation of reviewer confidentiality and double-blind review. These issues are discussed below.
Persistent web-based review repository. The typical practice today is to create a new submission and review website for each instance of each conference. All conferences choosing to participate in the proposed scheme would need to participate in a shared site that all future instances of these conferences would employ. Our suggestion is that the relevant special interest groups (SIGs) and technical committees (TCs) agree to fund the development and maintenance of such a website. Such a website will be similar to that of journal submissions (e.g., acm.manuscriptcentral.com), which will also reduce the load on the program chairs because they will not have to deal with establishing a web site for their conference. In addition, the web site will allow flexibility in terms of what information is available/used by different conferences. A stopgap alternative would be to create a mechanism for migrating the relevant reviews from earlier conferences to future conferences through a database query/load approach. For this to work, each conference would have to agree to use a compatible version of software, and would have to maintain its review database for a year or two, so that reviews would be available for migration to future conferences. In our view, such a stopgap alternative should only be pursued as a last resort.
Additional paper submission functionality. For each subsequent conference, the new website must allow authors to submit a revised version of an earlier manuscript, along with a cover letter describing how they have modified the paper to address comments from prior reviews. This is identical to the functionality provided by current journal paper submission sites (e.g. acm.manuscriptcentral.com). Of course, the author would also have the option of creating a fresh submission that will not be connected to the earlier version or its reviews. The program chairs will also have the option of forwarding the new submission to the reviewers with or without the cover letter. This way, the program chairs will be free in soliciting fresh reviews.
Additional program chair functionality. The website must allow future program chairs to access prior reviews for those papers (and only those papers) that are submitted as revised versions to the current conference. The program chair can then, at his or her discretion, request the earlier reviewers to re-review the paper based on the changes described in the cover letter, solicit all-new reviews for the paper, or some combination of the two. Note that it would advisable for every submitted paper to have at least one review from a member of the current program committee, so that the paper can be discussed authoritatively at the PC meeting. We must highlight that the use of this new submission process is completely at the discretion of the program chairs. Our goal is to provide a resource that can be utilized to reduce the load of reviewers and increase the overall quality. However, it is clear that if the program chairs decide to ignore/omit this option, they will be free to do so.
Guaranteeing reviewer confidentiality. As mentioned above, program chairs should only ever have access to reviews that pertain to papers submitted to the current conference. Any such papers whose authors have a conflict with the program chair will be handled through surrogates, just as they are today. Per ACM and IEEE rules that forbid program chairs from submitting papers to the conference they are chairing, there should not be any such papers authored by the program chair, and hence no access to reviews for previously submitted papers.
Maintaining double-blind review. Clearly, the standard practice of double-blind review will not be possible for papers that are submitted as revised manuscripts, since the identity of the authors is usually disclosed during the PC meeting. This effect can be minimized by the program chair by only disclosing author identities for papers that are discussed at the meeting, and even for those only when necessary. Finally, the program chair should follow double-blind review procedures for any new reviews that are solicited for the paper by preventing the new reviewers from accessing the prior reviews until the program committee meeting or rebuttal period.