Reviewing has started for NeurIPS 2019 and this time around I am an area chair (AC). We’ve been given a lot of instructions and some tasks: bidding on papers, bidding on reviewers, adjusting reviewers, identifying what we think are likely rejects in the batch of papers we are handling, and so on. It’s a little more involved than being an AC for ICML, but that’s to be expected since the whole reviewing game has been evolving rapidly to adapt to the massive increase in submissions.
Since there is yet another tier of TPC above the ACs (the Senior ACs), how should one approach the meta-review? One view is that the meta-review is AC’s decision/opinion informed by the reviews, the response, the discussion, and their own reading of the paper. This makes the AC a bit like an associate editor at a journal. This also gives the AC quite a bit of flexibility: if the discussion is limited or not particularly useful, the AC can fill in the gap by adding their own voice. The downside is that ACs might bring more of their own preferences (or biases) to the process.
A different approach is to make the meta-review akin to a panel summary as part of an NSF review. In the panels I have been on, there are N people who write reviews of each proposal, one of whom leads the discussion. There is also a scribe for the discussion who has not written a review: a dispassionate observer. The whole panel (even those who didn’t read the proposal) participates in the discussion. The scribe is supposed to draft a summary/synthesis of the discussion and runs it past the panel for edits until they reach a consensus. The N reviews are still there though, with their diversity of opinion.
I think I might prefer the second model. The setup is a bit different, since authors get to respond to the reviews. The meta-review is supposed to augment the existing reviews by incorporating the discussion and author response. The AC is supposed to guide the discussion, which is a role shared by the lead discussant and program officer in the NSF model. The only problem is that the amount of discussion on each paper is highly variable. It’s sometimes like pulling teeth to get reviewers to respond/interact. Reviewers, for their part, might be participating in 5 different discussions, so context switching to each paper can be tough. But for papers with some reasonable discussion, the meta-review as panel summary might be a good way to go.
One complaint about panel summaries is that they often feel anodyne. However, I think this might be desirable in a meta-review, since it could lead to fewer angry authors. One aspect of the NSF model which I think could be adopted, regardless of how the AC views their job, is running the meta-review past the reviewers. I did this for ICML and got some edits and feedback from the reviewers that improved the final review.