"Shadow PC" Exercise for NSDI'04
This year, NSDI is running a "Shadow PC" exercise in which students
can review NSDI submissions when consent is given by the authors and the
reviewing occurs in a supervised setting. The aim of this exercise is to educate
students in the art of reviewing by using a more realistic setting than has been
previously available -- real submissions and real reviewing software. We believe that doing so will provide a long term benefit to
the community as a whole.
The remainder of this page describes the process to be followed by consenting
authors and Shadow PC chairs, along with its safeguards. The
exercise an experiment, and we will solicit feedback after it is concluded to
assess whether it should be run in future years. In the meantime, we welcome
your feedback and thank the NSDI PC chairs for their support. We hope that you, as an author,
will consent to the use of your paper in this exercise, and you, as faculty or
student, will make best use of this opportunity.
Steps for Authors
- The submission page includes a checkbox that allows you to "opt
in" to the Shadow PC exercise at the time you submit and for each paper
that you submit. Only papers for which the authors give explicit consent
will be made available to Shadow PCs.
- Shadow PC reviewing will occur in parallel and independent of the main review process on a blinded set of
papers. The shadow and main reviews will not be mixed, and the shadow
process will play
no role in determining the outcome for your paper. The shadow review process will
be held to the same standards of confidentiality and accountability as the
main review process. In particular, the names of the Shadow PC chairs will
be made public but the shadow review process itself will be double-blind,
with neither Shadow PC members nor Shadow PC chairs knowing the identities
of authors.
- Shadow PC reviews will be returned to consenting authors as a benefit of
participating by December 18, one week after the
NSDI notification of acceptance. The exercise is then concluded. We hope
that this additional feedback will prove useful to authors.
Steps for Shadow PC Chairs
- We anticipate that Shadow PC chairs will make use of the exercise via a
class or weekly reading group. In a typical setting, students will be
assigned one or two submissions to review before class (using a similar Web
setup and review form as the main conference). The class will then meet and
function as its own PC by discussing the set of reviews and merits of each
paper, including whether the submissions should be accepted. Shadow PC
Chairs must accept responsibility for the following:
- Organizing a Shadow PC made up of an accountable group of students.
- Advising the students on the ethics of the reviewing process and its
codes of confidentiality and accountability before the exercise
commences.
- Obtaining reviews from participating students before December 11, when
NSDI authors receive notifications, to facilitate their
(blind) return to the paper authors.
- Safeguarding the confidentiality of submissions during and after the exercise.
- Returning feedback gathered from the Shadow PC to the organizers at the conclusion of
the exercise.
- Any faculty member at a higher educational institution may request that a
Shadow PC be set up on their behalf by sending mail to Tom Anderson (tom@cs.washington.edu)
before September 22 (one week after the NSDI submission deadline).
Students, please find a faculty sponsor at your institution rather than mail
us directly. The
mail should describe the intended use of the submissions and indicate that
the Shadow PC chair accepts all of the conditions above.
- We will set up Shadow PC review sites based on the merit of the request
and subject to our resource constraints. The reviewing sites will allow the Shadow PC chair to create their
own PC and assign review work to the PC. We expect to undertake to:
- Populate the review sites with a selection of anonymized submissions
from consenting authors and NSDI review forms. This will occur shortly
after the NSDI submission deadline.
- Provide Shadow PC chairs with administrative access to the site to
allow them to manage their own PC.
- Return Shadow PC reviews to authors at the conclusion of the exercise.
We expect this to occur shortly after the NSDI notifications.
Tom Anderson (tom@cs.washington.edu)
and David Wetherall (djw@cs.washington.edu)
University of Washington