The PLDI conference has an artifact evaluation committee (I’m on the committee) in an effort to make research reproducible. Bidding for PLDI 2016 artifacts has started today, which brought to my attention two problems of anonymity for reviewers and authors.
In computer science a research paper and an accompanying artifact are peer-reviewed usually in a “blind fashion” — one or either of the sides (the reviewer and the author) doesn’t know who is the other side. The paper is usually submitted to a centralized paper review service not controlled by the author nor reviewer. However, the artifact is usually just linked to from the paper and its metadata. The artifact may reside on a server controlled by the author. This jeopardizes anonymity in two ways: 1) that of the reviewer, as the author could see the reviewer’s IP address from reviewer accessing the artifact, from which the author could infer who is the reviewer, and 2) that of the author, if the link (URL) to the artifact on the server is given with a domain name of the server. (There are more steps to ensure anonymity of both sides, but I will not address them in this post.) Here are simple solutions to both of the problems.
Anonymity of the author
The domain name of the server could reveal the institution of the author. For example, the URL to the artifact might be something like:
http://www.cs.utah.edu/pldi-2016/virtual-machine.ovaFrom the URL it is clear the author is from the computer science department at the University of Utah. What the author should do is to replace the domain name www.cs.utah.edu in the URL with its IP address:
http://155.98.65.24/pldi-2016/virtual-machine.ovaObviously, there shouldn’t be any other parts of the URL identifying the author, such as a home directory, e.g. /~john/.
Anonymity of the reviewer
The reviewer could reveal their identity to the author via their IP address when accessing the artifact’s web server under control of the author. Therefore, the reviewer should access the web server and download the artifact through an anonymity network, e.g. Tor. For browsing the artifact’s website probably the Tor browser would be a good choice. If the reviewer needs to download the artifact to an evaluation machine other than their desktop computer or laptop — which is highly recommended because the daily tasks running on the reviewer’s machine shouldn’t interfere in any way with the artifact under evaluation — they could still download the artifact through Tor. On the evaluation machine running Debian, install Tor (by default, it will run a client only):
sudo apt-get install torThen to download the artifact from the URL above, use wget through Tor on the evaluation machine:
torsocks wget \
http://155.98.65.24/pldi-2016/virtual-machine.ovaNeither the reviewer nor the author should take steps to infer the identity of the other, but it’s good to follow these two simple steps to strengthen anonymity of both sides.
Additionally, in case you are on an artifact evaluation committee, you might find my artifact evaluation setup useful.