PubPeer – a website to comment on scientific papers

If you are interested in scientific integrity, you will probably know PubPeer, a website where you can leave anonymous or signed comments on scientific papers.

PubPeer was launched in late 2012 by neuroscientist Brendon Stell and brothers Richard and George Smith, with Boris Barbour and Gabor Brasnjo acting as advisers. At the start, the founders and advisers were anonymous, but they revealed their identity in 2015.

Post-Publication Peer Review

The PubPeer website features post-publication peer review (PPPR), a place to leave scientific praise or criticism after a paper has been published.

Scientific papers traditionally are reviewed before publication in a process called peer-review. During peer review, manuscripts are sent to a couple of other scientists, who were not involved with the study, but who have the expertise to read and improve the paper. Peer reviewers can comment on weaknesses of the manuscript, such as flawed methods, far-fetched conclusions, missing references, or missed hypotheses, and suggest improvements. After their evaluation, the peer reviewers will determine if the manuscript could be accepted for publication if the suggested edits are included. The authors then have a chance to respond and improve the paper, and hopefully get it accepted for publication.

Pre-publication peer review has long been viewed as the golden standard in evaluating and improving the quality of scientific papers. However, many scientists (including myself) think that a paper can also be commented on after publication. Scientific knowledge keeps on growing, and scientific papers are science’s stepping stones, but scientists should be allowed to comment on papers, even after they have been published.

Before PubPeer was launched, there were only limited options to comment on published papers. One could write a letter, comment, or reply to the journal in which the original paper had been published. However, this process was cumbersome and would take months, so most scientists would rather spend their time on something else. It is also hard to check if a paper had been commented on.

Alternative PPPR websites

A year after PubPeer was launched, in the fall of 2013, a similar service was introduced by the National Institute of Health (NIH) / National Center for Biotechnology Information (NCBI), PubMed Commons. This platform was integrated with NCBI’s PubMed, the main access site for biomedical papers, and quickly gathered a number of devoted commenters. However, it never became very popular, and it was discontinued in March 2018. Only 6,000 of the 28 million articles indexed in PubMed had been commented on by that time. Comments on PubMed Commons could not be left anonymously, which might have been the main reason that the service was not used a lot.

In addition, some journals such as those published by the Public Library of Science (PLOS) or BioMed Central (BMC), have their own way to comment on papers. As with the now discontinued PubMed Commons tool, users can only comment using their real name. These features are not used a lot.

Publons, the site where scientists can keep track of their peer reviews and editorial contributions, also has the possibility to comment on papers after their publication, but users need to register first.

Commenting on a scientific paper, in particular when the comments are not positive, is risky business for academics, which depend on letters of recommendation, faculty track evaluations, and collaborations for their career. Therefore, most academics are reluctant to leave comments under their full name.

PubPeer offers anonymity

PubPeer uses a different model than PubMed Commons or other sites. It allows users to choose between leaving signed comments (under their real name) or anonymous – which quickly became the most popular choice.

There are many opinions about the possibility to comment on scientific papers anonymously. Anonymity allows people to reveal scientific misconduct without repercussions for their career. Traditional, pre-publication peer review has been anonymous for that exact same reason. On the other hand, people have argued that it would lead to hateful or false comments by competitors. But, as with pre-publication peer review, most scientists are very capable of formulating their critiques in an objective manner. Of course, we all have experienced the sour comments by reviewer #3, but anonymous scientific peer-review usually remains civil and polite.

When PubPeer was first launched, there were two options to leave comments. All users had to register, but could chose to post under their full name, or as anonymous “peers”. The first comment on a post would always be “Peer 1”, and additional comments were listed as “Peer 2”, “Peer 3”, etc.

In 2013, PubPeer introduced a third option to leave comments, anonymously and without registration. Such comments would be shown as “unregistered”, and quickly became the most popular option. Most “unreg” comments revealed serious concerns about papers, such as image duplication and manipulation.

In 2016, PubPeer introduced an additional layer of security for people who left unregistered comments. It stopped keeping track of IP addresses. This was the result of a lawsuit filed by an author of many papers flagged by PubPeer users for possible image problems. This scientist claimed to have lost a job offer because of the negative PubPeer comments, and sued the site in an attempt to get the IP addresses of the unregistered commenters. Although PubPeer won this legal battle, they realized that it was better to not log IP addresses in the first place, so there would be nothing to hand over to legal authorities if they were subpoenaed again.

In this new approach, commenters can register without IP tracking, by assigning them a user name – a plant, animal, or microorganism name from the tree of life. Users also get a “secret key”, a password that they can use to log in. Commenters can thus create multiple accounts, and switch between them, or switch between anonymous and full-name comment mode easily.

In order to keep the discussions constructive and civil, PubPeer has several guidelines for posting comments. Statements need to stick to facts and be publicly verifiable. For example, one could say that two images look very similar to each other, but not say that the authors committed fraud. Comments from new accounts are moderated and occasionally removed if they violate those rules.

PubPeer’s effects

Since its start in 2012, PubPeer has become a very useful tool for the public posting of shortcomings or concerns in scientific papers. It has led to many corrections and retractions of problematic papers that otherwise would not have been noticed. One could compare this to the effect of social media postings on misconduct by airlines or police officers – where video footage taken by bystanders has led to public outcry about people being treated unfairly, and a change of protocols or behavior (although there is still much that needs to change).

Similarly, for scientific literature, Paul S. Brookes has shown that internet coverage of problematic papers leads to an increased chance of the paper being corrected or retracted.

Indeed, PubPeer has played an important role in the critical assessment and subsequent retraction of published papers, such as the Obokata paper on STAP cells, which was commented on immediately after publication. And the scientist mentioned above, who lost the lawsuit against PubPeer to get commenters’ IP addresses, currently has 77 papers flagged on PubPeer, and over 20 retractions.

How to search on PubPeer

Searching on PubPeer is easy. You can type in the name of the DOI, PMID or author to search for papers. You can also search for particular commenters by typing in: Users: <name>.

Update: The search engine for users appears to be a bit picky. These searches work :

• Users: bik
• Users: elisabeth
• Users: “Elisabeth M Bik”

Some power commenters to look out for are:

• users: “Hoya Camphorifolia”
• users: “Actinopolyspora Biskrensis”
• users: “Indigofera Tanganyikensis”

There are also very helpful PubPeer extensions for your favorite browser (Chrome, Firefox), that alert you for comments on papers, for example during PubMed searches. It is not perfect, but it is very helpful. Here is a screenshot of how that might look like in PubMed:

How to comment on PubPeer

It can be scary to comment on a scientific paper using PubPeer, especially the first couple of times. I would recommend to not start posting under your full name, unless you are confident that your comment would not harm your career, or if you have something positive to say. Positive comments are always appreciated, of course!

To create an anonymous (or a signed) account, click on the “create account” button on the top right to register. Anonymous accounts will get a name from the tree of life (e.g., “Homo sapiens”) and “a secret key”- a 16 digit password that you will need to store in a safe place.

PubPeer’s Frequently Asked Questions section offers many helpful guidelines, or you can just browse the most recent comments to get an idea of what people might discuss.

Further reading

• PubPeer, the online journal club
• How to Publish a Scientific Comment in 1 2 3 Easy Steps [Satire] – Rick Trebino – Semantic Scholar
• PubMed Commons closes its doors to comments – Elie Dolgin – Nature
• Post publication peer-review: Everything changes, and everything stays the same – Bonnie Swoger – Scientific American – 2014
• 5 Steps to Writing a Winning Post-Publication Peer Review – Publons.com
• PubPeer wins appeal of court ruling to unmask commenters – Alison McCook – RetractionWatch – 2016
• Meet PubPeer 2.0: New version of post-publication peer review site launches today – Alison McCook – RetractionWatch – 2017