peer review

Guest post: Creating value for peer review. Why not?

DOAJ has approximately 100 volunteers carrying out important reviewing work for it. These volunteers come from a variety of backgrounds within academia, journal publishing or from libraries. DOAJ relies on these volunteers to keep applications flowing through the system.

In the first of a new series, we are highlighting our volunteers’ skills and interests which are connected to scholarly publishing. This is a guest post by Alessandro Pierno, DOAJ Associate Editor and Head of the Editorial Board for ReviewerCredits.

image001  Creating value for peer review. Why not?

The peer review process is the cornerstone of scientific communication and should be considered as a key performance indicator (KPI) for scientists given the fact that, by its nature, the task demands time, knowledge and professionalism.

Still, lack of recognition for the peer reviewing work is an unsolved issue which certainly contributes to the shortage of scientists accepting the task. At the same time, with the growth of open access and the “pay to publish” model, journals need to prove the reliability and trustworthiness of their peer review process. It is clear that reviewers, journals and publishers would greatly benefit from an efficient system, rewarding reviewers and validating journals. These considerations triggered a growing interest with companies like ORCID and Publons which started offering authors the opportunity to add reviews that they had performed to their scientific profile. Going one step further, we think that adding a tangible benefit to the peer reviewing work would represent the real game-changer in the field. 

The ReviewerCredits solution

ReviewerCredits.com (RC) keeps a verified history of all reviews performed by each registered reviewer, and – for each verified review – assigns a number of credits, which can be used in a “virtual store” representing, for the first time ever, a tangible benefit. After completing the review of a scientific paper, reviewers log in to the portal and enter a completed review claim for any journal (whether registered with RC or not). RC verifies that the review has been performed by asking a confirmation from the journal’s editorial office. This step is critical in order to create a reliable and traceable history of activity performed. The review is added to the scientist’s personal account earning a variable number of “credits”.

reviewercredits

Furthermore, journals using PKP’s Open Journal Systems for article submission and review can now integrate a free plugin which allows a direct transfer of the claim to the RC platform.

Finally, RC has the ability to add to the author profile any talk delivered during scientific conferences, complementing the information available on the author.

What is unique in ReviewerCredits?

There are three major assets that make ReviewerCredits really unique: 

(a) all reviews claimed must be certified by the Journal Editor and so 100% of the data are double checked and there is no room for inaccuracy or improper claims; 

(b) performing reviews means accruing credits that can be exchanged for benefits across a number of journals and services; 

(c) conference talks can be added to the author profile.

What are the benefits?

The first advantage is that registered scientists will be able to record in one single place all their certified reviews, as well as conference activity. A PDF certificate can be downloaded at any time and will list all the activity performed in chronological order. Additionally, as RC assigns virtual credits, they can use their credits to access selected, discounted services, specifically tailored for authors.

Registered journals benefit from transparency and higher engagement from their reviewers. By rewarding reviewers on behalf of journals, RC creates value for the work they perform, thus helping to motivate them and contributing to reducing the burden on journal Editors. Journals can take an active part in the process supported by RC registering their journal and encouraging their peer reviewers to record their completed reviews.

RC provides a valuable qualifier for a journal by documenting that proper peer review is performed before acceptance of manuscripts. This is particularly relevant at a time when the explosive growth of low quality or questionable journals (journals who accept any manuscript, independently of their quality, for a fee and  provide little or no peer review) is making journal selection very difficult for authors outside the top indexed journals. 

In RC the peer review activity performed by each journal is clearly visible to all and can be shared with a PDF certificate, listing all confirmed reviews performed on behalf of the journal. This is a huge asset for journals wanting to be transparent and it does not affect the confidentiality of review comments.

In conclusion, the topic of recognition of peer review has finally reached the spotlight due to the initiatives of Orcid, Publons and others recently highlighted by Peer Review Week. In this evolving scenario reviewercredits.com can contribute to creating real value for peer reviewers.

 

About ReviewerCredits
ReviewerCredits is a spin-off company endorsed by the University of Milan-Bicocca, launched in 2016. Its core business is the development, maintenance and upgrade of an online platform which has the purpose of certifying peer reviews and conference talks. ReviewerCredits is listed as an innovative startup company in the Italian company register. It was co-founded by two academic researchers, Giacomo Bellani, and Robert Fruscio with a growing team of experts in digital entrepreneurship and STM publishing (Veronica Mariani, Alessandro Pierno, Lucia Steele and Giulio Zuanetti).

Quality of DOAJ listed journals

A recently published article on a comparison of blacklists and whitelists draws the conclusion that “In the DOAJ, more criteria relate to transparency of business and publishing practices rather than to the quality of peer review. This indicates a risk of falsely endorsing the legitimacy of a journal based on its transparent nature, while at the same time ignoring journals’ lack of best practices in peer review”  

Perhaps we could have done a better job in explaining how DOAJ assess the quality of journals. When the DOAJ list started in 2003, peer review was one of four criteria used in the evaluation.

After the upgrade in 2014, to include more than 40 criteria, it is certainly true that most of these pertain to the transparency of publishing and business practices.

At the same time however, peer review has remained a key criteria for judging the quality of journals that apply for inclusion in the DOAJ index. So the aspect of quality peer review weighs heavily in the assessment of the quality of journals.

In contrast to what the authors of the article state on peer review procedures DOAJ requires peer review by at least two independent reviewers.
Page 13 ; “Both blacklists and whitelists include criteria stating that a journal needs to have a “rigorous” peer review system in place (see list of criteria in supplementary file 2). Both whitelists do not define “rigorous”, however, Cabell’s whitelist implies that peer review should be anonymous and conducted by at least two reviewers.

As stated in the article, peer review is one of the intermediate verifiable criteria. That means that when a publishers states on the website that they have for instance double blind peer-review our editors usually check the correctness of this by verifying an accompanying description of the peer review process. However in case of any doubt concerning the journal’s quality, a special editorial team will do a more detailed analysis on quality criteria including peer review practices, editorial board competence, content comparison of published articles, plagiarism checking and other factors. It is safe to say that our users will have a hard time finding journals in DOAJ with no or inadequate peer review procedures in place.

Because peer review until now has been the holy grail of scientific quality control, it is understandable that people link the quality of, or even the mere presence of peer review with the quality of a given journal. The relationship is unfortunately not so clear cut as many want to believe. Peer review by good connections, friends of friends, even colleagues is often seen. In addition independent peer review panels of experts come to very different conclusions regarding one and the same scholarly work. Because of this, the entire peer review procedure is in a state of rapid change.  Indexing services like the DOAJ have to be aware of the shortcomings of the current system and therefore avoid overrating peer review as THE criteria to assess quality. We think that good publishing practices other than peer review and good quality editorial boards are at least as important and more easy to verify as details of peer review practices.

I want to end with a short word on blacklists. We note that blacklists are depending for a large part on difficult verifiable criteria and subjective judgment, while DOAJ depends largely (77%) on easily verifiable criteria related to transparency and business practices. Blacklists also tend to give a lasting sting to the reputation of journals. More often than not, there exist inadequate and non-transparent procedures for a journal to be removed from a blacklist after improving a journal. For this reason blacklists are often inaccurate and out of date. This risk is even more prominent for one of the lists in the PeerJ study, Bealls list, which has officially stopped to exist but has been resurrected by some people with very unclear policies regarding updates, inclusion and removal of journals from the revived list. In addition, blacklists can never be inclusive, while whitelists are inclusive (ie. most journals in the whitelist will be of good quality while many blacklisted journals will not be predatory at all).

It will not come as a surprise that we strongly recommend to users to  use whitelists and not blacklists to check the quality of journals. Let it also be clear that we do not believe in any complementary nature between both list types and there is another important difference between the (DOAJ) whitelist and blacklists: in contrast to blacklists, DOAJ is not in the business of stigmatizing publishers, rather we spend substantial resources helping journals to improve.

Tom Olijhoek – DOAJ Editor in Chief