Quality of DOAJ listed journals

A recently published article on a comparison of blacklists and whitelists draws the conclusion that “In the DOAJ, more criteria relate to transparency of business and publishing practices rather than to the quality of peer review. This indicates a risk of falsely endorsing the legitimacy of a journal based on its transparent nature, while at the same time ignoring journals’ lack of best practices in peer review”  

Perhaps we could have done a better job in explaining how DOAJ assess the quality of journals. When the DOAJ list started in 2003, peer review was one of four criteria used in the evaluation.

After the upgrade in 2014, to include more than 40 criteria, it is certainly true that most of these pertain to the transparency of publishing and business practices.

At the same time however, peer review has remained a key criteria for judging the quality of journals that apply for inclusion in the DOAJ index. So the aspect of quality peer review weighs heavily in the assessment of the quality of journals.

In contrast to what the authors of the article state on peer review procedures DOAJ requires peer review by at least two independent reviewers.
Page 13 ; “Both blacklists and whitelists include criteria stating that a journal needs to have a “rigorous” peer review system in place (see list of criteria in supplementary file 2). Both whitelists do not define “rigorous”, however, Cabell’s whitelist implies that peer review should be anonymous and conducted by at least two reviewers.

As stated in the article, peer review is one of the intermediate verifiable criteria. That means that when a publishers states on the website that they have for instance double blind peer-review our editors usually check the correctness of this by verifying an accompanying description of the peer review process. However in case of any doubt concerning the journal’s quality, a special editorial team will do a more detailed analysis on quality criteria including peer review practices, editorial board competence, content comparison of published articles, plagiarism checking and other factors. It is safe to say that our users will have a hard time finding journals in DOAJ with no or inadequate peer review procedures in place.

Because peer review until now has been the holy grail of scientific quality control, it is understandable that people link the quality of, or even the mere presence of peer review with the quality of a given journal. The relationship is unfortunately not so clear cut as many want to believe. Peer review by good connections, friends of friends, even colleagues is often seen. In addition independent peer review panels of experts come to very different conclusions regarding one and the same scholarly work. Because of this, the entire peer review procedure is in a state of rapid change.  Indexing services like the DOAJ have to be aware of the shortcomings of the current system and therefore avoid overrating peer review as THE criteria to assess quality. We think that good publishing practices other than peer review and good quality editorial boards are at least as important and more easy to verify as details of peer review practices.

I want to end with a short word on blacklists. We note that blacklists are depending for a large part on difficult verifiable criteria and subjective judgment, while DOAJ depends largely (77%) on easily verifiable criteria related to transparency and business practices. Blacklists also tend to give a lasting sting to the reputation of journals. More often than not, there exist inadequate and non-transparent procedures for a journal to be removed from a blacklist after improving a journal. For this reason blacklists are often inaccurate and out of date. This risk is even more prominent for one of the lists in the PeerJ study, Bealls list, which has officially stopped to exist but has been resurrected by some people with very unclear policies regarding updates, inclusion and removal of journals from the revived list. In addition, blacklists can never be inclusive, while whitelists are inclusive (ie. most journals in the whitelist will be of good quality while many blacklisted journals will not be predatory at all).

It will not come as a surprise that we strongly recommend to users to  use whitelists and not blacklists to check the quality of journals. Let it also be clear that we do not believe in any complementary nature between both list types and there is another important difference between the (DOAJ) whitelist and blacklists: in contrast to blacklists, DOAJ is not in the business of stigmatizing publishers, rather we spend substantial resources helping journals to improve.

Tom Olijhoek – DOAJ Editor in Chief