Advice and best practice

Guest post: Creating value for peer review. Why not?

DOAJ has approximately 100 volunteers carrying out important reviewing work for it. These volunteers come from a variety of backgrounds within academia, journal publishing or from libraries. DOAJ relies on these volunteers to keep applications flowing through the system.

In the first of a new series, we are highlighting our volunteers’ skills and interests which are connected to scholarly publishing. This is a guest post by Alessandro Pierno, DOAJ Associate Editor and Head of the Editorial Board for ReviewerCredits.

image001  Creating value for peer review. Why not?

The peer review process is the cornerstone of scientific communication and should be considered as a key performance indicator (KPI) for scientists given the fact that, by its nature, the task demands time, knowledge and professionalism.

Still, lack of recognition for the peer reviewing work is an unsolved issue which certainly contributes to the shortage of scientists accepting the task. At the same time, with the growth of open access and the “pay to publish” model, journals need to prove the reliability and trustworthiness of their peer review process. It is clear that reviewers, journals and publishers would greatly benefit from an efficient system, rewarding reviewers and validating journals. These considerations triggered a growing interest with companies like ORCID and Publons which started offering authors the opportunity to add reviews that they had performed to their scientific profile. Going one step further, we think that adding a tangible benefit to the peer reviewing work would represent the real game-changer in the field. 

The ReviewerCredits solution

ReviewerCredits.com (RC) keeps a verified history of all reviews performed by each registered reviewer, and – for each verified review – assigns a number of credits, which can be used in a “virtual store” representing, for the first time ever, a tangible benefit. After completing the review of a scientific paper, reviewers log in to the portal and enter a completed review claim for any journal (whether registered with RC or not). RC verifies that the review has been performed by asking a confirmation from the journal’s editorial office. This step is critical in order to create a reliable and traceable history of activity performed. The review is added to the scientist’s personal account earning a variable number of “credits”.

reviewercredits

Furthermore, journals using PKP’s Open Journal Systems for article submission and review can now integrate a free plugin which allows a direct transfer of the claim to the RC platform.

Finally, RC has the ability to add to the author profile any talk delivered during scientific conferences, complementing the information available on the author.

What is unique in ReviewerCredits?

There are three major assets that make ReviewerCredits really unique: 

(a) all reviews claimed must be certified by the Journal Editor and so 100% of the data are double checked and there is no room for inaccuracy or improper claims; 

(b) performing reviews means accruing credits that can be exchanged for benefits across a number of journals and services; 

(c) conference talks can be added to the author profile.

What are the benefits?

The first advantage is that registered scientists will be able to record in one single place all their certified reviews, as well as conference activity. A PDF certificate can be downloaded at any time and will list all the activity performed in chronological order. Additionally, as RC assigns virtual credits, they can use their credits to access selected, discounted services, specifically tailored for authors.

Registered journals benefit from transparency and higher engagement from their reviewers. By rewarding reviewers on behalf of journals, RC creates value for the work they perform, thus helping to motivate them and contributing to reducing the burden on journal Editors. Journals can take an active part in the process supported by RC registering their journal and encouraging their peer reviewers to record their completed reviews.

RC provides a valuable qualifier for a journal by documenting that proper peer review is performed before acceptance of manuscripts. This is particularly relevant at a time when the explosive growth of low quality or questionable journals (journals who accept any manuscript, independently of their quality, for a fee and  provide little or no peer review) is making journal selection very difficult for authors outside the top indexed journals. 

In RC the peer review activity performed by each journal is clearly visible to all and can be shared with a PDF certificate, listing all confirmed reviews performed on behalf of the journal. This is a huge asset for journals wanting to be transparent and it does not affect the confidentiality of review comments.

In conclusion, the topic of recognition of peer review has finally reached the spotlight due to the initiatives of Orcid, Publons and others recently highlighted by Peer Review Week. In this evolving scenario reviewercredits.com can contribute to creating real value for peer reviewers.

 

About ReviewerCredits
ReviewerCredits is a spin-off company endorsed by the University of Milan-Bicocca, launched in 2016. Its core business is the development, maintenance and upgrade of an online platform which has the purpose of certifying peer reviews and conference talks. ReviewerCredits is listed as an innovative startup company in the Italian company register. It was co-founded by two academic researchers, Giacomo Bellani, and Robert Fruscio with a growing team of experts in digital entrepreneurship and STM publishing (Veronica Mariani, Alessandro Pierno, Lucia Steele and Giulio Zuanetti).

Article uploads: the importance of an ISSN

This is a reminder that in February this year we tightened up procedures for article uploads. This includes checks that are carried out on uploads particularly ISSN, DOI, and full text URL information.

This means that, whether uploading article XML to us or JSON via the API, the metadata must include the exact same ISSN value(s) as those we hold in the journal record in DOAJ. If you have two ISSNs on the website and we only have one, you should contact us to have the second ISSN added. Usually, this is a quick process but sometimes you might need to contact ISSN.org to have them update their information before we can continue.

If you have any questions on this, do contact us.

Why do we ask applicants to wait 6 months before they apply again?

When we reject an application, the rejection email contains details about why the application was not successful and usually tells applicants that they must wait 6 months before submitting another application for the same journal to us. Why do we do this?

One reason is that it is an attempt to discourage repeat applications, made in haste, and we get many, many of those. Repeat and duplicate applications clog up the system and take our dedicated volunteers and team away from those applications which need some time spent on them. (In 10 months alone, DOAJ received 221 duplicate applications!)

The other reason is that many of the recommendations that we make in our rejection emails, recommendations made to help journals meet our criteria, take time to implement. Adding words to a website isn’t enough. Changes need to be implemented properly, communicated to stakeholders, tested, and managed. Some changes will require other parties to implement changes too. This all takes time. After the 6 months has passed, we welcome a new application but we ask that the journal website demonstrates very clearly that our recommendations have been put into practice and our editorial team will be very careful to check that all our recommendations have been implemented.

Quality of DOAJ listed journals

A recently published article on a comparison of blacklists and whitelists draws the conclusion that “In the DOAJ, more criteria relate to transparency of business and publishing practices rather than to the quality of peer review. This indicates a risk of falsely endorsing the legitimacy of a journal based on its transparent nature, while at the same time ignoring journals’ lack of best practices in peer review”  

Perhaps we could have done a better job in explaining how DOAJ assess the quality of journals. When the DOAJ list started in 2003, peer review was one of four criteria used in the evaluation.

After the upgrade in 2014, to include more than 40 criteria, it is certainly true that most of these pertain to the transparency of publishing and business practices.

At the same time however, peer review has remained a key criteria for judging the quality of journals that apply for inclusion in the DOAJ index. So the aspect of quality peer review weighs heavily in the assessment of the quality of journals.

In contrast to what the authors of the article state on peer review procedures DOAJ requires peer review by at least two independent reviewers.
Page 13 ; “Both blacklists and whitelists include criteria stating that a journal needs to have a “rigorous” peer review system in place (see list of criteria in supplementary file 2). Both whitelists do not define “rigorous”, however, Cabell’s whitelist implies that peer review should be anonymous and conducted by at least two reviewers.

As stated in the article, peer review is one of the intermediate verifiable criteria. That means that when a publishers states on the website that they have for instance double blind peer-review our editors usually check the correctness of this by verifying an accompanying description of the peer review process. However in case of any doubt concerning the journal’s quality, a special editorial team will do a more detailed analysis on quality criteria including peer review practices, editorial board competence, content comparison of published articles, plagiarism checking and other factors. It is safe to say that our users will have a hard time finding journals in DOAJ with no or inadequate peer review procedures in place.

Because peer review until now has been the holy grail of scientific quality control, it is understandable that people link the quality of, or even the mere presence of peer review with the quality of a given journal. The relationship is unfortunately not so clear cut as many want to believe. Peer review by good connections, friends of friends, even colleagues is often seen. In addition independent peer review panels of experts come to very different conclusions regarding one and the same scholarly work. Because of this, the entire peer review procedure is in a state of rapid change.  Indexing services like the DOAJ have to be aware of the shortcomings of the current system and therefore avoid overrating peer review as THE criteria to assess quality. We think that good publishing practices other than peer review and good quality editorial boards are at least as important and more easy to verify as details of peer review practices.

I want to end with a short word on blacklists. We note that blacklists are depending for a large part on difficult verifiable criteria and subjective judgment, while DOAJ depends largely (77%) on easily verifiable criteria related to transparency and business practices. Blacklists also tend to give a lasting sting to the reputation of journals. More often than not, there exist inadequate and non-transparent procedures for a journal to be removed from a blacklist after improving a journal. For this reason blacklists are often inaccurate and out of date. This risk is even more prominent for one of the lists in the PeerJ study, Bealls list, which has officially stopped to exist but has been resurrected by some people with very unclear policies regarding updates, inclusion and removal of journals from the revived list. In addition, blacklists can never be inclusive, while whitelists are inclusive (ie. most journals in the whitelist will be of good quality while many blacklisted journals will not be predatory at all).

It will not come as a surprise that we strongly recommend to users to  use whitelists and not blacklists to check the quality of journals. Let it also be clear that we do not believe in any complementary nature between both list types and there is another important difference between the (DOAJ) whitelist and blacklists: in contrast to blacklists, DOAJ is not in the business of stigmatizing publishers, rather we spend substantial resources helping journals to improve.

Tom Olijhoek – DOAJ Editor in Chief

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

Are you using an up-to-date version of DOAJ metadata? Apparently not.

Did you know that DOAJ’s metadata refreshes every hour? Given how many changes we, the DOAJ Team, make to the database on a weekly basis plus all the article metadata which is continuously being uploaded to us, the hourly update is absolutely necessary. DOAJ is seeing more use of its API than ever before (2018 total: 266,255,000 hits; 2017 total: 187,212,674) and most of that is by publishers uploading articles.

DOAJ metadata is EVERYWHERE. It is incorporated into all the major discovery services, other online databases and many library catalogues.

You might remember this post of mine from 2017, where I explained that third parties using our data may not have a recent copy of it. We know for a fact that some systems downstream of our data only update the copy every 6 months and, sadly, we get many emails from end users asking why they can’t find an article.

So if you are a researcher, reading this and wondering what happened to that article in DOAJ which is listed in your library catalogue or discovery service, contact your librarians and ask them to update their metadata. If you’re a librarian then do please reach out to sources responsible for your metadata and get them to update their DOAJ metadata.

If you have any questions about this or need some help in getting a more recent copy of our metadata then do leave a comment here or contact us at feedback@doaj.org.

El DOAJ, independencia y la importancia de la imparcialidad

El DOAJ recibe regularmente preguntas y, a veces, quejas de bibliotecas, consorcios de bibliotecas y otras instituciones académicas sobre el papel que desempeñan en el DOAJ las editoriales académicas tradicionales. Un error común es que el DOAJ es propiedad de estas organizaciones, o está totalmente subvencionado por ellas, y que el DOAJ sólo beneficia a estas organizaciones. Si usted mirara la página de inicio del DOAJ hace dos años, podríamos entender esta suposición: casi todos nuestros patrocinadores eran editoriales y estaban en nuestra página de inicio. Parecía que el DOAJ era propiedad de estas editoriales.

(Evitar esa idea errónea es también la razón por la cual el DOAJ es muy cuidadoso sobre con quién se asocia). Las organizaciones con las que nos asociamos deben compartir los mismos valores que el DOAJ, tener la misma visión y al menos tener la intención de proporcionar servicios a la comunidad. El DOAJ se enorgullece de asociarse con organizaciones como Redalyc, SciELO, ISSN y COPE.

La suposición de que el DOAJ es propiedad de las editoriales o está totalmente subvencionado por ellas es, por supuesto, incorrecta. Las editoriales desempeñan un importante papel financiero en el apoyo de los servicios que el DOAJ proporciona a todas las partes interesadas. Algunas editoriales nos donan un patrocinio anual; algunas de ellas son miembros de DOAJ como editorial. Estas contribuciones permiten al DOAJ proporcionar servicios continuos, y mejoras a estos servicios. En 2017, las contribuciones de las editoriales representaban el 40% de los ingresos, mientras que las contribuciones de las instituciones del sector público representaban el resto (60%). En 2018, los ingresos de las instituciones del sector público representarán el 70%. Todas las contribuciones al DOAJ son voluntarias; todos los servicios proporcionados por el DOAJ, incluyendo la evaluación de revistas, son gratuitos.

El DOAJ es útil para bibliotecarios, para editores, para investigadores, para estudiantes, para propietarios de revistas, para todos. Más del 50% de nuestro consejo asesor proviene de la comunidad de bibliotecas y consorcios. El DOAJ es global y no está atado a fronteras geográficas. Somos una organización virtual que emplea a personas de todo el mundo y que cuenta con nuestros propios embajadores que fomentan las mejores prácticas en sus territorios de origen. El DOAJ es 100% independiente. El DOAJ es 100% imparcial. La compañía holding del DOAJ, IS4OA C.I.C., está registrada de una manera que hace imposible que el DOAJ sea comprado, adquirido o vendido.

La imparcialidad juega un papel importante en el progreso que el DOAJ ha hecho en los últimos 5 años y el equipo del DOAJ trabaja duro para asegurar que la imparcialidad sea lo más importante en todo lo que hacemos. Esta es una de las razones por las que el DOAJ se adhiere y alienta a otros a adherirse a la transparencia y las buenas prácticas; mejores prácticas que se desarrollan, adoptan y reconocen a nivel internacional. También reconocemos que la aplicación de estas normas y el funcionamiento dentro de ellas puede ser un reto, por lo que tenemos que mantener cierta flexibilidad. Un buen ejemplo sería la concesión de licencias.

licensing

Ejemplo de información sobre licencias de una revista capturada en DOAJ.

A veces nos preguntan: ¿por qué el DOAJ acepta las licencias más restrictivas cuando la definción BOAI (definición a la que el DOAJ se adhiere como uno de sus principios) es muy clara sobre lo que significa “abierto”?

Cuando el DOAJ elaboró su formulario de solicitud ampliado en 2013, vimos muy claro que el DOAJ tenía que aceptar las 6 variaciones de las licencias CCBY, con sus distintos grados de apertura para asegurarnos que el mayor número posible de revistas pudieran solicitar el ingreso en el DOAJ. Al usuario típico del DOAJ se le deben presentar opciones y se le debe permitir tomar una decisión informada basada en la información que mostramos sobre las políticas de licencias y derechos de autor de una revista.

Sin embargo, el DOAJ considera el uso de las licencias CC, en particular el uso de las licencias más abiertas, como una de las mejores prácticas y las promueve como uno de los criterios para el Sello DOAJ.

En 2017 el DOAJ fue el noveno en una lista mundial de plataformas que apoyan el uso de licencias Creative Commons. Creemos que esto es el resultado directo de nuestra preferencia por las licencias CC y la influencia del Sello DOAJ.

Para que una iniciativa como el DOAJ funcione, debe seguir siendo lo más relevante posible en todo el mundo. Debe seguir siendo imparcial e independiente.