Inside the Quality Team – Part 1: how we preserve quality at DOAJ

At DOAJ, we are dedicated to upholding trust in scholarly publishing by ensuring that questionable journals are not included in our directory. Our quality team is central to this effort, conducting in-depth investigations into cases of concern. In 2024 alone, the team dedicated more than 830 hours on over 470 investigations. Our Deputy Head of Editorial for Quality, Cenyu Shen, and our Managing Editor and member of the Quality Team, Paula Marjamäki, give us some insights into this important DOAJ function.

(This is Part One of a two-part blog post. Part Two will be published soon and linked to here.)

Quality team and our work

At DOAJ, we are dedicated to upholding trust in scholarly publishing by ensuring that questionable journals are not included in our directory. Feedback from our recent community survey reaffirms that users regard DOAJ as a reliable source of trustworthy and high-quality open access journals. This recognition has heightened expectations for DOAJ to take an increasingly active role in maintaining quality standards.

Our quality team is central to this effort, conducting in-depth investigations into cases of concern. This work is fundamental to advancing our mission of promoting the visibility, accessibility, reputation, usage and impact of quality, peer-reviewed, open access journals worldwide. By curating a directory that prioritizes quality, we help to enhance the reputation of listed journals and support researchers, libraries, and funding bodies in identifying dependable sources for publishing research and verifying the legitimacy of open access journals.

When concerns regarding a journal or publisher’s quality and publishing practices arise during our regular review process or are flagged by community members, such cases are escalated to the quality team, which carefully verifies and investigates them. As publishing ethics and research integrity gain increasing attention in the evolving landscape of scholarly communication, the scope and complexity of our investigative work continue to grow. In 2024 alone, the team dedicated more than 830 hours to over 470 investigations.

Among all investigations, journals and publishers previously identified as having questionable practices are also re-evaluated if they apply again. These repeated investigations account for more than 26% of our total cases. In most instances, a lack of improvement or the escalation of questionable publishing practices leads to the decision to maintain their exclusion from our index.

How does the quality team evaluate journals?

The quality team’s investigation is more thorough and time-intensive than the standard application review. It focuses on the five key areas outlined below.

1. Presence of clear and reliable key information on the journal’s website
   The journal’s website should clearly present information about the editorial board, their affiliations, the journal’s ownership, and accurate contact details. It should also provide transparent details about its publisher, including the country of operation, funding models, and related business activities. Editorial policies should be prominently displayed, original, and not copied from other publishers.
   Concerns arise when key information is missing — for instance, if the publisher is not based where it claims, affiliations or partnerships with legitimate organizations are false, or if a journal claims large operations without clear funding sources, raising questions about the sustainability of its business model.
   
2. Composition of the journal’s editorial board
   The editorial board list should be current, and members’ credentials should be carefully verified. Including individuals with fake affiliations or links to questionable journals casts doubt on the journal’s due diligence in selecting its editorial board. Other red flags include a board composition that does not align with the journal’s scope and target audience, a lack of international representation despite claims of being an international journal or a board size that is inappropriate for the journal’s publishing volume and the range of subjects it covers.
   
3. Peer review process
   A journal’s peer review process must be thorough and robust to ensure the reliability of its published content. Issues such as appointing unqualified reviewers, assigning reviews outside their area of expertise, or failing to address conflicts of interest undermine the integrity of the process. Inadequate review timelines can also compromise the quality of their assessments.
   The Editor-in-Chief should play a central role in overseeing editorial decisions to ensure fairness and effectiveness in evaluations. Indicators of an inadequate peer review process include the publication of articles outside the journal’s scope or other signs of poor review standards.
   
4. Reputation of the journal
   Displaying “self-created” impact factors, using unverified metrics or false claims to be indexed in recognized scholarly databases, is a sign of questionable practices. Journals should avoid misleading information or any other non-standard metrics used to artificially inflate its reputation.
   
5. Journal management and operation
   The journal should maintain rigorous oversight of the entire publishing process. Signs of failure in this area include, for example, a lack of clear editorial independence from the journal’s administrative operations, which can lead to external influence and conflicts of interest.

In addition to conducting our own investigations, as a small team, we also rely on collaborations with our community and other organizations working in the same field. Their feedback is vital to us in identifying potential issues with journals, enabling us to respond more swiftly and comprehensively to questionable publishing practices. Users can submit feedback to the helpdesk (helpdesk@doaj.org) or via the widget on the DOAJ website and our social media. Please note, however, that we cannot investigate individual disputes regarding editorial decisions or other cases beyond our defined scope.

Maintaining the integrity of our index by preventing the inclusion of questionable journals is, and always will be, our foremost responsibility — a responsibility that has grown even more critical in today’s publishing landscape. As we navigate this evolving environment, we strive to remain adaptable in our role, committed to safeguarding trust in the scholarly publishing ecosystem.