What is a scholarly journal? A case study of applying the DOAJ criteria

Matt Hodgkinson, Head of Editorial, DOAJ

Matt Hodgkinson, Head of Editorial at DOAJ, discusses DOAJ’s inclusion criteria and a recent decision to remove Verfassungsblog from the index. He explains the criteria, their basis, and the reassessment process, and invites feedback from the community.

At DOAJ, we don’t usually publicly discuss our decisions whether to index a journal. We are making an exception here, with the agreement of Verfassungsblog, to invite discussion from the community about how we apply our criteria and whether and how we might revise them to be more inclusive.

This case study highlights some of the boundary work we do at DOAJ: What is a scholarly journal? What counts as research? What review process is sufficient and does this vary by field? How do we apply our criteria fairly, globally and across all fields?

This post is a response to their own, which shared our decision letter with our permission: Evin Dalkilic, Access Denied: Verfassungsblog, DOAJ, and the Meaning of “Scholarly”, Verfassungsblog, 13 June 2025

What happened

Verfassungsblog applied for DOAJ indexing in 2020. We discussed their processes with them and accepted the application in September 2021.

We have a process to reassess indexed journals and late last year we rechecked all journals with average submission-to-publication times of under three weeks. Verfassungsblog was among these and was discussed by the Quality Team in January 2025; this was my first meeting as Head of Editorial. We decided to remove the journal, with the possibility of applying again in six months (the standard delay when rejected or removed from DOAJ) or appealing the decision. As part of the same reassessment process, we removed another blog publication at around the same time.

Verfassungsblog appealed in May and the Appeals Committee decided to uphold the decision. We allow only one appeal per decision.

How we applied our criteria

In 2021, we were concerned about the appearance of the publication as a blog rather than a journal and the lack of details around peer review. However, we considered the response satisfactory at the time.

In 2025, the Quality Team and Appeals Committee were both unanimous in finding that the publication does not meet the DOAJ criteria. We based this on the following:

1. Not a research journal (“The journal must be actively publishing scholarly research”). We logged this as “Has not published enough articles this calendar year”, meaning insufficient research articles; the categories we use for removals are fixed and limited, something we’re looking at changing;
2. Not peer reviewed (“All articles must pass through a quality control system (peer review) before publication. The type and details of the peer review process must be stated clearly on the website, including the process for special issues, if applicable. At least two independent reviewers should review each article.”);
3. A lack of scholarly referencing (though not explicitly articulated in our criteria, this is a standard expectation for scholarly journals – see below for more details);
4. A lack of clarity around publication fees, which has since been resolved.

We allow an exception to peer review for arts and humanities journals, for which editorial review is allowed. However, law is not considered an arts and humanities field by the UNESCO definitions we follow: “Arts and humanities journals. For these disciplines (pages 10-16 only), DOAJ can accept journals that undertake editorial review, rather than peer review. Editorial review must be handled with a minimum of two editors.” All the German law journals we index use peer review and have reference lists.

What went wrong

DOAJ should not have indexed the publication in 2021, at least without a more in-depth discussion. We’ve since improved our processes to have wider discussion of such unusual cases within the team.

Since January, we’ve introduced a process to allow diamond journals more time to make changes and clarify their processes before we proceed with removal. Although we believe the decision would have been the same, the process may have been less abrupt.

What this decision means

We believe that Verfassungsblog doesn’t meet our current criteria for indexing. This doesn’t mean it has no value for the legal and academic communities, but rather that it falls outside DOAJ’s scope.

The Conversation is a similar open access academic journalism site with an editorial process; despite its clear value, we believe it would likewise fall outside our scope were it to apply. Books also fall outside our scope and are indexed by DOAB.

We understand that some institutions and communities use DOAJ indexing as a criterion for funding decisions, but rejection or removal from the index is not always based on poor practice but rather on not being the kind of publication we consider. We hope funders will bear this in mind.

Open questions

In some ways it would be easier to regularly allow exceptions to the criteria, but then we would face reasonable accusations of favouritism and inconsistency; it is likely that more exceptions will come from the Global North. When we consider important edge cases like this one, we want to be able to articulate the principles on which a decision is based and revise or add to our criteria if necessary so that we can apply them again in future. If we change or disregard our criteria without careful thought, we risk opening ourselves up to unintended consequences such as allowing in questionable journals without concern for scholarship and ethics.

Our Editorial Policy Advisory Group, which I was part of until I started as Head of Editorial, is a group of volunteers who regularly advise us on these kinds of policy questions.

Here are the open questions that this case highlights.

1. Should DOAJ be limited to research journals?

Since joining DOAJ, I have become aware that our current criteria inadvertently exclude journals which are valuable to the scholarly community, i.e., scholarly journals that do not publish research.

Current criteria:

• The journal must be actively publishing scholarly research
  • Any research subject area
  • Publish at least five research articles per year
  • Primary target audience of researchers or practitioners

As the first step towards widening the scope towards all scholarly journals, at the end of June DOAJ updated its criteria for clinical case reports journals, which we already accepted, to recognise that they do not need to publish research. We have until now required that they publish at least five case series per year (which we considered to be research articles), but we have decided this was an unnecessary barrier. DOAJ will now be considering them purely as case reports journals, rather than trying to fit a square peg into a round hole.

We already have additional criteria for some other types of journals: conference proceedings journals; data journals; overlay journals; student-run journals; flipped journals; and, as of the end of June, Subscribe to Open (S2O) journals. The next step will be to develop criteria for review journals and develop a wider definition of a scholarly journal. What other types of scholarly journals are there that we should consider and develop guidance for?

2. When should peer review be required?

We know peer review isn’t perfect, but it is the recognised norm in most scholarly fields. Editorial selection of scholarly articles was standard until the late 20th century — Einstein was offended by his work being peer reviewed — but a series of scientific and medical scandals in the 1960s and 1970s led to academics adopting peer review, i.e., independent assessment by invited experts. A further scandal in 2009 led to editorial selection being dropped at the journal Medical Hypotheses in favour of peer review.

As discussed above, we have an exception to allow editorial review in the arts and humanities – is that exception still reasonable? If so, should it be widened and to what fields (such as law), settings, or types of journal or article?

3. What is a scholarly journal?

Our decision in this case was based in part on the definition of a scholarly journal, though our guide to applying does not explicitly define this. A scholarly journal has an ISSN, editorial oversight, and (usually) peer review, and is aimed at a scholarly or academic audience, i.e., those interested in the systematic study of learning and knowledge. Ever since 1665, scholarly journals have provided registration, certification, dissemination, and archiving.

Scholarly articles present, review, or comment on primary research, with authors who have expertise or experience in the field. Written formally and using field-specific terminology, they are structured, including a title, author list, affiliations, abstract, relevant sections such as methods, appropriate declarations such as of ethics, funding, and interests, and a reference list; previous relevant work must be acknowledged and discussed, and sources must be clearly identified.

DOAJ doesn’t index trade and professional journals, popular science magazines, newspapers, newsletters, or abstracts journals. Additionally, closed access journals, free-to-access journals, or hybrid journals are not indexed.

Are these expectations of scholarly journals out-of-date or too inflexible? If so, how should they be widened without allowing in titles that are not academic or scholarly?

Feedback welcome

We value community consultation at DOAJ. Please let us know your thoughts on this case and our criteria in the comments or by email, so we can continue to best serve all users.