DOAJ as an information retrieval system: A LIS students’ perspective

This post is a summary of the graduation research project undertaken by Esraa Kamal, Esraa Abo El Magd, Rawan Ashraf and Yasmine Essam – senior students of the Department of Libraries, Archives, and Information Science at Cairo University in Egypt. The research evaluated DOAJ as an information retrieval system and focused on search and recall capabilities, in addition to using some metrics to evaluate information retrieval.

The Directory of Open Access Journals (DOAJ) has emerged as a prominent reference tool and a key player in the open access movement. It serves as a gateway to peer-reviewed scientific journals, aiming to enhance unrestricted dissemination and accessibility of scholarly knowledge. This evaluation assessed the extent to which DOAJ adheres to quality and scholarly publishing standards, highlighting the strengths and suggesting some development strategies to improve the platform.

Objectives

The study aimed to evaluate DOAJ by analyzing its performance, inclusion standards, and the quality of services provided to scholars and the academic community. The key objectives included:

• Analyzing the nature and role of DOAJ in supporting open access to scientific information.
• Evaluating its compliance with international standards for scholarly publishing.
• Assessing the quality of services and information provided to researchers and library professionals.
•  Identify strengths and weaknesses in the platform and examine challenges that affect its internal effectiveness.
• Offering recommendations to enhance the system’s performance and maximize its impact on the academic and research community.

Scope

The assessment is limited to peer-reviewed scholarly journals that are indexed in DOAJ, ensuring a concentrated analysis of academically rigorous content. The evaluation is grounded in the most recent available version of the DOAJ at the time of the study, providing an up-to-date perspective on the platform’s standards and practices.

 Methodology

The study evaluated DOAJ according to direct and indirect metrics. The direct metrics include: recall, precision, response time, user interface, scope, quality of information, and usability. The indirect metrics include:

·       Recall@K: Measures the proportion of relevant items retrieved in the top K results, using the following formula: Recall@K= # of relevant items in top K​ / Total # of relevant items.

·       Mean Reciprocal Rank (MRR): Averages the reciprocal of the rank of the first relevant result across queries.

·       Mean Average Precision@K (MAP@K): Averages the precision at each position K where a relevant item occurs, across all queries.

Evaluation results

The evaluation of DOAJ from an operational and performance perspective reveals a well-structured, globally inclusive platform offering access to over 21,000 peer-reviewed scholarly journals in 89 languages, supported by a simple yet effective user interface and robust search tools. Administered under UK and Danish regulation, DOAJ provides comprehensive journal metadata, user support, and free access without requiring special software. Direct evaluation metrics show a low precision rate of 2.3%, reflecting a limited proportion of relevant results, but a high recall rate of 80.2%, indicating strong retrieval of actually relevant documents. Coverage analysis suggests timely indexing of relevant articles, although references to 2025 may indicate either projections or inconsistencies. Indirect metrics further affirm the platform’s retrieval effectiveness, with a recall@K of 80.2%, a Mean Reciprocal Rank indicating that users may need to scan several results before finding relevant content, MRR average was 1.5. The high Mean Average Precision@K (94%), suggesting that relevant articles are well-ranked once retrieved.

The Operational Features and Services Perspective

·       Administration: DOAJ has a strong institutional oversight under Regulation 1540A (UK), and was recently registered in Denmark.

·       Scope: DOAJ covers more than 21,000 scholarly journals and 11 million articles in 89 languages, and it extends to all countries worldwide.

·       Retrieval: Simple and advanced search tools are available, including filtering by language, country, publisher, date added, etc.

·       User Support: DOAJ provides help, contact, frequently asked questions, and a search guide.

·       Quality of Information: DOAJ provides comprehensive information about journals and articles and is characterized by accuracy, comprehensiveness, modernity, and simplicity.

·       Facilities and Costs: DOAJ is free, easy to use, and does not require special software.

·       User interface: An attractive and simple interface supports browsing and searching, but journals lack cover pictures.

Direct Evaluation Metrics:

a. Precision

In the study, 16,900 articles were retrieved from the DOAJ when searching for content related to “Artificial Intelligence”. However, only 394 of those were actually relevant to the topic. This means the precision rate was just 2.3%, or about 2 out of every 100 articles. While this is a low percentage, it’s important to note that DOAJ only includes open-access journals, which limits the range of results compared to broader platforms like Google Scholar.

b. Recall

DOAJ performed well in terms of recall. Out of the 492 articles that were considered truly relevant, DOAJ successfully retrieved 394 of them. This gives a recall rate of 80.2%, indicating that the platform is good at finding most of the important articles on the topic. In other words, while DOAJ may bring up many unrelated results, it still manages to include a high number of the right ones.

c. Coverage

The study identified a total of 523 articles related to “Artificial Intelligence,” showing that the topic is well-covered in the database, especially when focusing on open-access content. Interestingly, the data included articles up to the year 2025, which is unusual. This could suggest the study involved future projections or it may simply be a mistake in the data. Normally, research databases focus on articles published in the past or present.

Indirect Evaluation Metrics:

a. Recall@K

This metric measures how many relevant documents the system successfully found out of all the documents that meet a certain quality standard. In this case, the system retrieved 394 high-quality documents, out of the 492 that were actually relevant. This gives a recall rate of 80.2%, showing that the system is effective at capturing a large share of the useful information. It reinforces the earlier observation that DOAJ is strong in finding relevant content, even if not every result is perfect.

b. Mean Reciprocal Rank (MRR)

MRR looks at how early in the search results the first relevant document appears. For example, if the first useful article shows up in the 5th position, the score would be 1/5, or 0.2. This tells us how much effort a user might need to put in before finding something helpful. A lower rank (closer to the top of the list) is better, as it means users get relevant results faster. In this case, the example of 1/5 suggests users might need to look through a few entries before hitting on something truly useful.

c. Mean Average Precision@K (MAP@K)

MAP@K is a measure of how well the system ranks relevant documents among the top results. A high MAP@K score means that when relevant articles are retrieved, they’re typically placed near the top of the list. In this study, the MAP@K score was an impressive 94%, which indicates that once relevant documents are found, they are very well ranked. This suggests the system’s sorting or ranking algorithm is working effectively.

Conclusions

DOAJ is a vital reference tool for discovering peer-reviewed open access journals, playing a key role in facilitating global access to academic resources. The evaluation confirms that DOAJ complies with many international quality standards, providing accurate, comprehensive, and easily accessible data. Its user-friendly interface, fast response time, and a high precision rate of 94% significantly enhance information retrieval for researchers. However, the platform has some limitations, including restricted multilingual support, the absence of journal cover images, and a lack of browsing options in Arabic, which may hinder usability for certain user groups.

Recommendations from student evaluation

• Display journal cover images to enhance visual recognition.
• Improve search through AI-powered intelligent algorithms that understand user intent and context.
•  Add auto-suggestion features for keywords and related topics.
• Enhance multilingual support, particularly for underrepresented languages like Arabic and French.
• Develop a recommendation system to suggest similar articles or journals.
• Providing a chatbot feature to provide real-time assistance and answer user queries effectively.

Thanks to senior students Esraa Kamal, Esraa Abo El Magd, Rawan Ashraf and Yasmine Essam, Department of Libraries, Archives and Information Science, Cairo University, Egypt for this contribution – and to Mahmoud Khalifa who organised this post in his role as MENA Ambassador.