database

Full DOAJ data dump now available

“We’re going to a hackathon and would love to work with DOAJ’s data! Do you have a dump of all DOAJ data?”

“Er, no. Sorry. I mean, we could get it for you but it will take a while. You could probably get it yourself if you know how to extract the JSON… When do you need it for?”

“Tomorrow.”

“Oh.”

We’ve had this conversation a few times. Or had requests from eager individuals and organisations who want to use the rich offerings of the DOAJ metadata. They’ve told us of the wonderful things they could do with the data (slicing, reporting, analysing, apps, databases, software…) and we’ve never been able to help them in good time. But now, we can….

Introducing, full data dumps of ALL the metadata in DOAJ, both journals and articles: https://doaj.org/public-data-dump

So what? Well, let me suggest to you why these are a good thing:

  • For the journal metadata, the CSV is really the only easy-to-use format. The journal data dump provides another way to do this.
  • The data dumps are updated weekly, so can keep you up-to-date on a reasonably short delay. (There is no change feed, just a full dump.)
  • When you want all of the DOAJ data for any reason, you can just take it!
  • Deep paging on the search API is no longer permitted – search is for search, not harvesting. The data dump allows you to harvest.
  • Whenever you want a subset of the DOAJ data, you can just download the data dump, then filter it locally for your needs. For example, if you are a publisher and you want to see all of your metadata in DOAJ, that is all in this data dump, and you can then filter by ISSN
  • You can use it to enhance any local data in your own system or database: you may have basic article metadata in your system, and you want to extend it with DOAJ metadata.
  • If you want to aggregate publications data from multiple sources, this is one way of quickly getting that information from DOAJ (versus using OAI-PMH).
  • These data dumps are more metadata rich than OAI-PMH
  • You may want to use the data for analysis or data mining or other forms of research, or hackathons.
  • The data dumps are also useful as a test dataset.

So there you have it. Despite a rather awkward name, data dumps are A Good Thing.

We’d love to know what you think, so do please leave a comment here or send us feedback: feedback@doaj.org

Parts of DOAJ metadata in my catalogue/discovery service/A-Z List have changed, are out-of-date or are missing completely. Why?

It’s time to update your index and/or ask your discovery service provider to do so!

We have received  several messages via our Feedback account from library catalogue administrators, their readers, and researchers asking why all of a sudden so many of the articles indexed from DOAJ are now missing? Within library catalogues, users are perhaps seeing a 404 message. On DOAJ itself, users are noticing that the article content included in DOAJ is much changed and reduced. Why is this?

Hopefully it is old news to you now that DOAJ has been undertaking its Reapplication project. Over the last 6 months, the DOAJ Team has made a final, huge push to process all the reapplications. This project is so very nearly complete: out of the 6359 reapplications submitted to us, only a handful are still to be re-checked. 2072 of the reapplications did not pass our new standards. This means that the range of journals in DOAJ has changed considerably; very quickly over a relatively short period of time.

Aggregators / Service Providers / Library Catalogue administrators: we strongly advise you to re-index / update / download again the DOAJ article metadata offering—via OAI-PMH, or our API—so that your catalogues have the most up-to-date version of our index. The DOAJ index changes weekly so it might be time to revisit how often you come to us for updates and increase that frequency. Learn more about how to interact with DOAJ.

If you are using the search or browse features at DOAJ, then be aware that some of the journals you are used to seeing in the Directory may no longer be there.

If you have any questions about the effect of the reapplication project on DOAJ, about indexing DOAJ via our API or OAI-PMH services, or even how to search and browse, then please leave a comment here or send email to feedback@doaj.org.

Updates to DOAJ will be switched off while we upgrade the site

Earlier this month I wrote about the twelve developments that we were working on. These are now ready to be released to the live site this week and we will be making the upgrade between Tuesday (21st April) evening and Wednesday (22nd April) morning British Summer Time (GMT+1). During this time, we will switch off all parts of the site that allow new data to be submitted to us:

  • the application form
  • the form for adding article metadata
  • the XML upload page for uploading article metadata
  • the CSV upload page for submitting bulk reapplications
  • the reapplication form

All other aspects of the site will continue to function as normal. We expect a full service to be reinstated by Wednesday 12 noon BST. I will post further updates here.