article metadata

Article uploads: the importance of an ISSN

This is a reminder that in February this year we tightened up procedures for article uploads. This includes checks that are carried out on uploads particularly ISSN, DOI, and full text URL information.

This means that, whether uploading article XML to us or JSON via the API, the metadata must include the exact same ISSN value(s) as those we hold in the journal record in DOAJ. If you have two ISSNs on the website and we only have one, you should contact us to have the second ISSN added. Usually, this is a quick process but sometimes you might need to contact ISSN.org to have them update their information before we can continue.

If you have any questions on this, do contact us.

We’ve added 2 new columns to the journal CSV.

In a series of enhancements to article metadata, we have released the third today.*

If you are familiar with DOAJ metadata then you probably already know that you can download a CSV file which contains the journal-level metadata for all the journals in DOAJ. It is updated every 30 minutes and is probably one of the most popular metadata services we have. It is certainly the best way to get an overview of all the journals in DOAJ.

Today, we’ve released a new version of that file which adds two new columns to it: the number of article records added to a journal in DOAJ and the date that the last article was added. (Columns BF and BG respectively)

The columns were added for two reasons:

  1. To give greater transparency to the information which we display on our homepage whcountere we state that 77% of the journals in DOAJ have article content loaded to them. This is slightly misleading because a journal which uploaded only 5 articles to us in 2013 is counted in that 77%. It is more interesting to know how recent the articles are from a particular title and how much content they have uploaded to us. While this information has always been available by selecting a journal ISSN and using the ‘articles’ filter in Search, we’ve never been able to show all the information in one place.I think that this development will be welcomed by all our users, especially publishers, librarians and those doing research on open access publishing developments.
  2. To allow us to review those journals which have been awarded the DOAJ Seal and remove that Seal from those which are not supplying article metadata to us. Supplying article metadata is one of the 7 Seal criteria and we haven’t yet been able to check, in an efficient manner, which journals are sticking to their promise. [In the application form, we ask if journals “intend” to supply metadata to us.] It’s going to take deeper analysis to get the final figure but I can see very quickly that 25 journals are going to lose their Seal.

I’d love to know what you think about this development so, as usual, do leave a comment or a question, or email me directly: dom@doaj.org.

P.S. The other development which I will post about soon is the removal of duplicated articles from the DOAJ database. More on that when I have it.

*The first and second developments are described in this blog post from February.

Full DOAJ data dump now available

“We’re going to a hackathon and would love to work with DOAJ’s data! Do you have a dump of all DOAJ data?”

“Er, no. Sorry. I mean, we could get it for you but it will take a while. You could probably get it yourself if you know how to extract the JSON… When do you need it for?”

“Tomorrow.”

“Oh.”

We’ve had this conversation a few times. Or had requests from eager individuals and organisations who want to use the rich offerings of the DOAJ metadata. They’ve told us of the wonderful things they could do with the data (slicing, reporting, analysing, apps, databases, software…) and we’ve never been able to help them in good time. But now, we can….

Introducing, full data dumps of ALL the metadata in DOAJ, both journals and articles: https://doaj.org/public-data-dump

So what? Well, let me suggest to you why these are a good thing:

  • For the journal metadata, the CSV is really the only easy-to-use format. The journal data dump provides another way to do this.
  • The data dumps are updated weekly, so can keep you up-to-date on a reasonably short delay. (There is no change feed, just a full dump.)
  • When you want all of the DOAJ data for any reason, you can just take it!
  • Deep paging on the search API is no longer permitted – search is for search, not harvesting. The data dump allows you to harvest.
  • Whenever you want a subset of the DOAJ data, you can just download the data dump, then filter it locally for your needs. For example, if you are a publisher and you want to see all of your metadata in DOAJ, that is all in this data dump, and you can then filter by ISSN
  • You can use it to enhance any local data in your own system or database: you may have basic article metadata in your system, and you want to extend it with DOAJ metadata.
  • If you want to aggregate publications data from multiple sources, this is one way of quickly getting that information from DOAJ (versus using OAI-PMH).
  • These data dumps are more metadata rich than OAI-PMH
  • You may want to use the data for analysis or data mining or other forms of research, or hackathons.
  • The data dumps are also useful as a test dataset.

So there you have it. Despite a rather awkward name, data dumps are A Good Thing.

We’d love to know what you think, so do please leave a comment here or send us feedback: feedback@doaj.org

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

Are you using an up-to-date version of DOAJ metadata? Apparently not.

Did you know that DOAJ’s metadata refreshes every hour? Given how many changes we, the DOAJ Team, make to the database on a weekly basis plus all the article metadata which is continuously being uploaded to us, the hourly update is absolutely necessary. DOAJ is seeing more use of its API than ever before (2018 total: 266,255,000 hits; 2017 total: 187,212,674) and most of that is by publishers uploading articles.

DOAJ metadata is EVERYWHERE. It is incorporated into all the major discovery services, other online databases and many library catalogues.

You might remember this post of mine from 2017, where I explained that third parties using our data may not have a recent copy of it. We know for a fact that some systems downstream of our data only update the copy every 6 months and, sadly, we get many emails from end users asking why they can’t find an article.

So if you are a researcher, reading this and wondering what happened to that article in DOAJ which is listed in your library catalogue or discovery service, contact your librarians and ask them to update their metadata. If you’re a librarian then do please reach out to sources responsible for your metadata and get them to update their DOAJ metadata.

If you have any questions about this or need some help in getting a more recent copy of our metadata then do leave a comment here or contact us at feedback@doaj.org.

Walt Crawford updates his analysis of DOAJ data

Walt Crawford is prolific! As if his first tranche of mega-analysis wasn’t enough work for him, he has released an update to The Gold Open Access Landscape 2011-2014, which I wrote about previously, that includes an initial analysis of the journals that were removed from DOAJ at the shut-down of the Reapplication project.* He completed the update in May 2016: this post is long overdue.

Gold Open Access Journals 2011-2015, sponsored by SPARC, ‘provides an empirical basis for evaluating Open Access sustainability models‘. The rest of the study is based entirely on DOAJ, refining and updating the previous work. To give you a taster of what was achieved, Walt visited the web site of every journal indexed in DOAJ, even doing a second pass at those sites who hadn’t posted any content the first time around. In the age of analysing large metadata sets programmatically, with computers, this is a huge feat and one that I admire greatly.

What did Walt’s analysis reveal? Well, you can buy the paperback, a snip at only $6, directly from Walt and you can download the PDF free of charge too.

What did DOAJ learn from Walt’s analysis? We hadn’t realised quite how heavily Japan had been hit by the removal: 73% of the Japanese journals indexed in DOAJ were wiped from the Directory. We have since determined that most of those are on J-STAGE and have started a targeted campaign to get as many of those journals to submit new applications as possible. Singapore, Bangladesh and Nepal were also heavily hit. We also learned that journals published by universities and colleges suffered heavy losses. We are looking into this area with our DOAJ Ambassadors to see if we can find out why these losses were so great.

I can thoroughly recommend Walt’s work, not only for the obvious reason that he focusses so greatly on the [great!] DOAJ data but because, to my knowledge, no-one else is doing this type of analysis with such granularity.

As I write, I see that Walt has released an analysis of Open Access by Country 2011 to 2015, the 3rd part of his Gold Open Access Journals trilogy. As always, you can purchase the paperback or download the PDF for free.

Now, get reading!

*The final figure was 2851 journals removed and not 3300 as we had originally anticipated. The discrepancy was because we included, in the original count, journals that had already been removed from DOAJ and therefore had a reapplication pending but which would never be submitted.

A lot goes a long way: data quality improvement at DOAJ

Since 2012, DOAJ has been on a path of data quality improvement. DOAJ metadata is used all over the world and all over the Web. Improving and fixing the quality of our metadata can be painstaking work but the effort goes a long way as changes propagate across the Web via search engines, aggregator databases, library portals and other databases.

Along these lines, the largest publisher (in terms of the number of journals) in DOAJ recently added missing abstracts to over 100,000 articles and fixed broken special characters in approximately 2000 more. This was a huge effort on their part and DOAJ is grateful for the work that has gone into this project. It is an achievement that will be welcomed by DOAJ metadata consumers.

To date, Hindawi has 161,334 articles loaded to DOAJ and until recently was the largest contributor of metadata to our index. That title was taken from them recently when DOAJ ingested the entire PLOS archive from Europe PMC.