article metadata

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

Are you using an up-to-date version of DOAJ metadata? Apparently not.

Did you know that DOAJ’s metadata refreshes every hour? Given how many changes we, the DOAJ Team, make to the database on a weekly basis plus all the article metadata which is continuously being uploaded to us, the hourly update is absolutely necessary. DOAJ is seeing more use of its API than ever before (2018 total: 266,255,000 hits; 2017 total: 187,212,674) and most of that is by publishers uploading articles.

DOAJ metadata is EVERYWHERE. It is incorporated into all the major discovery services, other online databases and many library catalogues.

You might remember this post of mine from 2017, where I explained that third parties using our data may not have a recent copy of it. We know for a fact that some systems downstream of our data only update the copy every 6 months and, sadly, we get many emails from end users asking why they can’t find an article.

So if you are a researcher, reading this and wondering what happened to that article in DOAJ which is listed in your library catalogue or discovery service, contact your librarians and ask them to update their metadata. If you’re a librarian then do please reach out to sources responsible for your metadata and get them to update their DOAJ metadata.

If you have any questions about this or need some help in getting a more recent copy of our metadata then do leave a comment here or contact us at feedback@doaj.org.

Walt Crawford updates his analysis of DOAJ data

Walt Crawford is prolific! As if his first tranche of mega-analysis wasn’t enough work for him, he has released an update to The Gold Open Access Landscape 2011-2014, which I wrote about previously, that includes an initial analysis of the journals that were removed from DOAJ at the shut-down of the Reapplication project.* He completed the update in May 2016: this post is long overdue.

Gold Open Access Journals 2011-2015, sponsored by SPARC, ‘provides an empirical basis for evaluating Open Access sustainability models‘. The rest of the study is based entirely on DOAJ, refining and updating the previous work. To give you a taster of what was achieved, Walt visited the web site of every journal indexed in DOAJ, even doing a second pass at those sites who hadn’t posted any content the first time around. In the age of analysing large metadata sets programmatically, with computers, this is a huge feat and one that I admire greatly.

What did Walt’s analysis reveal? Well, you can buy the paperback, a snip at only $6, directly from Walt and you can download the PDF free of charge too.

What did DOAJ learn from Walt’s analysis? We hadn’t realised quite how heavily Japan had been hit by the removal: 73% of the Japanese journals indexed in DOAJ were wiped from the Directory. We have since determined that most of those are on J-STAGE and have started a targeted campaign to get as many of those journals to submit new applications as possible. Singapore, Bangladesh and Nepal were also heavily hit. We also learned that journals published by universities and colleges suffered heavy losses. We are looking into this area with our DOAJ Ambassadors to see if we can find out why these losses were so great.

I can thoroughly recommend Walt’s work, not only for the obvious reason that he focusses so greatly on the [great!] DOAJ data but because, to my knowledge, no-one else is doing this type of analysis with such granularity.

As I write, I see that Walt has released an analysis of Open Access by Country 2011 to 2015, the 3rd part of his Gold Open Access Journals trilogy. As always, you can purchase the paperback or download the PDF for free.

Now, get reading!

*The final figure was 2851 journals removed and not 3300 as we had originally anticipated. The discrepancy was because we included, in the original count, journals that had already been removed from DOAJ and therefore had a reapplication pending but which would never be submitted.

A lot goes a long way: data quality improvement at DOAJ

Since 2012, DOAJ has been on a path of data quality improvement. DOAJ metadata is used all over the world and all over the Web. Improving and fixing the quality of our metadata can be painstaking work but the effort goes a long way as changes propagate across the Web via search engines, aggregator databases, library portals and other databases.

Along these lines, the largest publisher (in terms of the number of journals) in DOAJ recently added missing abstracts to over 100,000 articles and fixed broken special characters in approximately 2000 more. This was a huge effort on their part and DOAJ is grateful for the work that has gone into this project. It is an achievement that will be welcomed by DOAJ metadata consumers.

To date, Hindawi has 161,334 articles loaded to DOAJ and until recently was the largest contributor of metadata to our index. That title was taken from them recently when DOAJ ingested the entire PLOS archive from Europe PMC.

DOAJ. Now with added PLOS.

Finally, after saying: “It’s coming” for almost a year, DOAJ has added the entire PLOS catalogue to its database. A total of 182,500 articles were added and the harvester is set to go out and collect new articles at 5.30am every morning. The harvester collects the metadata from Europe PMC.

This is a milestone for DOAJ: not only because it has been eager to get the PLOS article metadata for a long time; but also because the harvester can be re-tuned to go and search for other large bodies of content from other publishers on E-PMC. Eventually, with some extra development, we’ll be able to point the harvester at any body of content.

If you think your publisher would benefit from this approach, do please get in touch! Thanks to our technical partners, Cottage Labs, for their help in reaching this milestone.

NEW: continuations, improved tables of contents, feedback on a journal, APC granularity, and more…

Very soon, we will be releasing new functionality on DOAJ that will improve the detail in the journal record, increase DOAJ’s accuracy even more and will encourage greater input from the community. We have also abandoned our policy to remove journals that have ceased publication.

Continuations

Ever since we migrated to our new platform, we have been trying to find a slot in our development schedule to correctly display journal continuations on our site. It was important for us to get this correct, both in terms of the metadata and the display, so that users could easily understand how one, or several, journals transformed into a journal with a new title and ISSN. This new development will allow us to correctly display a journal’s timeline as it goes through name and ISSN changes.

Send us feedback on a journal

A decreasing amount of the data in DOAJ is old and DOAJ is always grateful to receive updates from users who get in touch to alert us to broken URLs or a change of title. With this in mind, and to make feeding back as simple as possible, we will add a ‘Tell us about this journal’ button to every journal entry in DOAJ. We hope this will encourage greater input from our users and lead to more updates to journal entries. This development is perfectly aligned with DOAJ being curated by the community and really facilitates an up-to-date resource.

Greater granularity for APC information

We know. The APC information on DOAJ was misleading. It lumps together 2 distinct groups of journals into one big ‘No’ group. We have broken that group out into: No, where we have up-to-date information from journals that they DO NOT charge APCs; and No information. ‘No information’ is the group of journals which has submitted a reapplication to us and we have yet to review it. The number of ‘No information’ journals will diminish in size as we work our way through the reapplications. I wrote about this inaccuracy previously.

Retaining the records – journals that have ceased publication

As long as they fulfil some basic criteria, we will no longer remove journals that cease publishing and we will even add back into DOAJ journals that were removed only because they stopped publishing. This change of policy could potentially add hundreds of articles back into DOAJ along with tens of archived journals.

Access from China

We have had several complaints from users in China that they were unable to use DOAJ. We realised that this is because the site uses Google javascript services and Google is banned in some parts of China, or by some institutions. We have now migrated most services off Google-related services (apart from Analytics) and are confident that the user experience for users in China will be vastly improved. This is perfectly times as we embark on a year-long project to increase awareness of open access in China.

Oh, and watch this space… In early June, we’ll be adding ~175 000 PLoS articles to DOAJ… More on that later.

 

DOAJ to remove approximately 3300 journals

UPDATE (11th May): the list of removed journals is here on the 3rd tab.

Today DOAJ will remove approximately 3300 journals for failure to submit a valid reapplication before the communicated deadline; a deadline which was extended twice to allow more time for reapplications. This batch removal is another step in DOAJ’s two year long project to increase the value and accuracy of the information provided in it.

Here are some details about the reapplication project from its launch in January 2015 to today:

  • The reapplication process is a necessary step towards ensuring that all journals in DOAJ (of which there were about 10000) met the higher criteria for indexing that DOAJ launched in March 2014. The criteria were produced as a response to the maturing open access arena, the greater demands made on open access publishing by questionable journals and publishers, and to retain DOAJ’s relevancy and importance in open access publishing.
  • Some journals have been in DOAJ since 2003 and have never refreshed their information with us.
  • As of today over 5000 journals have already submitted their reapplication to us and we are busy assessing those. Many reapplications have been accepted back into DOAJ.
  • The contact for every journal to be removed from DOAJ was emailed at least 4 times, informing them of our intention to remove their journals if they failed to submit a reapplication by the agreed deadline.
  • We send email via Mailchimp and took all the necessary precautions to ensure that our emails didn’t end up in Spam, get trapped in institutional firewalls, or failed to deliver for other reasons. We used the Mailchimp authentication options to “verify” that our emails were from a genuine source.
  • The first email, announcing the reapplication project and inviting people to reapply, was sent out in January 2015 and went to publishers with 11 or more journals in DOAJ. The second email went out to publishers with 10 or less journals in DOAJ in June 2015.
  • Reminders were sent out regularly, once a month as well as announcing the deadline to our largest communities: via this blog, Twitter and Facebook.
  • To ensure that our emails ended up with the correct contact, we spent a considerable amount of time tidying up our contacts database: we updated at least 1000 records.

Removed journals are welcome to submit a new application to DOAJ at any time. They will be placed in the queue along with other applications. We will add a third tab to our spreadsheet ‘DOAJ: journals added and removed‘ that will list all of the journals removed.

When a journal is removed from DOAJ, any article metadata will also become unavailable. This is standard functionality. We are confident that the majority of the journals removed have never supplied article metadata to us, or have done once but haven’t sent us anything for at least 2 years.

If you use DOAJ as a data source and would like to do your own analysis of the journals indexed,  download our journals CSV (https://doaj.org/csv) today before 11am BST, 12pm CEST. A copy of that spreadsheet is also available here.