New feature

Full DOAJ data dump now available

“We’re going to a hackathon and would love to work with DOAJ’s data! Do you have a dump of all DOAJ data?”

“Er, no. Sorry. I mean, we could get it for you but it will take a while. You could probably get it yourself if you know how to extract the JSON… When do you need it for?”

“Tomorrow.”

“Oh.”

We’ve had this conversation a few times. Or had requests from eager individuals and organisations who want to use the rich offerings of the DOAJ metadata. They’ve told us of the wonderful things they could do with the data (slicing, reporting, analysing, apps, databases, software…) and we’ve never been able to help them in good time. But now, we can….

Introducing, full data dumps of ALL the metadata in DOAJ, both journals and articles: https://doaj.org/public-data-dump

So what? Well, let me suggest to you why these are a good thing:

  • For the journal metadata, the CSV is really the only easy-to-use format. The journal data dump provides another way to do this.
  • The data dumps are updated weekly, so can keep you up-to-date on a reasonably short delay. (There is no change feed, just a full dump.)
  • When you want all of the DOAJ data for any reason, you can just take it!
  • Deep paging on the search API is no longer permitted – search is for search, not harvesting. The data dump allows you to harvest.
  • Whenever you want a subset of the DOAJ data, you can just download the data dump, then filter it locally for your needs. For example, if you are a publisher and you want to see all of your metadata in DOAJ, that is all in this data dump, and you can then filter by ISSN
  • You can use it to enhance any local data in your own system or database: you may have basic article metadata in your system, and you want to extend it with DOAJ metadata.
  • If you want to aggregate publications data from multiple sources, this is one way of quickly getting that information from DOAJ (versus using OAI-PMH).
  • These data dumps are more metadata rich than OAI-PMH
  • You may want to use the data for analysis or data mining or other forms of research, or hackathons.
  • The data dumps are also useful as a test dataset.

So there you have it. Despite a rather awkward name, data dumps are A Good Thing.

We’d love to know what you think, so do please leave a comment here or send us feedback: feedback@doaj.org

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

We have improved our XML validation

xml

We are happy to announce that we have improved the features of our xml validation. As an article XML is uploaded to DOAJ from the publisher area we will check to see if the uploaded file meets a number of criteria:

  • Is it actually XML? (have you given us a PDF by mistake?)
  • Does the XML meet the DOAJ article schema requirements?
If your upload doesn’t meet either of those criteria you will get an error message straight away.  It will either tell you we didn’t think the file was XML, or it will give you a message from our validator which identifies the first problem found when validating.
If your file does pass those checks, it will get entered into our upload queue.  You will need to check back on your upload page in a while to see if the upload was processed successfully.  When we process your upload, we carry out some more checks to make sure that the data in DOAJ remains consistent:
  • Do you reference any ISSNs that belong to journals you don’t own?  You can’t attach articles to someone else’s journals.
  • Do you reference any ISSNs that don’t appear in DOAJ at all?  We only take uploads for journals we know about, and your journal records need to contain all the correct identifiers if you want to reference them.
  • Do you reference ISSNs that are owned by more than one user?  Sometimes it’s possible for the DOAJ to think that more than one user has a stake in an ISSN.  In these cases we need to resolve who the true owner is before you can upload.
If any of these situations arise with your file upload, we won’t import any of the articles, and instead record an error against the upload.  When you look at the upload record on the page you will see a link in the Notes column which says “show error details“.  When you click on that it will tell you in what way the articles failed to process, and which ISSNs were causing the problems.
If you see any other error messages on your upload page, you should click on the “show error details” link if it is available.  If not, you should contact us at feedback@doaj.org and we will get back to you as soon as we can.

 

DOAJ. Now with added PLOS.

Finally, after saying: “It’s coming” for almost a year, DOAJ has added the entire PLOS catalogue to its database. A total of 182,500 articles were added and the harvester is set to go out and collect new articles at 5.30am every morning. The harvester collects the metadata from Europe PMC.

This is a milestone for DOAJ: not only because it has been eager to get the PLOS article metadata for a long time; but also because the harvester can be re-tuned to go and search for other large bodies of content from other publishers on E-PMC. Eventually, with some extra development, we’ll be able to point the harvester at any body of content.

If you think your publisher would benefit from this approach, do please get in touch! Thanks to our technical partners, Cottage Labs, for their help in reaching this milestone.

NEW: continuations, improved tables of contents, feedback on a journal, APC granularity, and more…

Very soon, we will be releasing new functionality on DOAJ that will improve the detail in the journal record, increase DOAJ’s accuracy even more and will encourage greater input from the community. We have also abandoned our policy to remove journals that have ceased publication.

Continuations

Ever since we migrated to our new platform, we have been trying to find a slot in our development schedule to correctly display journal continuations on our site. It was important for us to get this correct, both in terms of the metadata and the display, so that users could easily understand how one, or several, journals transformed into a journal with a new title and ISSN. This new development will allow us to correctly display a journal’s timeline as it goes through name and ISSN changes.

Send us feedback on a journal

A decreasing amount of the data in DOAJ is old and DOAJ is always grateful to receive updates from users who get in touch to alert us to broken URLs or a change of title. With this in mind, and to make feeding back as simple as possible, we will add a ‘Tell us about this journal’ button to every journal entry in DOAJ. We hope this will encourage greater input from our users and lead to more updates to journal entries. This development is perfectly aligned with DOAJ being curated by the community and really facilitates an up-to-date resource.

Greater granularity for APC information

We know. The APC information on DOAJ was misleading. It lumps together 2 distinct groups of journals into one big ‘No’ group. We have broken that group out into: No, where we have up-to-date information from journals that they DO NOT charge APCs; and No information. ‘No information’ is the group of journals which has submitted a reapplication to us and we have yet to review it. The number of ‘No information’ journals will diminish in size as we work our way through the reapplications. I wrote about this inaccuracy previously.

Retaining the records – journals that have ceased publication

As long as they fulfil some basic criteria, we will no longer remove journals that cease publishing and we will even add back into DOAJ journals that were removed only because they stopped publishing. This change of policy could potentially add hundreds of articles back into DOAJ along with tens of archived journals.

Access from China

We have had several complaints from users in China that they were unable to use DOAJ. We realised that this is because the site uses Google javascript services and Google is banned in some parts of China, or by some institutions. We have now migrated most services off Google-related services (apart from Analytics) and are confident that the user experience for users in China will be vastly improved. This is perfectly times as we embark on a year-long project to increase awareness of open access in China.

Oh, and watch this space… In early June, we’ll be adding ~175 000 PLoS articles to DOAJ… More on that later.

 

Widgets! Embed DOAJ directly into your site.

Happy New Year to all of you! We have a gift: two shiny widgets! You can now embed a set of search results or a DOAJ search box directly into your web sites.

Embedding search results
It works like this: 1) carry out a search in DOAJ 2) filter and refine the results until they are as precise as you need them 3) use the Embed button ‘share|embed’ button at the top of the search results to reveal the code that you will paste into your web page. The results that you embed in your site update automatically as DOAJ updates.

Some possible uses might be: a publisher can now display a list of all its journals or articles, indexed in DOAJ, on its own site; a library can display all DOAJ indexed journals with no APCs that have the Seal; a researcher might embed a list of all open access articles with a CC BY license that are pertinent to her research group.

Embed a DOAJ search box in your site
You can also embed a DOAJ search box directly into your site, allowing your users to search for open access, peer reviewed journals and articles directly. A user carrying out a search in this way will be taken to the search results in DOAJ. The search box can be embedded into your site, library portal, web page or blog, making quality open access material more easily accessible alongside other content.

We’ve created a new page for the widgets which has a little more technical detail on how to configure them so they behave exactly as you want them to. As ever, we’re always interested in your feedback!

 

 

Development plan for 2015/6

Recent discussions with a partner reminded me that we haven’t published a development plan.

I’ve put together a high level list of our developments: what we are currently doing and what we have next on the list. As it stands, it will take us through to mid-2016 but it is a very fluid document. We have the flexibility to add in items and move others so the 2016 items will be revisited in January 2016 and reviewed to make sure nothing else has cropped up in the meantime.

Some of these items I have already announced in earlier posts. I’ve collected those here.

As always, I’d welcome your feedback.