New feature

Sign up to receive updates from us

DOAJ has never had a method to push updates to our users. We have always just posted information on this blog in the hope that it will find its audience. We are aware that some stakeholders miss important updates so we have launched an update service.

We will use the update service to inform you about:

  1. Our strategy and changes to it;
  2. Our technical roadmap and progress updates;
  3. Funding and funding models;
  4. If we add a new feature to the website.
  5. If we start a new campaign for signing up journals.
  6. Governance, particularly if we enrol new members to our Board, Council, or Editorial Sub-Committee
  7. When we need more volunteers to help us review journals.

Sign up here: http://eepurl.com/g3Y4v5

All sign-ups are managed via our Mailchimp account. DOAJ has a Privacy Policy which adheres to the guidelines laid out by the GDPR.

[This pst was UPDATED and CORRECTED on 25th May. DOAJ is not launching a newsletter service but a service which will push updates to people who have signed up.]

We can now fix article metadata ourselves

As part of our continued strategy to enhance our metadata offering, and looking forward to 2021 when our strategic focus will be on DOAJ’s place in the discovery chain, we are constantly looking for ways to improve the quality and recency of the DOAJ article metadata.

Errors in article metadata are usually reported to us by users of discovery services at libraries or by the discovery services themselves. The errors are often a broken URL or a missing article.

Since 2013, DOAJ’s stance on correcting metadata has been to contact the journal’s publishers and ask them to send an update to us. However, this approach is of limited success:

  • some publishers don’t respond,
  • some don’t know how to make the correction and,
  • since we do matching on the URL, the process often leads to duplicated articles.

As of yesterday, DOAJ can edit article metadata itself, including the URL, directly in the database. This makes changes immediate and goes a long way to improve the reliability of the DOAJ metadata. Importantly, it allows us to act quickly on queries sent to us by our discovery service colleagues.

The crux of the matter, when it comes to the accuracy of DOAJ metadata, remains unchanged however: all 3rd party services should update their copy of the DOAJ metadata regularly. DOAJ metadata updates every 30 minutes. Some 3rd party services only update their copy of our metadata every 6 months.

DOAJ será compatible con Crossref

metadata-from-crossref-logo-200En una serie de mejoras de metadatos, DOAJ aceptará la carga de XML en formato Crossref a partir del 18 de febrero de 2020.

 

En 2018, preguntamos a nuestros editores qué facilitaría su interacción con DOAJ y el 46% dijo que les gustaría que aceptásemos Crossref XML. Hoy en día sólo aceptamos XML en nuestro formato DOAJ.

Hay algunas aclaraciones que nos gustaría hacer en torno a esta mejora:

  1. No vamos a desconectar DOAJ XML. Si ya nos están suministrando DOAJ XML, pueden continuar haciéndolo. No hay cambios en este sentido.
  2. Si quieren cambiar a Crossref XML, pueden hacerlo cuando quieran. Sin embargo:
  3. No se pueden mezclar XML de Crossref y DOAJ en el mismo archivo.
  4. El  ISSN no siempre está presente en los XML de Crossref. Cuando se envía un archivo XML de Crossref a DOAJ, se deben incluir los ISSN(s) para que la carga sea exitosa.
  5. El proceso de subida de metadatos de artículos en DOAJ no cambia: por ahora, todavía tendrán que seguir enviándolos. DOAJ no cosecha los artículos automáticamente.
  6. Los XMLs de Crossref se suben desde la misma página en la que se cargan los XML con formato DOAJ. https://doaj.org/publisher/  Ahora deben elegir qué formato quieren enviarnos:
    format
  7. Las revistas editadas en plataforma OJS que usan el plugin de DOAJ para generar XML, deberían continuar haciéndolo. No hay cambios en este sentido.

Si su revista tiene contenido en el formato Crossref que les interese enviar, sólo tienen que subirnos el archivo de manera normal. Crossref facilita varias formas de recuperarsus metadatos de ellos si aún no disponen de ellos.

Se harán más anuncios en los próximos días.

DOAJ to add Crossref compatibility

metadata-from-crossref-logo-200In a series of metadata improvements, publishers will be able to upload XML in the Crossref format to us from 18th February 2020.

In 2018, we asked our publishers what would make their interaction with DOAJ easier and 46% said that they would like us to accept Crossref XML. Today we only accept XML formatted to our proprietary DOAJ format.

There are a few clarifications that we’d like to make around this improvement:

  1. We are not switching off DOAJ XML. If you are already supplying DOAJ XML to us, you may continue to do so. Nothing changes for you.
  2. If you want to switch to Crossref XML, you may do so whenever you like. However,…
  3. You cannot mix Crossref and DOAJ XML in the same file.
  4. An ISSN is usually but not always present in Crossref XML. When sending Crossref XML to DOAJ, an ISSN must be included for the upload to be successful.
  5. The process of getting article metadata into DOAJ doesn’t change: you will still need to continue to send it to us, for now.
  6. Crossref XML is uploaded from the same page that DOAJ XML is uploaded to us: https://doaj.org/publisher/ but you must now choose which format you want to send to us:
    format
  7. Publishers on the OJS platform, using the DOAJ plugin to generate XML, should continue to do so. No change is needed.

If you have back data in the Crossref format that you want to send to us, just upload the file to us in the normal way. Crossref facilitate various ways for you to retrieve your metadata from them if you don’t have it already.

Further announcements will be made next week.

 

Two updates: ORCID integration; updated ISO code lists for country, language and currency

On Monday 3rd February we will release two updates to the site.

ORCID integration

ORCID 'ID' logoFrom Monday, it will be possible to upload to DOAJ article metadata containing ORCID iDs.

We will then display ORCID iDs on article pages in accordance with the ORCID display guidelines. Changes will be made to the search interface and API to allow search and retrieval of ORCID iDs.

Details on how to tag ORCID iDs in DOAJ XML will be available on our XML page. For those of you who provide metadata to us via the manual input form, a new ORCID field will be available there.

ORCID iDs will be available downstream in our metadata.

We are delighted to release this addition to the DOAJ platform. DOAJ recognises the importance of permanent identifiers (PIDs) such as ORCID iDs and encourages their use in article metadata.

Updated ISO code lists for country, language and currency

We will also be releasing an update to the code lists for countries, currencies and languages. These haven’t been updated since 2013 so there are quite a few changes, most of them standardisation of a name or spelling plus some obvious geopolitical changes too.

We have used a service called Pycountry which takes in the standard ISO lists for these 3 lists.

You can see the full list of changes here.

For those publishers supplying metadata to us who wish to update a language code in metadata that has already been sent to us, please contact us via the Feedback account.

Full DOAJ data dump now available

“We’re going to a hackathon and would love to work with DOAJ’s data! Do you have a dump of all DOAJ data?”

“Er, no. Sorry. I mean, we could get it for you but it will take a while. You could probably get it yourself if you know how to extract the JSON… When do you need it for?”

“Tomorrow.”

“Oh.”

We’ve had this conversation a few times. Or had requests from eager individuals and organisations who want to use the rich offerings of the DOAJ metadata. They’ve told us of the wonderful things they could do with the data (slicing, reporting, analysing, apps, databases, software…) and we’ve never been able to help them in good time. But now, we can….

Introducing, full data dumps of ALL the metadata in DOAJ, both journals and articles: https://doaj.org/public-data-dump

So what? Well, let me suggest to you why these are a good thing:

  • For the journal metadata, the CSV is really the only easy-to-use format. The journal data dump provides another way to do this.
  • The data dumps are updated weekly, so can keep you up-to-date on a reasonably short delay. (There is no change feed, just a full dump.)
  • When you want all of the DOAJ data for any reason, you can just take it!
  • Deep paging on the search API is no longer permitted – search is for search, not harvesting. The data dump allows you to harvest.
  • Whenever you want a subset of the DOAJ data, you can just download the data dump, then filter it locally for your needs. For example, if you are a publisher and you want to see all of your metadata in DOAJ, that is all in this data dump, and you can then filter by ISSN
  • You can use it to enhance any local data in your own system or database: you may have basic article metadata in your system, and you want to extend it with DOAJ metadata.
  • If you want to aggregate publications data from multiple sources, this is one way of quickly getting that information from DOAJ (versus using OAI-PMH).
  • These data dumps are more metadata rich than OAI-PMH
  • You may want to use the data for analysis or data mining or other forms of research, or hackathons.
  • The data dumps are also useful as a test dataset.

So there you have it. Despite a rather awkward name, data dumps are A Good Thing.

We’d love to know what you think, so do please leave a comment here or send us feedback: feedback@doaj.org

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

We have improved our XML validation

xml

We are happy to announce that we have improved the features of our xml validation. As an article XML is uploaded to DOAJ from the publisher area we will check to see if the uploaded file meets a number of criteria:

  • Is it actually XML? (have you given us a PDF by mistake?)
  • Does the XML meet the DOAJ article schema requirements?
If your upload doesn’t meet either of those criteria you will get an error message straight away.  It will either tell you we didn’t think the file was XML, or it will give you a message from our validator which identifies the first problem found when validating.
If your file does pass those checks, it will get entered into our upload queue.  You will need to check back on your upload page in a while to see if the upload was processed successfully.  When we process your upload, we carry out some more checks to make sure that the data in DOAJ remains consistent:
  • Do you reference any ISSNs that belong to journals you don’t own?  You can’t attach articles to someone else’s journals.
  • Do you reference any ISSNs that don’t appear in DOAJ at all?  We only take uploads for journals we know about, and your journal records need to contain all the correct identifiers if you want to reference them.
  • Do you reference ISSNs that are owned by more than one user?  Sometimes it’s possible for the DOAJ to think that more than one user has a stake in an ISSN.  In these cases we need to resolve who the true owner is before you can upload.
If any of these situations arise with your file upload, we won’t import any of the articles, and instead record an error against the upload.  When you look at the upload record on the page you will see a link in the Notes column which says “show error details“.  When you click on that it will tell you in what way the articles failed to process, and which ISSNs were causing the problems.
If you see any other error messages on your upload page, you should click on the “show error details” link if it is available.  If not, you should contact us at feedback@doaj.org and we will get back to you as soon as we can.

 

DOAJ. Now with added PLOS.

Finally, after saying: “It’s coming” for almost a year, DOAJ has added the entire PLOS catalogue to its database. A total of 182,500 articles were added and the harvester is set to go out and collect new articles at 5.30am every morning. The harvester collects the metadata from Europe PMC.

This is a milestone for DOAJ: not only because it has been eager to get the PLOS article metadata for a long time; but also because the harvester can be re-tuned to go and search for other large bodies of content from other publishers on E-PMC. Eventually, with some extra development, we’ll be able to point the harvester at any body of content.

If you think your publisher would benefit from this approach, do please get in touch! Thanks to our technical partners, Cottage Labs, for their help in reaching this milestone.

NEW: continuations, improved tables of contents, feedback on a journal, APC granularity, and more…

Very soon, we will be releasing new functionality on DOAJ that will improve the detail in the journal record, increase DOAJ’s accuracy even more and will encourage greater input from the community. We have also abandoned our policy to remove journals that have ceased publication.

Continuations

Ever since we migrated to our new platform, we have been trying to find a slot in our development schedule to correctly display journal continuations on our site. It was important for us to get this correct, both in terms of the metadata and the display, so that users could easily understand how one, or several, journals transformed into a journal with a new title and ISSN. This new development will allow us to correctly display a journal’s timeline as it goes through name and ISSN changes.

Send us feedback on a journal

A decreasing amount of the data in DOAJ is old and DOAJ is always grateful to receive updates from users who get in touch to alert us to broken URLs or a change of title. With this in mind, and to make feeding back as simple as possible, we will add a ‘Tell us about this journal’ button to every journal entry in DOAJ. We hope this will encourage greater input from our users and lead to more updates to journal entries. This development is perfectly aligned with DOAJ being curated by the community and really facilitates an up-to-date resource.

Greater granularity for APC information

We know. The APC information on DOAJ was misleading. It lumps together 2 distinct groups of journals into one big ‘No’ group. We have broken that group out into: No, where we have up-to-date information from journals that they DO NOT charge APCs; and No information. ‘No information’ is the group of journals which has submitted a reapplication to us and we have yet to review it. The number of ‘No information’ journals will diminish in size as we work our way through the reapplications. I wrote about this inaccuracy previously.

Retaining the records – journals that have ceased publication

As long as they fulfil some basic criteria, we will no longer remove journals that cease publishing and we will even add back into DOAJ journals that were removed only because they stopped publishing. This change of policy could potentially add hundreds of articles back into DOAJ along with tens of archived journals.

Access from China

We have had several complaints from users in China that they were unable to use DOAJ. We realised that this is because the site uses Google javascript services and Google is banned in some parts of China, or by some institutions. We have now migrated most services off Google-related services (apart from Analytics) and are confident that the user experience for users in China will be vastly improved. This is perfectly times as we embark on a year-long project to increase awareness of open access in China.

Oh, and watch this space… In early June, we’ll be adding ~175 000 PLoS articles to DOAJ… More on that later.