New feature

DOAJ será compatible con Crossref

metadata-from-crossref-logo-200En una serie de mejoras de metadatos, DOAJ aceptará la carga de XML en formato Crossref a partir del 18 de febrero de 2020.

 

En 2018, preguntamos a nuestros editores qué facilitaría su interacción con DOAJ y el 46% dijo que les gustaría que aceptásemos Crossref XML. Hoy en día sólo aceptamos XML en nuestro formato DOAJ.

Hay algunas aclaraciones que nos gustaría hacer en torno a esta mejora:

  1. No vamos a desconectar DOAJ XML. Si ya nos están suministrando DOAJ XML, pueden continuar haciéndolo. No hay cambios en este sentido.
  2. Si quieren cambiar a Crossref XML, pueden hacerlo cuando quieran. Sin embargo:
  3. No se pueden mezclar XML de Crossref y DOAJ en el mismo archivo.
  4. El  ISSN no siempre está presente en los XML de Crossref. Cuando se envía un archivo XML de Crossref a DOAJ, se deben incluir los ISSN(s) para que la carga sea exitosa.
  5. El proceso de subida de metadatos de artículos en DOAJ no cambia: por ahora, todavía tendrán que seguir enviándolos. DOAJ no cosecha los artículos automáticamente.
  6. Los XMLs de Crossref se suben desde la misma página en la que se cargan los XML con formato DOAJ. https://doaj.org/publisher/  Ahora deben elegir qué formato quieren enviarnos:
    format
  7. Las revistas editadas en plataforma OJS que usan el plugin de DOAJ para generar XML, deberían continuar haciéndolo. No hay cambios en este sentido.

Si su revista tiene contenido en el formato Crossref que les interese enviar, sólo tienen que subirnos el archivo de manera normal. Crossref facilita varias formas de recuperarsus metadatos de ellos si aún no disponen de ellos.

Se harán más anuncios en los próximos días.

DOAJ to add Crossref compatibility

metadata-from-crossref-logo-200In a series of metadata improvements, publishers will be able to upload XML in the Crossref format to us from 18th February 2020.

In 2018, we asked our publishers what would make their interaction with DOAJ easier and 46% said that they would like us to accept Crossref XML. Today we only accept XML formatted to our proprietary DOAJ format.

There are a few clarifications that we’d like to make around this improvement:

  1. We are not switching off DOAJ XML. If you are already supplying DOAJ XML to us, you may continue to do so. Nothing changes for you.
  2. If you want to switch to Crossref XML, you may do so whenever you like. However,…
  3. You cannot mix Crossref and DOAJ XML in the same file.
  4. An ISSN is usually but not always present in Crossref XML. When sending Crossref XML to DOAJ, an ISSN must be included for the upload to be successful.
  5. The process of getting article metadata into DOAJ doesn’t change: you will still need to continue to send it to us, for now.
  6. Crossref XML is uploaded from the same page that DOAJ XML is uploaded to us: https://doaj.org/publisher/ but you must now choose which format you want to send to us:
    format
  7. Publishers on the OJS platform, using the DOAJ plugin to generate XML, should continue to do so. No change is needed.

If you have back data in the Crossref format that you want to send to us, just upload the file to us in the normal way. Crossref facilitate various ways for you to retrieve your metadata from them if you don’t have it already.

Further announcements will be made next week.

 

Two updates: ORCID integration; updated ISO code lists for country, language and currency

On Monday 3rd February we will release two updates to the site.

ORCID integration

ORCID 'ID' logoFrom Monday, it will be possible to upload to DOAJ article metadata containing ORCID iDs.

We will then display ORCID iDs on article pages in accordance with the ORCID display guidelines. Changes will be made to the search interface and API to allow search and retrieval of ORCID iDs.

Details on how to tag ORCID iDs in DOAJ XML will be available on our XML page. For those of you who provide metadata to us via the manual input form, a new ORCID field will be available there.

ORCID iDs will be available downstream in our metadata.

We are delighted to release this addition to the DOAJ platform. DOAJ recognises the importance of permanent identifiers (PIDs) such as ORCID iDs and encourages their use in article metadata.

Updated ISO code lists for country, language and currency

We will also be releasing an update to the code lists for countries, currencies and languages. These haven’t been updated since 2013 so there are quite a few changes, most of them standardisation of a name or spelling plus some obvious geopolitical changes too.

We have used a service called Pycountry which takes in the standard ISO lists for these 3 lists.

You can see the full list of changes here.

For those publishers supplying metadata to us who wish to update a language code in metadata that has already been sent to us, please contact us via the Feedback account.

Full DOAJ data dump now available

“We’re going to a hackathon and would love to work with DOAJ’s data! Do you have a dump of all DOAJ data?”

“Er, no. Sorry. I mean, we could get it for you but it will take a while. You could probably get it yourself if you know how to extract the JSON… When do you need it for?”

“Tomorrow.”

“Oh.”

We’ve had this conversation a few times. Or had requests from eager individuals and organisations who want to use the rich offerings of the DOAJ metadata. They’ve told us of the wonderful things they could do with the data (slicing, reporting, analysing, apps, databases, software…) and we’ve never been able to help them in good time. But now, we can….

Introducing, full data dumps of ALL the metadata in DOAJ, both journals and articles: https://doaj.org/public-data-dump

So what? Well, let me suggest to you why these are a good thing:

  • For the journal metadata, the CSV is really the only easy-to-use format. The journal data dump provides another way to do this.
  • The data dumps are updated weekly, so can keep you up-to-date on a reasonably short delay. (There is no change feed, just a full dump.)
  • When you want all of the DOAJ data for any reason, you can just take it!
  • Deep paging on the search API is no longer permitted – search is for search, not harvesting. The data dump allows you to harvest.
  • Whenever you want a subset of the DOAJ data, you can just download the data dump, then filter it locally for your needs. For example, if you are a publisher and you want to see all of your metadata in DOAJ, that is all in this data dump, and you can then filter by ISSN
  • You can use it to enhance any local data in your own system or database: you may have basic article metadata in your system, and you want to extend it with DOAJ metadata.
  • If you want to aggregate publications data from multiple sources, this is one way of quickly getting that information from DOAJ (versus using OAI-PMH).
  • These data dumps are more metadata rich than OAI-PMH
  • You may want to use the data for analysis or data mining or other forms of research, or hackathons.
  • The data dumps are also useful as a test dataset.

So there you have it. Despite a rather awkward name, data dumps are A Good Thing.

We’d love to know what you think, so do please leave a comment here or send us feedback: feedback@doaj.org

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

We have improved our XML validation

xml

We are happy to announce that we have improved the features of our xml validation. As an article XML is uploaded to DOAJ from the publisher area we will check to see if the uploaded file meets a number of criteria:

  • Is it actually XML? (have you given us a PDF by mistake?)
  • Does the XML meet the DOAJ article schema requirements?
If your upload doesn’t meet either of those criteria you will get an error message straight away.  It will either tell you we didn’t think the file was XML, or it will give you a message from our validator which identifies the first problem found when validating.
If your file does pass those checks, it will get entered into our upload queue.  You will need to check back on your upload page in a while to see if the upload was processed successfully.  When we process your upload, we carry out some more checks to make sure that the data in DOAJ remains consistent:
  • Do you reference any ISSNs that belong to journals you don’t own?  You can’t attach articles to someone else’s journals.
  • Do you reference any ISSNs that don’t appear in DOAJ at all?  We only take uploads for journals we know about, and your journal records need to contain all the correct identifiers if you want to reference them.
  • Do you reference ISSNs that are owned by more than one user?  Sometimes it’s possible for the DOAJ to think that more than one user has a stake in an ISSN.  In these cases we need to resolve who the true owner is before you can upload.
If any of these situations arise with your file upload, we won’t import any of the articles, and instead record an error against the upload.  When you look at the upload record on the page you will see a link in the Notes column which says “show error details“.  When you click on that it will tell you in what way the articles failed to process, and which ISSNs were causing the problems.
If you see any other error messages on your upload page, you should click on the “show error details” link if it is available.  If not, you should contact us at feedback@doaj.org and we will get back to you as soon as we can.

 

DOAJ. Now with added PLOS.

Finally, after saying: “It’s coming” for almost a year, DOAJ has added the entire PLOS catalogue to its database. A total of 182,500 articles were added and the harvester is set to go out and collect new articles at 5.30am every morning. The harvester collects the metadata from Europe PMC.

This is a milestone for DOAJ: not only because it has been eager to get the PLOS article metadata for a long time; but also because the harvester can be re-tuned to go and search for other large bodies of content from other publishers on E-PMC. Eventually, with some extra development, we’ll be able to point the harvester at any body of content.

If you think your publisher would benefit from this approach, do please get in touch! Thanks to our technical partners, Cottage Labs, for their help in reaching this milestone.