Using the DOAJ

We’ve added 2 new columns to the journal CSV.

In a series of enhancements to article metadata, we have released the third today.*

If you are familiar with DOAJ metadata then you probably already know that you can download a CSV file which contains the journal-level metadata for all the journals in DOAJ. It is updated every 30 minutes and is probably one of the most popular metadata services we have. It is certainly the best way to get an overview of all the journals in DOAJ.

Today, we’ve released a new version of that file which adds two new columns to it: the number of article records added to a journal in DOAJ and the date that the last article was added. (Columns BF and BG respectively)

The columns were added for two reasons:

  1. To give greater transparency to the information which we display on our homepage whcountere we state that 77% of the journals in DOAJ have article content loaded to them. This is slightly misleading because a journal which uploaded only 5 articles to us in 2013 is counted in that 77%. It is more interesting to know how recent the articles are from a particular title and how much content they have uploaded to us. While this information has always been available by selecting a journal ISSN and using the ‘articles’ filter in Search, we’ve never been able to show all the information in one place.I think that this development will be welcomed by all our users, especially publishers, librarians and those doing research on open access publishing developments.
  2. To allow us to review those journals which have been awarded the DOAJ Seal and remove that Seal from those which are not supplying article metadata to us. Supplying article metadata is one of the 7 Seal criteria and we haven’t yet been able to check, in an efficient manner, which journals are sticking to their promise. [In the application form, we ask if journals “intend” to supply metadata to us.] It’s going to take deeper analysis to get the final figure but I can see very quickly that 25 journals are going to lose their Seal.

I’d love to know what you think about this development so, as usual, do leave a comment or a question, or email me directly: dom@doaj.org.

P.S. The other development which I will post about soon is the removal of duplicated articles from the DOAJ database. More on that when I have it.

*The first and second developments are described in this blog post from February.

Full DOAJ data dump now available

“We’re going to a hackathon and would love to work with DOAJ’s data! Do you have a dump of all DOAJ data?”

“Er, no. Sorry. I mean, we could get it for you but it will take a while. You could probably get it yourself if you know how to extract the JSON… When do you need it for?”

“Tomorrow.”

“Oh.”

We’ve had this conversation a few times. Or had requests from eager individuals and organisations who want to use the rich offerings of the DOAJ metadata. They’ve told us of the wonderful things they could do with the data (slicing, reporting, analysing, apps, databases, software…) and we’ve never been able to help them in good time. But now, we can….

Introducing, full data dumps of ALL the metadata in DOAJ, both journals and articles: https://doaj.org/public-data-dump

So what? Well, let me suggest to you why these are a good thing:

  • For the journal metadata, the CSV is really the only easy-to-use format. The journal data dump provides another way to do this.
  • The data dumps are updated weekly, so can keep you up-to-date on a reasonably short delay. (There is no change feed, just a full dump.)
  • When you want all of the DOAJ data for any reason, you can just take it!
  • Deep paging on the search API is no longer permitted – search is for search, not harvesting. The data dump allows you to harvest.
  • Whenever you want a subset of the DOAJ data, you can just download the data dump, then filter it locally for your needs. For example, if you are a publisher and you want to see all of your metadata in DOAJ, that is all in this data dump, and you can then filter by ISSN
  • You can use it to enhance any local data in your own system or database: you may have basic article metadata in your system, and you want to extend it with DOAJ metadata.
  • If you want to aggregate publications data from multiple sources, this is one way of quickly getting that information from DOAJ (versus using OAI-PMH).
  • These data dumps are more metadata rich than OAI-PMH
  • You may want to use the data for analysis or data mining or other forms of research, or hackathons.
  • The data dumps are also useful as a test dataset.

So there you have it. Despite a rather awkward name, data dumps are A Good Thing.

We’d love to know what you think, so do please leave a comment here or send us feedback: feedback@doaj.org

Feature: article upload enhancements

At DOAJ, we work hard to maintain a high level of recency and accuracy in our metadata. All of our metadata is freely available, in various formats, to those who want it. This means that any errors in it get distributed freely around the web. To reduce these and negate the knock-on effect, DOAJ works with its technical partners, Cottage Labs, to clean the metadata.

On 21st February, we will be releasing two small but fairly important enhancements to our article upload function. The two changes are as follows:

Spaces will be stripped from DOIs and full text URLs upon ingest.
This is to improve matching in our database on DOIs and URLs. We use DOIs and full text URLs to version articles, thereby allowing corrections or enhancements to article metadata to be uploaded without the existing version being deleted first.

We regularly receive metadata with badly formatted URLs or DOIs, with preceding spaces, trailing spaces or spaces right in the middle of a DOI or URL. This means matching doesn’t occur, we end up with multiple versions of the same article in the database and an increased number of duplicates.

Duplicates in the same file will be prevented upon upload
To upload article metadata to us, a file of article metadata must be sent to our article ingester. We will introduce an enhancement here which will prevent a file from uploading if duplicates within it are detected. We carry out no such checks at the moment.

Both enhancements will be implemented in all 3 article ingest front ends: the XML uploader, the manual article uploader, and the API.

Both changes are two small steps toward a larger project of eradicating all duplicated content in the database. We don’t know yet how much article content is duplicated but it will be enough to cause a noticeable reduction in the current number of articles in DOAJ (3,767,076 articles at the time or writing).

If you have any questions about these enhancements or are wondering if you have duplicates in your own article metadata, do please leave a comment here. 

 

 

 

Are you using an up-to-date version of DOAJ metadata? Apparently not.

Did you know that DOAJ’s metadata refreshes every hour? Given how many changes we, the DOAJ Team, make to the database on a weekly basis plus all the article metadata which is continuously being uploaded to us, the hourly update is absolutely necessary. DOAJ is seeing more use of its API than ever before (2018 total: 266,255,000 hits; 2017 total: 187,212,674) and most of that is by publishers uploading articles.

DOAJ metadata is EVERYWHERE. It is incorporated into all the major discovery services, other online databases and many library catalogues.

You might remember this post of mine from 2017, where I explained that third parties using our data may not have a recent copy of it. We know for a fact that some systems downstream of our data only update the copy every 6 months and, sadly, we get many emails from end users asking why they can’t find an article.

So if you are a researcher, reading this and wondering what happened to that article in DOAJ which is listed in your library catalogue or discovery service, contact your librarians and ask them to update their metadata. If you’re a librarian then do please reach out to sources responsible for your metadata and get them to update their DOAJ metadata.

If you have any questions about this or need some help in getting a more recent copy of our metadata then do leave a comment here or contact us at feedback@doaj.org.

El DOAJ, independencia y la importancia de la imparcialidad

El DOAJ recibe regularmente preguntas y, a veces, quejas de bibliotecas, consorcios de bibliotecas y otras instituciones académicas sobre el papel que desempeñan en el DOAJ las editoriales académicas tradicionales. Un error común es que el DOAJ es propiedad de estas organizaciones, o está totalmente subvencionado por ellas, y que el DOAJ sólo beneficia a estas organizaciones. Si usted mirara la página de inicio del DOAJ hace dos años, podríamos entender esta suposición: casi todos nuestros patrocinadores eran editoriales y estaban en nuestra página de inicio. Parecía que el DOAJ era propiedad de estas editoriales.

(Evitar esa idea errónea es también la razón por la cual el DOAJ es muy cuidadoso sobre con quién se asocia). Las organizaciones con las que nos asociamos deben compartir los mismos valores que el DOAJ, tener la misma visión y al menos tener la intención de proporcionar servicios a la comunidad. El DOAJ se enorgullece de asociarse con organizaciones como Redalyc, SciELO, ISSN y COPE.

La suposición de que el DOAJ es propiedad de las editoriales o está totalmente subvencionado por ellas es, por supuesto, incorrecta. Las editoriales desempeñan un importante papel financiero en el apoyo de los servicios que el DOAJ proporciona a todas las partes interesadas. Algunas editoriales nos donan un patrocinio anual; algunas de ellas son miembros de DOAJ como editorial. Estas contribuciones permiten al DOAJ proporcionar servicios continuos, y mejoras a estos servicios. En 2017, las contribuciones de las editoriales representaban el 40% de los ingresos, mientras que las contribuciones de las instituciones del sector público representaban el resto (60%). En 2018, los ingresos de las instituciones del sector público representarán el 70%. Todas las contribuciones al DOAJ son voluntarias; todos los servicios proporcionados por el DOAJ, incluyendo la evaluación de revistas, son gratuitos.

El DOAJ es útil para bibliotecarios, para editores, para investigadores, para estudiantes, para propietarios de revistas, para todos. Más del 50% de nuestro consejo asesor proviene de la comunidad de bibliotecas y consorcios. El DOAJ es global y no está atado a fronteras geográficas. Somos una organización virtual que emplea a personas de todo el mundo y que cuenta con nuestros propios embajadores que fomentan las mejores prácticas en sus territorios de origen. El DOAJ es 100% independiente. El DOAJ es 100% imparcial. La compañía holding del DOAJ, IS4OA C.I.C., está registrada de una manera que hace imposible que el DOAJ sea comprado, adquirido o vendido.

La imparcialidad juega un papel importante en el progreso que el DOAJ ha hecho en los últimos 5 años y el equipo del DOAJ trabaja duro para asegurar que la imparcialidad sea lo más importante en todo lo que hacemos. Esta es una de las razones por las que el DOAJ se adhiere y alienta a otros a adherirse a la transparencia y las buenas prácticas; mejores prácticas que se desarrollan, adoptan y reconocen a nivel internacional. También reconocemos que la aplicación de estas normas y el funcionamiento dentro de ellas puede ser un reto, por lo que tenemos que mantener cierta flexibilidad. Un buen ejemplo sería la concesión de licencias.

licensing

Ejemplo de información sobre licencias de una revista capturada en DOAJ.

A veces nos preguntan: ¿por qué el DOAJ acepta las licencias más restrictivas cuando la definción BOAI (definición a la que el DOAJ se adhiere como uno de sus principios) es muy clara sobre lo que significa “abierto”?

Cuando el DOAJ elaboró su formulario de solicitud ampliado en 2013, vimos muy claro que el DOAJ tenía que aceptar las 6 variaciones de las licencias CCBY, con sus distintos grados de apertura para asegurarnos que el mayor número posible de revistas pudieran solicitar el ingreso en el DOAJ. Al usuario típico del DOAJ se le deben presentar opciones y se le debe permitir tomar una decisión informada basada en la información que mostramos sobre las políticas de licencias y derechos de autor de una revista.

Sin embargo, el DOAJ considera el uso de las licencias CC, en particular el uso de las licencias más abiertas, como una de las mejores prácticas y las promueve como uno de los criterios para el Sello DOAJ.

En 2017 el DOAJ fue el noveno en una lista mundial de plataformas que apoyan el uso de licencias Creative Commons. Creemos que esto es el resultado directo de nuestra preferencia por las licencias CC y la influencia del Sello DOAJ.

Para que una iniciativa como el DOAJ funcione, debe seguir siendo lo más relevante posible en todo el mundo. Debe seguir siendo imparcial e independiente.

 

DOAJ, Independence and the Importance of Impartiality

DOAJ receives regularly questions and, sometimes, complaints from libraries, library consortia and other academic institutions about the role which publishers—by “publishers” I mean the traditional publishing organisations in academic publishing—play in DOAJ. A common misconception is that DOAJ is owned by, or wholly subsidised by these organisations and that DOAJ is only of benefit to these organisations. If you looked at the DOAJ homepage two years ago, you’d be forgiven for making that assumption: nearly all of our sponsors were publishing organisations and they were all on our homepage. It looked like DOAJ was owned by publishers.

(Avoiding that misconception is also why DOAJ is very careful about who it goes into partnership with. The organisations we partner with must share the same values as DOAJ, have the same vision and at least be intent on providing services to the community. DOAJ is proud to call organisations like Redalyc, SciELO, ISSN and COPE partners.)

The assumption that DOAJ is owned by, or wholly subsidised by publishers is of course incorrect. Publishers play an important financial role in supporting the services which DOAJ provides to all stakeholders. Some publishers donate an annual sponsorship to us; some of them are publisher members. These contributions enable DOAJ to provide continuous services, and improvements to these services, for everyone but any direct influence on DOAJ stops there. In 2017 contributions from publishers accounted for 40% of the income, whereas contributions from public sector institutions accounted for the rest (60%). In 2018 income from public sector institutions will account for 70%. All contributions to DOAJ are made voluntarily; all services provided by DOAJ, including the evaluation of journals, are free.

DOAJ is for librarians, for publishers, for researchers, for students, for journal owners, for everyone. More than 50% of our advisory board is from the library and consortia community. DOAJ is global and isn’t tied down to geographic borders. We are a virtual organisation, employing people from all over the world and with our own Ambassadors encouraging best practice in their home territories. DOAJ is 100% independent. DOAJ is 100% impartial. DOAJ’s holding company, IS4OA C.I.C., is registered in a way that makes it impossible for DOAJ to be purchased, acquired or sold.

Impartiality plays a huge role in the progress that DOAJ has made over the last 5 years and the DOAJ Team works hard to ensure that impartiality is foremost in everything we do. This is one of the reasons that DOAJ adheres to, and encourages other to adhere to, transparency, best practice and standards; best practices and standards which are internationally grown, adopted and recognised. We also recognise that applying those standards and operating within them can be challenging so we have to retain a certain amount of flexibility. A good example here would be licensing.

licensing

Example of the licensing information captured by DOAJ about a journal’s policies.

People often ask: why does DOAJ accept some of the more closed licenses when the BOAI definition [that DOAJ follows as a principle] is clear about what open means? When DOAJ put together its extended application form in 2013, it was clear that DOAJ needed to allow 6 variations on the CC BY license, with varying degrees of openness, to ensure that as many titles as possible could apply to be indexed. The typical DOAJ user should be presented with options and they should be allowed to make an informed choice based on the information we display about a journal’s licensing and copyright policies. However, DOAJ considers the use of CC licenses, particularly use of the most open licenses, as Best Practice and we promote that as one of the criteria for the DOAJ Seal.

In 2017 DOAJ was 9th in a worldwide list of platforms supporting the use of Creative Commons licenses. We believe this to be a direct result of our  preference for CC licenses and the influence of the DOAJ Seal.

For an initiative like DOAJ to work, it must remain as relevant as possible around the world. It must remain impartial and it must remain independent.

’10 web design features that are now browser history’ – or, at least, they should be.

A great blog post from the summer, published on the DataSalon blog and written by Jon Monday, caught our attention recently. Written in celebration of 25 years of the World Wide Web, the blog post looks ‘back on some design features from the early days of the Web’ which have been consigned to “browser history”.

Or at least they should have been.

At DOAJ, we review hundreds of journal applications every month from all corners of the globe, covering every topic imaginable. The applications themselves are always of varying degrees of quality and, sadly, so are the websites they refer to. From mid-December 2017 to mid-September 2018, our Triage function rejected without review 1754 poorly completed applications! Many of the reasons for rejection without review refer directly to a poorly constructed, poorly signposted website.

At least 6 of the items in Jon’s post are still seen by the DOAJ Team on a regular basis: hit counters; guestbooks; blinking text; scrolling text; animated gifs; and under constructions signs. So saying that they have been consigned forever to the bin may not be completely accurate but it certainly is wishful thinking on our part!