The Data Gap

What did we find? We were able to access 11 mammography datasets. There were over 2.5 million mammograms in total, with 2 million of them being from one UK dataset. The datasets were mainly from high-income settings and there was a lack of reporting of individuals’ attributes. Only three datasets reported race/ethnicity and none provided clear definition for these categories within their dataset documentation. Ten of the datasets were open access whereas for the final dataset, we needed to submit a data access request (regulated access). For this regulated access dataset, we were able to ask questions and clarify when dataset documentation was unclear, meaning we could gain a greater understanding of the data. 

What does this mean? Many mammography datasets do not transparently report 'who' is represented and 'how' they are represented meaning developers of AI technologies using these datasets are less able to understand the limitations of the data. We found value in being able to contact dataset curators to clarify our understanding of the data but there is work to be done in making large, diverse and representative mammography datasets accessible to allow for development of AI technologies that are safe and effective for everyone.

Click here to see the full list of datasets identified as part of this review

What did we find? There were 119 COVID-19 datasets identified which represented individuals from 28 different countries. Seventeen datasets included data from different geographical areas. A large proportion of the datasets identified were combined (i.e. they were formed from several, smaller datasets) and eight of these were derived from a chest x-ray dataset which contained images of children - often, it was only through review of the images within the dataset that this was noted as there was limited reference to there being paediatric data within dataset documentation. Most of the datasets included chest x-ray images and some included CT thorax images. One dataset included brain MRI images. There was a distinct lack of reporting of individuals' attributes: only 35% reported age, 31% reported sex/gender and 8% reported race/ethnicity of individuals included in the dataset.

What does this mean? The rapid development of datasets in response to the COVID-19 pandemic meant that curators focused on quantity rather than quality of data. Documentation for COVID-19 datasets is limited which means AI developers are less able to understand the data they are using. A lack of dataset documentation is particularly problematic where datasets have been combined -  there is a risk that data quality is compromised yet users are unlikely to know where the issues lie if they are not documented. There is a need for clear documentation about how datasets are curated and who is included within them.

Click here to see the full list of datasets identified as part of this review

What did we find? Within the 28 accessible datasets identified, there are over 1,080,000 individuals represented from 14 different countries - however,  over 1,000,000 of those individuals are accounted for by the UK and USA alone. Data available is varied: the most common being ECG data, health record data and echocardiography. Nine datasets reported race/ethnicity (32%) and six (21%) reported socioeconomic status. No datasets reported definitions for demographic terms except for socioeconomic status where explanations were given for how the groups were stratified.  The methods for determining socioeconomic status varied greatly across different datasets, some used recognised national scoring systems while others used data about employment or education status.

What does this mean? Accessibility of data around individuals' attributes can allow AI developers to understand how their technologies might perform for different groups. However, definitions for different attributes vary depending on context e.g. race and ethnicity mean different things in different places, and socioeconomic status can be summarised in lots of different ways. For data around attributes to be useful, definitions must be provided so users of the data can understand what it is they are assessing.

Click here to see the full list of datasets identified as part of this review