Our Work

Work Packages

Defining best practice for dataset curation and use

Scoping review of existing dataset standards

Draft items for multi-stakeholder eDelphi (click HERE to find out more)

Mapping dataset deficiencies in priority disease areas

Identify datasets used in priority disease areas: COVID-19, breast cancer, heart failure

Assess datasets using proposed standards

Identifying and overcoming barriers to curating datasets

Conduct semi-structured interviews with dataset curators/users about implementing the standards

Publications

CORRESPONDENCE

Tackling bias in AI health datasets through the STANDING Together initiative

The STANDING Together Working Group, Nature Medicine

To launch the first round of the STANDING Together Delphi Study, an announcement paper was published in Nature Medicine in September 2022.

The full paper can be found at: https://doi.org/10.1038/s41591-022-01987-w

REVIEW

A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability

Summary by Dr Saad Khan

The availability of health datasets has accelerated digital health research. Ophthalmology has been one of the leading areas of innovation, where several public datasets for ophthalmic imaging have been use in machine learning research. Datasets are a critical component for machine learning algorithm development, hence these need careful scrutiny prior to use. Prior to our review, it was previously unknown how many ophthalmic datasets existed, their degree of accessibility... Read more

REVIEW

Characteristics of publicly available skin cancer image datasets: a systematic review

Summary by Dr David Wen

Freely available (open access) datasets containing skin images are frequently used to develop deep learning algorithms for skin cancer diagnosis. As these algorithms are heavily influenced by the images that they are trained on, it is important that the composition and characteristics of datasets are outlined, such as which populations images are taken from. This information is often... Read more

PRESENTATION
STANDING Together: STANdards for Data Diversity, Inclusivity and Generalisability

Presented by Dr Joe Alderman at Machine Learning for Healthcare, 2022

Click here to view the poster

Click here to watch the presentation

Poster reference list

Artificial Intelligence - how to get it right. NHSX. October 2019. Available from URL: https://transform.england.nhs.uk/media/documents/NHSX_AI_report.pdf
Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. The Lancet Digital Health. 2019 Oct 1;1(6):e271–97. Available online at: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(19)30123-2/fulltext
Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019 Oct 25;366(6464):447–53. Available online at: https://www.science.org/doi/10.1126/science.aax2342
Seyyed-Kalantari L, Zhang H, McDermott MBA, Chen IY, Ghassemi M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med. 2021 Dec;27(12):2176–82. Available online at: https://www.nature.com/articles/s41591-021-01595-01.
The health of people from ethnic minority groups in England [Internet]. The King’s Fund. 2021 [cited 2022 Apr 12]. Available from: https://www.kingsfund.org.uk/publications/health-people-ethnic-minority-groups-england1.
Khan SM, Liu X, Nath S, Korot E, Faes L, Wagner SK, et al. A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability. The Lancet Digital Health. 2021 Jan 1;3(1):e51–66. Available online at: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(20)30240-5/fulltext1.
Wen D, Khan SM, Xu AJ, Ibrahim H, Smith L, Caballero J, et al. Characteristics of publicly available skin cancer image datasets: a systematic review. The Lancet Digital Health. 2022 Jan 1;4(1):e64–74. Available online at: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(21)00252-1/fulltext1.
Ibrahim H, Liu X, Zariffa N, Morris AD, Denniston AK. Health data poverty: an assailable barrier to equitable digital health care. The Lancet Digital Health. 2021 Apr 1;3(4):e260–5. Available online at: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(20)30317-4/fulltext1.
Parikh RB, Teeple S, Navathe AS. Addressing Bias in Artificial Intelligence in Health Care. JAMA. 2019 Dec 24;322(24):2377–8. Available online at: https://jamanetwork.com/journals/jama/article-abstract/2756196 1.
Chen IY, Pierson E, Rose S, Joshi S, Ferryman K, Ghassemi M. Ethical Machine Learning in Healthcare. Annu Rev Biomed Data Sci. 2021 Jul;4:123–44. Available online at: https://www.annualreviews.org/doi/10.1146/annurev-biodatasci-092820-114757 1.
Bennett JE, Pearson-Stuttard J, Kontis V, Capewell S, Wolfe I, Ezzati M. Contributions of diseases and injuries to widening life expectancy inequalities in England from 2001 to 2016: a population-based analysis of vital registration data. The Lancet Public Health. 2018 Dec 1;3(12):e586–97. Available online at: https://www.thelancet.com/journals/lanpub/article/PIIS2468-2667(18)30214-7/fulltext 1.
Marmot M, Allen J, Boyce T, Goldblatt P, Morrison J. Health Equity in England: The Marmot Review 10 Years On - The Health Foundation [Internet]. London: Institute of Health Equity; 2020 [cited 2022 Apr 13]. Available online at: https://www.health.org.uk/publications/reports/the-marmot-review-10-years-on