About
STANDING Together: Building STANdards for data Diversity, INclusivity, & Generalisablity. Established in 2021 as part of the NHS AI Lab’s AI Ethics initiative, it is a partnership between over 30 academic, regulatory, policy, industry, and charitable organisations worldwide. STANDING Together is funded by the NHS AI Lab at the NHS Transformation Directorate and The Health Foundation and managed by the National Institute for Health and Care Research (AI_HI200014).
The Opportunity of Artificial Intelligence (AI) in Healthcare
The power of AI lies in its ability to learn patterns from large amounts of data, in a way that exceeds human abilities. However, this also means the reliability of AI algorithms is closely linked to the data it is trained upon, and may perform poorly when confronted by new data examples – a failure of ‘AI generalisability’. To be sure that algorithms work for everybody, we need to test them on datasets that represent the diverse range of people it is intended to be used in.
The Problem
There is concern that many health datasets do not adequately represent the diversity of populations. The extent of the problem is not yet known because many datasets do not provide detailed demographic information - a failure of 'data transparency'. This problem has arisen partly because the creators of large datasets for AI often prioritise quantity of data over quality, inclusivity or fairness. We need recommendations to encourage transparency around ‘who’ is represented in the data, ‘how’ they are represented, and how health data is used.
Addressing the Problem Together
This project has developed recommendations that ensure datasets for training and testing AI systems are diverse, inclusive, and promote AI generalisability.
Patients, public, health professionals, researchers, ethicists and policy-makers are working together to agree what the essential criteria for datasets should be. We have developed new recommendations for transparency of AI datasets, which will help gatekeepers (regulators, commissioners, policy-makers and health data institutions) assess whether datasets and the algorithms developed by them are suitable for the target population. This means we will have better datasets for development and testing of AI and, and in the long-term, better health outcomes for all.
By getting the data foundation right, STANDING Together ensures that 'no-one is left behind' as we seek to unlock the benefits of AI in healthcare.
Want to find out more?
Here's a video with Dr Joe Alderman talking about the STANDING Together Project
Health Data Research Midlands Inequalities and Diversity Webinar, October 2023
What have we done so far?
1
Built consensus on best practice for dataset curation and use
Scoping review of existing dataset standards
Conducted an international multi-stakeholder eDelphi study
2
Mapped dataset gaps in priority disease areas
Identified datasets used in priority disease areas: COVID-19, breast cancer, heart failure, ophthalmology and skin cancer.
Assessed datasets using proposed recommendations
3
Identified barriers and enablers for better curation and use of data
Conduct semi-structured interviews with dataset curators/users about implementing the recommendations
How did we build consensus?
We conducted a multi-staged e-Delphi study in 2023-2024 where clinical and AI experts, sociology and ethics scholars, policy-makers, editors, patients, regulators and funders helped us refine a list of recommendations to improve bias in health data and AI.
We spoke to many experts before and after, as well as patient groups and international communities. Overall, we consulted with >350 stakeholders across 58 countries.
Funding and Support
