Characteristics of publicly available skin cancer image datasets: a systematic review

Freely available (open access) datasets containing skin images are frequently used to develop deep learning algorithms for skin cancer diagnosis. As these algorithms are heavily influenced by the images that they are trained on, it is important that the composition and characteristics of datasets are outlined, such as which populations images are taken from. This information is often detailed in metadata accompanying individual datasets.

Prior to this review, the total number of publicly available datasets globally and their respective content had not previously been explored. We searched for open access skin image datasets used to develop machine learning algorithms for skin cancer diagnosis and systematically evaluated their characteristics including associated metadata.

We identified 21 open access datasets containing 106,950 freely available images. With regards to their general characteristics: 

• Fourteen of 21 datasets reported which country they originated from and of those, eleven contained images from Europe, North America or Oceania only. 

• Nineteen of 21 datasets contained images from one modality only (either macroscopic photographs or dermoscopic images – pictures taken with a special hand-held magnifier). Only two of the 21 datasets included images taken with both of these methods, which better reflects how dermatologists examine lesions in clinical practice. 

• Many datasets were also missing other important information, such as how images were chosen to be included, and evidence of ethical approval or patient consent. 

Regarding metadata reporting for individual images in the open access datasets: 

• Approximately 75% of individual images had metadata labels for age, sex and lesion site. 

• Only 2% of individual images had metadata labels for skin type, and only 1% for ethnicity. 

• Of the 2,436 images from three datasets where skin type information was available, ten images were from subjects with Fitzpatrick type V (brown) skin, and one image was from an individual with Fitzpatrick type VI (dark brown or black) skin. 

• Of the 1,585 images from two datasets where ethnicity was available, no images were from individuals with an African, Afro-Caribbean or South Asian background. 

Our review highlighted that better reporting of dataset characteristics and metadata is required with the aim of producing more transparent skin image datasets. Quality standards outlining what should be reported in datasets may facilitate this through providing guidance for dataset curators. Dataset standards can also detail what constitutes a representative dataset and who should be included, with the aim of producing datasets that are representative of the target populations that any developed algorithms will be deployed in, translating into more effective algorithms for all groups of people.