aDepartment of Radiology and Nuclear Medicine, VUmc, Amsterdam, The NetherlandsbDepartment of Anatomy and Neuroscience, VUmc, Amsterdam, The NetherlandscDepartment of Neurology, CHU-Bordeaux, Bordeaux, FrancedUniversity of Bordeaux, Bordeaux, FranceeInserm U-1215 Magendie Neurocenter-Pathophysiology of Neural Plasticity, CHU-Bordeaux, Bordeaux, FrancefDepartment of Neurology, Medical University of Graz, Graz, AustriagNeuroimaging Research Unit, Institute of Experimental Neurology, Division of Neuroscience, San Raffaele Scientific Institute, UniSR, Milan, ItalyhDepartment of Neuroinflammation, Institute of Neurology, UCL, London, UKiDepartment of Neurobiology, Care Sciences and Society, Karolinska Institute, Stockholm, SwedenjLaboratory of Epidemiology, Neuroimaging and Telemedicine, IRCCS Centro “S. Giovanni di Dio-F.B.F.”, Brescia, ItalykMemory Clinic and LANVIE - Laboratory of Neuroimaging of Aging, HUG, Geneva, SwitzerlandlCentre d'Esclerosi Múltiple de Catalunya (Cemcat), Department of Neurology/Neuroimmunology, VHIR, Barcelona, SpainmMagnetic Resonance Unit, Department of Radiology (IDI), VHIR, Barcelona, SpainnDivision of Neuroradiology, Vascular and Interventional Radiology, Department of Radiology, Medical University of Graz, Graz, AustriaoDepartment of Neurology, Glostrup University Hospital, Copenhagen, DenmarkpUK/NIHR UCL-UCLH Biomedical Research Centre, Institute of Neurology, UCL, London, UKqNeurologic Clinic and Policlinic, University Hospital, University of Basel, SwitzerlandrInstitutes of Neurology & Healthcare Engineering, UCL, London, UK
Checking for direct PDF access through Ovid
Background and Purpose:In vivo identification of white matter lesions plays a key-role in evaluation of patients with multiple sclerosis (MS). Automated lesion segmentation methods have been developed to substitute manual outlining, but evidence of their performance in multi-center investigations is lacking. In this work, five research-domain automated segmentation methods were evaluated using a multi-center MS dataset.Methods:70 MS patients (median EDSS of 2.0 [range 0.0–6.5]) were included from a six-center dataset of the MAGNIMS Study Group (www.magnims.eu) which included 2D FLAIR and 3D T1 images with manual lesion segmentation as a reference. Automated lesion segmentations were produced using five algorithms: Cascade; Lesion Segmentation Toolbox (LST) with both the Lesion growth algorithm (LGA) and the Lesion prediction algorithm (LPA); Lesion-Topology preserving Anatomical Segmentation (Lesion-TOADS); and k-Nearest Neighbor with Tissue Type Priors (kNN-TTP). Main software parameters were optimized using a training set (N = 18), and formal testing was performed on the remaining patients (N = 52). To evaluate volumetric agreement with the reference segmentations, intraclass correlation coefficient (ICC) as well as mean difference in lesion volumes between the automated and reference segmentations were calculated. The Similarity Index (SI), False Positive (FP) volumes and False Negative (FN) volumes were used to examine spatial agreement. All analyses were repeated using a leave-one-center-out design to exclude the center of interest from the training phase to evaluate the performance of the method on ‘unseen’ center.Results:Compared to the reference mean lesion volume (4.85 ± 7.29 mL), the methods displayed a mean difference of 1.60 ± 4.83 (Cascade), 2.31 ± 7.66 (LGA), 0.44 ± 4.68 (LPA), 1.76 ± 4.17 (Lesion-TOADS) and −1.39 ± 4.10 mL (kNN-TTP). The ICCs were 0.755, 0.713, 0.851, 0.806 and 0.723, respectively. Spatial agreement with reference segmentations was higher for LPA (SI = 0.37 ± 0.23), Lesion-TOADS (SI = 0.35 ± 0.18) and kNN-TTP (SI = 0.44 ± 0.14) than for Cascade (SI = 0.26 ± 0.17) or LGA (SI = 0.31 ± 0.23). All methods showed highly similar results when used on data from a center not used in software parameter optimization.Conclusion:The performance of the methods in this multi-center MS dataset was moderate, but appeared to be robust even with new datasets from centers not included in training the automated methods.HighlightsMuch-needed study on quantitative evaluation and objective comparison of WM lesion segmentation methods.Using different scanners and different MR protocols in a real-life setting similar to phase-III trials and everyday clinical practice.The methods perform almost equally well whether parameter tuning is performed using data from the same center or not.