Purpose: To evaluate different centerline analysis applications using objective ground truth from realistic aortic aneurysm phantoms with precisely defined geometry and centerlines to overcome the lack of unknown true dimensions in previously published in vivo validation studies. Methods: Three aortic phantoms were created using computer-aided design (CAD) software and a 3-dimensional (3D) printer. Computed tomography angiograms (CTAs) of phantoms and 3 patients were analyzed with 3 clinically approved and 1 research software application. The 3D centerline coordinates, intraluminal diameters, and lengths were validated against CAD ground truth using a dedicated evaluation software platform. Results: The 3D centerline position mean error ranged from 0.7±0.8 to 2.9±2.5 mm between tested applications. All applications calculated centerlines significantly different from ground truth. Diameter mean errors varied from 0.5±1.2 to 1.1±1.0 mm among 3 applications, but exceeded 8.0±11.0 mm with one application due to an unsteady distortion of luminal dimensions along the centerline. All tested commercially available software tools systematically underestimated centerline total lengths by −4.6±0.9 mm to −10.4±4.3 mm (maximum error −14.6 mm). Applications with the highest 3D centerline accuracy yielded the most precise diameter and length measurements. Conclusion: One clinically approved application did not provide reproducible centerline-based analysis results, while another approved application showed length errors that might influence stent-graft choice and procedure success. The variety and specific characteristics of endovascular aneurysm repair planning software tools require scientific evaluation and user awareness.