Down syndrome, the most common single cause of human birth defects, produces alterations in physical growth and mental retardation. If missed before birth, the early detection of Down syndrome is crucial for the management of patients and disease. However, the diagnostic accuracy for pediatricians prior to cytogenetic results is moderate and the access to specialists is limited in many social and low-economic areas. In this study, we propose a simple, non-invasive and automated framework for Down syndrome detection based on disease-specific facial patterns. Geometric and local texture features are extracted based on automatically detected anatomical landmarks to describe facial morphology and structure. To accurately locate the anatomical facial landmarks, a hierarchical constrained local model using independent component analysis (ICA) is proposed. We also introduce a data-driven ordering method for selecting dominant independent components in ICA. The hierarchical structure of the model increases the accuracy of landmark detection by fitting separate models to different groups. Then the most representative features are selected and we also demonstrate that they match clinical observations. Finally, a variety of classifiers are evaluated to discriminate between Down syndrome and healthy populations. The best performance achieved 0.967 accuracy and 0.956 Symbol score using combined features and linear discriminant analysis. The method was also validated on a dataset with mixed genetic syndromes and high performance (0.970 accuracy and 0.930 Symbol score) was also obtained. The promising results indicate that our method could assist in Down syndrome screening effectively in a simple, non-invasive way, and extensible to detection of other genetic syndromes.