Identifying modifiable behaviors associated with prevention of activities of daily living (ADL) impairments is vital to implement preventive strategies for independent living in elderly. We aimed to examine the associations between objectively measured breaks in sedentary time with ADL impairments and physical independence. Cross-sectional assessments were carried out in 371 participants (131 male) aged 65–103 years from the Portuguese surveillance system of physical activity. Physical activity (PA) and sedentary time (ST) were assessed with accelerometry, and ADL impairments and physical independence with the self-reported 12-item composite physical function scale. Using ROC analyses a cut-off of 7 hourly breaks in sedentary time was identified which maximized the sensitivity and specificity in diagnosing physical dependence. Logistic regression analysis demonstrated that, independently of moderate-to-vigorous PA (MVPA), participants performing ≤7 hourly breaks in sedentary time had 2 to 7 fold increased odds for impairment in 10 of the 12 ADL. When stratifying ADL into basic, instrumental and advanced ADL we verified that less than 7 hourly breaks in sedentary time was associated with a 2 to 5 fold increased odds for impairments and physical dependence, independent of MVPA. No associations were observed between meeting PA guidelines and basic, instrumental, and advanced ADL impairment although time in MVPA was lower in participants showing impairments. In conclusion, the frequency of breaks in sedentary time in older ages is independently associated with lower risk for ADL impairments and physical dependence. Our findings support interventions to encourage older adults to increase overall PA by interrupting sedentary time.