Fluctuating cognition (FC) is a core feature of dementia with Lewy bodies (DLB) but is challenging to assess. This study assessed the reliability and validity of the Clinician Assessment of Fluctuation (CAF), which assesses FC in patients with dementia. Interrater agreement of CAF outcomes (FC present and FC severe) was evaluated between physicians and nonphysicians in 141 patients with Alzheimer’s disease (AD) or DLB. Frequency of CAF outcomes by clinical and neuropathological diagnosis was examined. We found that interrater reliability was fair on FC present and almost perfect on FC severe, and both outcomes were higher in patients with clinical DLB than with clinical AD and were qualitatively more often endorsed in cases with neuropathological evidence of Lewy bodies. We conclude that the CAF is a reliable measure of FC and can be valuable in differential dementia diagnosis.