The effectiveness of lumbar fusion for chronic low back pain after surgery for disc herniation has not been evaluated in a randomized controlled trial. The aim of the present study was to compare the effectiveness of lumbar fusion with posterior transpedicular screws and cognitive intervention and exercises. Sixty patients aged 25–60 years with low back pain lasting longer than 1 year after previous surgery for disc herniation were randomly allocated to the two treatment groups. Experienced back surgeons performed transpedicular fusion. Cognitive intervention consisted of a lecture intended to give the patient an understanding that ordinary physical activity would not harm the disc and a recommendation to use the back and bend it. This was reinforced by three daily physical exercise sessions for 3 weeks. The primary outcome measure was the Oswestry Disability Index (ODI). Outcome data were analyzed on an intention-to-treat basis. Ninety-seven percent of the patients, including seven of eight patients who had either not attended treatment (n = 5) or changed groups (n = 2), completed 1-year follow-up. ODI was significantly improved from 47 to 38 after fusion and from 45 to 32 after cognitive intervention and exercises. The mean difference between treatments after adjustment for gender was −7.3 (95% CI −17.3 to 2.7, p = 0.15). The success rate was 50% in the fusion group and 48% in the cognitive intervention/exercise group. For patients with chronic low back pain after previous surgery for disc herniation, lumbar fusion failed to show any benefit over cognitive intervention and exercises.