Over 50% of cancer survivors lose their job or quit working. Cancer survivors who experience job loss may face different challenges regarding return to work, compared to cancer survivors with employers. This qualitative study aimed to explore barriers and facilitators for return to work in cancer survivors with job loss and in insurance physicians who assist cancer survivors in their return to work. We conducted five focus groups and one interview (cancer survivors, N = 17; insurance physicians, N = 23). Topics included, among others, experience of job loss and barriers and facilitators for return to work. Data were audio recorded and analysed using thematic analysis. Our main finding was that cancer survivors experienced a double loss: loss of job on top of loss of health. As a result, cancer survivors feared for job applications, lacked opportunities to gradually increase work ability, and faced reluctance from employers in hiring them. Insurance physicians expressed a need for more frequent and longer consultations with cancer survivors with job loss. We conclude that cancer survivors who experience double loss encounter specific barriers in the return to work process. This calls for a tailored approach regarding return to work support.