Extracellular vesicles (EVs) are heterogeneous multi-signal messengers that support cancer growth and dissemination by mediating the tumor-stroma crosstalk. Exosomes are a subtype of EVs that originate from the limiting membrane of late endosomes, and as such contain information linked to both the intrinsic cell “state” and the extracellular signals cells received from their environment. Resolving the signals affecting exosome biogenesis, cargo sorting and release will increase our understanding of tumorigenesis. In this review we highlight key cell biological processes that couple exosome biogenesis to cargo sorting in cancer cells. Moreover, we discuss how the bidirectional communication between tumor and non-malignant cells affect cancer growth and metastatic behavior.