Males and females share most of their genomes, and differences between the sexes can therefore not evolve through sequence divergence in protein coding genes. Sexual dimorphism is instead restricted to occur through sex-specific expression and splicing of gene products. Evolution of sexual dimorphism through these mechanisms should, however, also be constrained when the sexes share the genetic architecture for regulation of gene expression. Despite these obstacles, sexual dimorphism is prevalent in the animal kingdom and commonly evolves rapidly. Here, we ask whether the genetic architecture of gene expression is plastic and easily molded by sex-specific selection, or if sexual dimorphism evolves rapidly despite pervasive genetic constraint. To address this question, we explore the relationship between the intersexual genetic correlation for gene expression (rMF), which captures how independently genes are regulated in the sexes, and the evolution of sex-biased gene expression. Using transcriptome data from Drosophila melanogaster, we find that most genes have a high rMF and that genes currently exposed to sexually antagonistic selection have a higher average rMF than other genes. We further show that genes with a high rMF have less pronounced sex-biased gene expression than genes with a low rMF within D. melanogaster and that the strength of the rMF in D. melanogaster predicts the degree to which the sex bias of a gene’s expression has changed between D. melanogaster and six other species in the Drosophila genus. In sum, our results show that a shared genome constrains both short- and long-term evolution of sexual dimorphism.