Previous studies have shown that information held in visual working memory is represented in the occipital, parietal, and frontal cortices. However, less is known about whether the mnemonic information of multi-feature objects is modulated by task demand in the parietal and frontal regions. To address this question, we asked participants to remember either color or orientation of one of the two colored gratings for a delay. Using fMRI and an inverted encoding model, we reconstructed population-level, feature-selective responses in the occipital, parietal and frontal cortices during memory maintenance. We found that not only orientation but also color information can be maintained in higher-order parietal and frontal cortices as well as the early visual cortex when it was cued to be remembered. Conversely, neither the task-irrelevant feature of the cued object, nor any feature of the uncued object was maintained in the occipital, parietal, or frontal cortices. These results suggest a highly selective mechanism of visual working memory that maintains task-relevant features only.