Theories of attention commonly refer to the “attentional template” as the collection of features in working memory that represent the target of visual search. Many models of attention assume that the template contains a veridical representation of target features, but recent studies have shown that the target representation is “shifted” away from distractor features in order to optimize their distinctiveness and facilitate visual search. Here, we manipulated the probability of target-similar distractors during a visual search task in 2 groups, and separately measured the contents of the attentional template. We hypothesized that having a high probability of target-similar distractors would increase pressure to shift and/or sharpen the target representation in order to increase the distinctiveness of targets from distractors. We found that the high-similarity group experienced less distractor interference during visual search, but only for highly target-similar distractors. Additionally, while both groups shifted the target representation away from the actual target color, the high-similarity group also had a sharper representation of the target color. We conclude that the contents of the attentional template in working memory can be flexibly adjusted with multiple mechanisms to increase target-to-distractor distinctiveness and optimize attentional selection.