When a behaviorally relevant stimulus has been previously associated with reward, behavioral responses are faster and more accurate compared to equally relevant but less valuable stimuli. Conversely, task irrelevant stimuli that were previously associated with a high reward can capture attention and distract processing away from relevant stimuli (e.g. the chocolate bar in the pantry when you are looking for a nice healthy apple). While increasing the value of task-relevant stimuli systematically up-regulates neural responses in early visual cortex to facilitate information processing, it is not clear if the value of task-irrelevant distractors influences behavior via competition in early visual cortex or via competition at later stages of decision-making and response selection. Here, we measured fMRI in human visual cortex while subjects performed a value-based learning task, and applied a multivariate inverted encoding model to assess the fidelity of distractor representations in early visual cortex. We found that the fidelity of neural representations related to task-irrelevant distractors increased when the distractors were previously associated with a high reward. Moreover, this value-based modulation of distractor representations only occurred when the distractors were previously selected as targets on preceding trials. Together, these findings suggest that value-driven attentional capture begins with sensory modulations of distractor representations in early areas of visual cortex.