The visual processes that support grasp planning are often studied by analyzing averaged kinematics of repeated movements, as in the literature on grasping and visual illusions. However, by recalibrating visuomotor mappings, the sensorimotor system can adjust motor outputs without changing visual processing, which complicates the interpretation of averaged behavior. We developed a dynamic model of grasp planning and adaptation that can explain why some studies find decrements in illusion effects on grasping while others do not. In two experiments, we tested grasping in a standard three-phase adaptation paradigm and analyzed adaptation aftereffects on the maximum grip aperture as well as the error correction parameters estimated by our model. Experiment 1 demonstrated that the model accounts for adaptive responses to positive and negative visual size perturbations. Experiment 2 supported the novel hypothesis that visuomotor mappings for grasp planning can compensate for opposing size perturbations when these perturbations are experienced in separate regions of space. Our findings serve to illustrate how the surprising flexibility of grasp adaptation can hide (especially in session-wise averages) the true effects of visual perturbations on the visual processes that drive grasp planning.