A PVA-based magnetic-sensitive hydrogel (ferrogel) was fabricated by physical cross-linking through a freezing–thawing method. The influence of the constituting components, i.e., Fe3O4 and PVA, on the magnetic-sensitive behavior of the ferrogels was systematically investigated in terms of permeability coefficient (P), partition coefficient (H), space restriction and magnetization. The results show that, the P value in these ferrogels decreases and displays a magnetic sensitivity when it is subjected to magnetic field (MF), which is correlated with the change of H value. In addition, it was found that although the factor of space restriction or magnetization exerts opposite effect on resulting magnetic-sensitive behavior, the superior magnetic-sensitive behavior was observed for the ferrogels with an optimal composition of 17–34% Fe3O4 and 10–12.5% PVA, and can be well correlated with theoretical calculation from the critical parameters of available free volume per nanoparticle (Vfree) and magnetization. A map of magnetic-sensitive behavior was constructed, where a region with relatively stable and highly stimuli-responsive behavior in terms of the concentration of Fe3O4 was observed, however, below or above the “saturation” region (17–34% Fe3O4), a reduction in the magnetic-sensitive behavior was detected. The resulting ferrogels can be engineered with a precise control of the opening and closure of pore configuration, which allows a burst release or no-release action of therapeutically active agent to be controlled externally and magnetically. This suggests that this type of ferrogel can be considered as a class of novel magnetically-tunable drug delivery system.