Phantoms, both numerical (software) and physical (hardware), can serve as a gold standard for the validation of MRI methods probing the brain microstructure. This review aims to provide guidelines on how to build, implement, or choose the right phantom for a particular application, along with an overview of the current state-of-the-art of phantoms dedicated to study brain microstructure with MRI. For physical phantoms, we discuss the essential requirements and relevant characteristics of both the (NMR visible) liquid and (NMR invisible) phantom materials that induce relevant microstructural features detectable via MRI, based on diffusion, intra-voxel incoherent motion, magnetization transfer or magnetic susceptibility weighted contrast. In particular, for diffusion MRI, many useful phantoms have been proposed, ranging from simple liquids to advanced biomimetic phantoms consisting of hollow or plain microfibers and capillaries. For numerical phantoms, the focus is on Monte Carlo simulations of random walk, for which the basic principles, along with useful criteria to check and potential pitfalls are reviewed, in addition to a literature overview highlighting recent advances. While many phantoms exist already, the current review aims to stimulate further research in the field and to address remaining needs.