The ability to generate regular spatial arrangements of particles is an important technological and fundamental aspect of colloidal science. We showed that colloidal particles confined to a few-micrometer-thick layer of a nematic liquid crystal form two-dimensional crystal structures that are bound by topological defects. Two basic crystalline structures were observed, depending on the ordering of the liquid crystal around the particle. Colloids inducing quadrupolar order crystallize into weakly bound two-dimensional ordered structure, where the particle interaction is mediated by the sharing of localized topological defects. Colloids inducing dipolar order are strongly bound into antiferroelectric-like two-dimensional crystallites of dipolar colloidal chains. Self-assembly by topological defects could be applied to other systems with similar symmetry.