An increasing number of herbicides are found in our groundwater environments. This underlines the need for examining the effects of herbicide exposure on the indigenous groundwater microbial communities, as microbial degradation is the major process responsible for the complete removal of most contaminants. We examined the effect of in situ exposure to realistic low concentrations of herbicides on the microbial diversity and community structure of sub-surface sediments from a shallow aquifer near Vejen (Denmark). Three different community analyses were performed: colony morphology typing, sole-carbon source utilisation in Biolog®EcoPlates, and denaturing gradient gel electrophoresis. Cluster analysis demonstrated that the microbial communities of those aquifer sediments that acclimated to the herbicide exposure also had similar community structure. This observation was concurrent for all three community analyses. In contrast, no significant effect was found on the bacterial diversity, except for the culturable fraction where a significantly increased richness and Shannon index was found in the herbicide acclimated sediments. The results of this study show that in situ exposure of sub-surface aquifers to realistic low concentrations of herbicides may alter the overall structure of a natural bacterial community, although significant effects on the genetic diversity and carbon substrate usage cannot be detected. The observed impact was probably due to indirect effects. In future investigations, the inclusion of methods that specifically detect relevant microbial sub-populations and functional genes is therefore recommended.