The saponins, as components of tea seed meal, are undesirable hemolytic components and should be degraded for reducing their hemolytic activity in order to be used in animal feed. In this study, β-glucuronidase was verified to be a potent hydrolase of tea seed saponins to reduce their hemolytic activity and a β-glucuronidase-producing Lactobacillus crustorum strain was screened from raw bovine milk. Next, solid-state fermentation with the isolated L. crustorum and a Bacillus subtilis natto strain, which can produce cellulase and hence improve the fermentation performance of tea seed meal, was carried out for detoxification of tea seed meal. The 50% hemolytic dosage (HD50) value of tea seed saponins was increased from 6.69 to 27.43 μg mL−1. The results of LC-MS analysis showed that the percentage of saponin aglycones increased from 30.95 to 84.25% after the fermentation. According to the roles of sugar moieties in hemolytic activity, and the enzymatic hydrolysis characteristics of β-glucuronidase, the degradation of tea seed saponins from glucosides to aglycones may contribute to the reduction of hemolytic activity. Therefore, tea seed meal may be used as animal feed after fermentation with the tested saponin-degrading microbial strains.