We introduce the concept of fusion algebras at algebraic level, as a purely algebraic concept for the fusion algebras which appear in conformal field theory in mathematical physics. We first discuss the connection between fusion algebras at algebraic level and character algebras, a purely algebraic concept for Bose-Mesner algebras of association schemes. Through this correspondence, we establish the condition when the matrix S of a fusion algebra at algebraic level is unitary or symmetric. We construct integral fusion algebras at algebraic level, from association schemes, in particular from group association schemes, whose matrix S is unitary and symmetric. Finally, we consider whether the modular invariance property is satisfied or not, namely whether there exists a diagonal matrix T satisfying the condition (ST)3 = S2. We prove that this property does not hold for some integral fusion algebras at algebraic level coming from the group association scheme of certain groups of order 64, and we also prove that the (nonintegral) fusion algebra at algebraic level obtained from the Hamming association scheme H(d, q) has the modular invariance property.