Chatter is an instability condition in machining processes characterized by nonlinear behavior, such as the presence of limit cycles, jump phenomenon, subcritical Hopf and period doubling bifurcations. Although the use of nonlinear techniques has provided a better understanding of chatter, neither a unifying model nor an exact solution has yet been developed due to the intricacy of the problem. This work proposes a weakly nonlinear model with square and cubic terms in both structural stiffness and regenerative terms, to represent self-excited vibrations in machining. An approximate solution is derived by using the method of multiple scales. In addition, a qualitative analysis of the effect of the nonlinear parameters on the stability of the system is performed. The structural cubic term gives a better representation of the nonlinear behavior, whereas the square term represents a distant attractor in the stability chart. Instability due to subcritical Hopf bifurcations is established in terms of the eigenvalues of the model in normal form. An important contribution of this analysis is the representation of hysteresis in terms of new lobes within the conventional stability limits, useful in restoring stability. This analysis leads to a further understanding of the nonlinear behavior of regenerative chatter.