Though Drug delivery systems have achieved accumulation at tumor sites via passive targeting and active targeting, the therapeutic effects are far from perfect. The unsatisfactory results are mainly due to limited drug release from the nanocarriers at tumor sites, while the pharmacological activities of the drug are attributed to the concentration of the free drug and the time maintained at the pharmacological targets. A pH-responsive chitosan based glycolipid-like nanocarrier (CSO-FBA-SA) was fabricated by conjugating stearyl alcohol (SA) to chitosan oligosaccharide (CSO) with the linkage of 4-formylbenzoic acid (FBA). FBA was a kind of aromatic aldehyde carbonyl compounds, which can form the benzoic-imine bond. In the presence of a Schiff’s base structure, the carrier showed improved properties and could be quickly degraded in an acidic environment. In order to explore the process and mechanism of the nanocarriers in focal cells, the method for determining the intracellular concentration of released free doxorubicin was established, and the time-dependent change of the DOX-loaded micelles was revealed. The sight of drug release was also obtained with CLSM. The cytotoxicity of the CSO-FBA-SA/DOX against human breast cancer MCF-7 cells increased by 2.75-fold and 3.77-fold in comparison with the CSO-SA/DOX and DOX, respectively. Furthermore, the CSO-FBA-SA/DOX showed a 2.12-fold higher cytotoxicity against the MCF-7 cells than that treated against human ovarian cancer SKOV-3 cells with lower intracellular pH value, which indicated that the cellular inhibition positively correlated with the intracellular pH value. High tumor accumulation and fast drug release of the CSO-FBA-SA/DOX in tumor was responsible for the remarkable tumor growth inhibitory effect. Moreover, the CSO-FBA-SA/DOX could selectively respond to the acidic environment and release DOX in tumor only, which had relatively minimal cytotoxicity towards normal tissues. The results showed that this newly developed glycolipid-like nanocarrier could act as a potential vector for delivering the drug effectively with a low systemic toxicity.