How does the brain process and control languages that are learned at a different age, when proficiency in all these languages is high? Early acquired strong languages are likely to have higher baseline activation levels than later learned less-dominant languages. However, it is still largely unknown how the activation levels of these different languages are controlled, and how interference from an irrelevant language is prevented. In this magnetoencephalography (MEG) study on language switching during auditory perception, early Finnish-Swedish bilinguals (N = 18) who mastered English with high proficiency after childhood were presented with spoken words in each of the three languages, while performing a simple semantic categorisation task. Switches from the later learned English to either of the native languages resulted in increased neural activation in the superior temporal gyrus (STG) 400–600 ms after word onset (N400m response), whereas such increase was not detected for switches from native languages to English or between the native languages. In an earlier time window of 350–450 ms, English non-switch trials showed higher activation levels in the inferior frontal gyrus (IFG), pointing to ongoing inhibition of the native languages during the use of English. Taken together, these asymmetric switch costs suggest that native languages are suppressed during the use of a non-native language, despite the receptive nature of the language task. This effect seems to be driven mostly by age of acquisition or language exposure, rather than proficiency. Our results indicate that mechanisms of control between two native languages differ from those of a later learned language, as upbringing in an early bilingual environment has likely promoted automatiation of language control specifically for the native languages.