Potential links between language and numbers and the laterality of symbolic number representations in the brain are still debated. Furthermore, reports on bilingual individuals indicate that the language-number interrelationships might be quite complex. Therefore, we carried out a visual half-field (VHF) and dichotic listening (DL) study with action words and different forms of symbolic numbers used as stimuli to test the laterality of word and number processing in single-, dual-language and mixed -task and language- contexts. Experiment 1 (VHF) showed a significant right visual field/left hemispheric advantage in response accuracy for action word, as compared to any form of symbolic number processing. Experiment 2 (DL) revealed a substantially reversed effect – a significant right ear/left hemisphere advantage for arithmetic operations as compared to action word processing, and in response times in single- and dual-language contexts for number vs. action words. All these effects were language independent. Notably, for within-task response accuracy compared across modalities significant differences were found in all studied contexts. Thus, our results go counter to findings showing that action-relevant concepts and words, as well as number words are represented/processed primarily in the left hemisphere. Instead, we found that in the auditory context, following substantial engagement of working memory (here: by arithmetic operations), there is a subsequent functional reorganization of processing single stimuli, whether verbs or numbers. This reorganization – their weakened laterality – at least for response accuracy is not exclusive to processing of numbers, but the number of items to be processed. For response times, except for unpredictable tasks in mixed contexts, the “number problem” is more apparent. These outcomes are highly relevant to difficulties that simultaneous translators encounter when dealing with lengthy auditory material in which single items such as number words (and possibly other types of key words) need to be emphasized. Our results may also shed a new light on the “mathematical savant problem”.