The manipulation of numbers required during calculation is known to rely on working memory (WM) resources. Here, we investigated the respective contributions of verbal and/or spatial WM manipulation brain networks during the addition of four numbers performed by adults, using functional magnetic resonance imaging (fMRI). Both manipulation and maintenance tasks were proposed with syllables, locations, or two-digit numbers. As compared to their maintenance, numbers manipulation (addition) elicited increased activation within a widespread cortical network including inferior temporal, parietal, and prefrontal regions. Our results demonstrate that mastery of arithmetic calculation requires the cooperation of three WM manipulation systems: an executive manipulation system conjointly recruited by the three manipulation tasks, including the anterior cingulate cortex (ACC), the orbital part of the inferior frontal gyrus, and the caudate nuclei; a left-lateralized, language-related, inferior fronto-temporal system elicited by numbers and syllables manipulation tasks required for retrieval, selection, and association of symbolic information; and a right superior and posterior fronto-parietal system elicited by numbers and locations manipulation tasks for spatial WM and attentional processes. Our results provide new information that the anterior intraparietal sulcus (IPS) is involved in tasks requiring a magnitude processing with symbolic (numbers) and nonsymbolic (locations) stimuli. Furthermore, the specificity of arithmetic processing is mediated by a left-hemispheric specialization of the anterior and posterior parts of the IPS as compared to a spatial task involving magnitude processing with nonsymbolic material.