Previous work with patients with hippocampal damage and profound declarative memory impairments using a collaborative referencing task that requires social interaction and constant communication with a familiar partner reported collaborative learning at an equal rate to comparison participants for referential labels resulting in rapid and efficient communication or “common ground” (Duff et al., 2006). Here, we test the hypothesis that a key aspect of the collaborative learning is intact social-emotional communication, and that individuals with damage to social-emotion-related brain areas (e.g., right hemisphere (RH)), ventromedial prefrontal cortex (VMPC)) will be impaired in the acquisition and use of common ground in communication with a familiar partner. Participants included 3 patients with RH damage and 6 patients with VMPC damage, and 8 healthy comparison participants, each interacting with a familiar partner. Results indicated that both patient groups were impaired on measures of time-to-completion and in card placement accuracy, particularly early in the task. The patients with VMPC damage performed superior to the right hemisphere patients. The findings provide preliminary support for the notion that neural systems involved in social-emotional processing are important for the acquisition and use of common ground in social communication.