Children with Down Syndrome (DS) experience cognitive delays with language being one of the most impaired domains. Exploring the effects of congenital heart defects (CHD), hospitalization, hearing impairment, and parental concern can provide a more precise view of factors affecting language development. Participants were 49 children with DS, 22 to 54 months of age. Expressive and receptive vocabulary size was obtained using a word count with the MacArthur Communication Development Inventory (MCDI). Medical information was obtained from the child’s medical file. Results showed expressive vocabulary was marginally significantly different between children with DS and no CHD, a CHD that did not require surgery, and a CHD that did require surgery, such that children with a CHD requiring surgery had the smallest vocabulary. Children had significantly more health problems when they had a CHD that required surgery. Expressive and receptive vocabularies were significantly smaller for children with hearing impairment.