Stephan and Stephan (1985) defined intergroup anxiety (IA) as anticipatory distress about interactions with people from groups one does not belong to. Researchers have yet, however, to identify precise conceptual boundaries for IA. For instance, it is unclear whether IA manifests with different intergroup concerns for members of groups that vary in power and privilege. In this study, I took a step toward delineating evidence-based conceptual boundaries for IA. I used SEM to assess how several ostensibly salient constructs are related and might vary as a function of racial group membership (Black, White). Results gave support to the hypothesis that the construct represents different fears and has a different factor structure for each group. For the privileged (White) group, IA appears to be more characterized as a social anxiety correlate, while for the marginalized (Black) group, IA might be more characterized as fears and precautions about intergroup interactions focused on one’s physical and social safety.