Objectives: Disorder has been measured by various data sources; however, little attention has been given to comparing the construct validity of different measures obtained through various methods in capturing social disorder and related phenomena. Methods: The multitrait-multimethod approach was used to triangulate the consistency between social disorder, prostitution and drug activity across resident surveys, systematic social observations, and police calls for service data. Results: Prostitution and drug activity showed convergent validity, while there was little evidence that social disorder was consistently measured across the three methods. None of the three social problem measures showed high discriminant validity. Drug activity seems to have highest trait-specific discriminant validity across measures, and prostitution is the most identifiable measure across data sources. Social disorder was found to have low discriminant validity. However, the agreement between databases varies across the type of social problems. Conclusions: Social disorder appears to the most difficult concept to define and measure consistently. The lack of correspondence across data sources cautions against the use of a single source of information in studying disorder. Future studies should explore the factors that shape perceptions of disorder and how to best measure disorder to test the broken windows thesis and related concepts.