Creativity is a fundamental component of innovation and critical for long-term business success. Identifying the products and ideas that are most creative, and therefore worthy of further development and investment, is an essential part of the creative process. However, experimental research into creativity over the past 20 years has yielded inconsistent and contradictory results. Moreover, this same research has shown that organizations struggle to identify their most creative products and ideas for further development. Critics suggest organizational creativity research may suffer from measurement misspecification due to a misalignment between existing construct definitions of creativity as a response that is both “novel and useful” and experimental studies that use only a single item, creativity, to measure creative output. This research investigates whether the theorized misalignment may be due, in part, to the research use of judges with little or no experience in the creative domain and their failure to understand the criteria, approaches, and techniques that expert judges actually use to assess creative output. To better understand how these issues may affect research results, this research utilized Naturalistic Decision Making field-study methods to investigate how expert judges assess the creative output of experienced professionals in the setting of a creativity awards contest. Through a series of interviews, observations, think-aloud protocols, and simulations with expert judges of creativity award contests, this research identifies six factors experts use to assess professional level creative output, and uncovers the processes, approach, and challenges involved in the real-world assessment of creative products and ideas. Recommendations for how assessing creativity can be improved in research and practice are discussed as well as suggestions for future research.