Economists typically rely on “correspondence audit studies” to examine discrimination and analyze how different candidate characteristics are valued by employers. If someone wanted to figure out how unemployment spells, military service, or minority status affects the likelihood of a candidate being hired, correspondence audit studies are a common way to find an answer. Despite its strengths, this approach has some glaring issues that make a new method of obtaining the same important insights invaluable.
Colin D. Sullivan, an assistant professor of economics at Purdue, and his coauthors Judd B. Kessler and Corinne Low, have developed an entirely new technique to study employer preferences and detect discrimination. They call this experimental approach “incentivized resume rating” (IRR) and outline its costs, benefits, and future applications in their paper “Incentivized Resume Rating: Eliciting Employer Preferences without Deception” published in the American Economic Review.
The main problem with correspondence audit studies is that they use deception to solicit relevant information from employers. In a correspondence audit study, a researcher sends out fake resumes under the premise that they are real to measure employer responses to different candidate characteristics. This practice not only wastes the employer’s time evaluating fake resumes and pursuing nonexistent candidates, but also damages the validity of future research and harms real job seekers whose resumes are similar to those sent by researchers.
Conversely, the incentivized resume rating system developed by Sullivan and his colleagues avoids using deception altogether. Using IRR, researchers inform employers that the resumes they send are not from real candidates from the outset. Those conducting the study then use the preferences the employers provide based on the fake resumes to match them with real job seekers. Unlike correspondence audit studies, incentivized resume rating rewards employers for their participation and motivates them to provide honest answers without deception.
Sullivan believes that there are two important reasons researchers should avoid deception in economics studies. “One is ethical,” says Sullivan, “I think there are ethical and moral reasons that we don’t want to deceive our subjects.”
“Aside from the ethical point,” Sullivan continued, “if people believe that we aren’t telling the truth in our experiments then they’re going to behave differently. If they think we’re gaming them, they’re going to game us right back and we won’t end up learning anything.”
Another issue with correspondence audit studies is that researchers might confuse an employer’s interest in a candidate with the employer’s expectation that the candidate would accept a job offer. Incentivized resume rating solves this problem by distinguishing between how much an organization wants to hire a candidate versus how likely an organization thinks a candidate will want to work there in the first place.
Sullivan and his fellow researchers demonstrated this by using IRR in partnership with the University of Pennsylvania Career Services office. They evaluated employers’ preferences for hiring hypothetical candidates by asking them how interested they would be in hiring the candidate and how likely they think it is that a candidate would accept a job offer if given one.
The development and implementation of incentivized resume rating is a first-of-its-kind methodology that answers many of the same questions that correspondence audit studies do without the need for deception and with clearer insights about employer preferences. Sullivan’s study serves as a proof of concept for IRR, as it found evidence of discrimination among employers recruiting STEM candidates, lower returns to prestigious internships for women and minorities, and that employers report that white female candidates are less likely to accept job offers than their white male counterparts.
“I think this method could be really useful in a lot of areas,” says Sullivan, citing the fact that IRR has been or could be used in other policy-relevant applications. Alex Chan, another economics researcher, used the methodology to understand how patients choose their doctors and detect discrimination against doctors of different ethnicities than their patients. Sullivan also contends that IRR could be used to detect discrimination in the housing market, which is further testament to the versatility of his new method.
