My grandmother—no doubt based in part on her experiences—saw racism everywhere, in every inequity, every statistic. Racial differences in wages? Racism. Racial differences in educational achievement? Racism. Racial differences in teen birth rates? Racism. This sort of casual empiricism—which has crept back into mainstream media and other institutions—was a competitive sport among my family and friends. Did you see the way that white woman tightened her grip on her purse, because I was behind her? Does this guy follow everyone around the store?

A decade after the McDonald’s incident, in graduate school, I read a 1995 paper titled “The Role of Premarket Factors in Black-White Wage Differences.” Using a nationally representative sample of more than 12,000 14- to 17-year-olds from 1979, Derek A. Neal and William R. Johnson estimated that blacks earned between 35% to 45% less than whites on average.

They examined how much of the wage gap could be attributed to present-day discrimination in the workplace versus differences in skill as measured by the Armed Forces Qualification Test, a cousin of the ACT and SAT. Importantly, they weren’t trying to measure the effect of America’s racist history running back to the early 17th century, when the first African slaves were brought to work the tobacco farms of Virginia, only the extent to which blacks earn lower incomes because employers today make racially discriminatory decisions. Nor did they focus on prejudice more broadly: They ignored that my grandmother was spat on in the parking lot at work and concentrated on whether she was treated fairly once she entered the building.

“We find,” they wrote in the abstract of their paper, “that this one test score explains all of the black-white wage gap for young women and much of the gap for young men.” With their approach, antiblack bias played no role in the divergent wages among women; a black woman with the same qualifications as a white woman made slightly more money. And it accounted for at most 29% of the racial difference among men, with 71% traceable to disparate performance on the AFQT. The AFQT itself was evaluated by the Pentagon, which found that black and white military recruits with similar AFQT scores performed similarly on the job—indicating no racial bias.

The paper felt like an attack on what I knew. An assault on all those conversations with my grandmother, which taught me that racism—present-tense racism—dictated black-white inequality. I told myself that Messrs. Neal and Johnson, both of whom were white, were probably bigots, and I set out on a mission to disprove their work.

I vented about my battle with Messrs. Neal and Johnson to a fellow graduate student at Penn State, a white guy from the cornfields of Southern Illinois. He was no more at home at a top-25 economics doctoral program than I was, and we spent a fair amount of time together during our first year, staring at Euler equations in our favorite textbook, “Recursive Methods in Economic Dynamics.”

I told him I was sure discrimination was a bigger factor than Messrs. Neal and Johnson were letting on, but “I just can’t get this data to cooperate.” He asked why I was so convinced, and I erupted in a rant about the prevalence of racism and recognizing bigots on sight. My grandmother would’ve nodded rhythmically along. My friend responded with a burst of loud, sharp laughter in my face.

He pointed out how far I was straying from our Euler equations. How on any subject other than race, I would have never given in to such sloppy thinking. The double identity—a classically trained economist taught to tease out causal relationships and a black Southern boy taught that discrimination is ubiquitous—had lived seamlessly inside me until that moment. Messrs. Neal and Johnson, as it turns out, aren’t bigots, and their conclusions have stood the test of time and my attempts to disprove them. I extended their analysis to unemployment, teen pregnancy, incarceration and other outcomes—all of which follow the same pattern. Moreover, the relationship between skills and wages has been confirmed by study after study, even when using different data and methods. Kevin Lang, an economist at Boston University, corrects important issues in Messrs. Neal and Johnson’s work but finds the same relationship between AFQT scores and wages. Taken together, an honest review of the evidence suggests that current racial inequities are more a result of differences in skill than differences in treatment of those with the same skill.

I write this with some degree of trepidation, in part because I still have my grandmother in my ear and in part because I am keenly aware of the harm in underestimating bias. But there is also a cost to overemphasizing its impact. A black kid who believes he will face daunting societal obstacles is likely to underinvest in trying to climb society’s rungs. Every black student in the country needs to know that his return on investment in education is, if anything, higher than for white students.

The solution is neither to stop fighting biased behavior nor to curb honest inquiry about race in America. We shouldn’t stop searching for and penalizing discriminatory employers, or trying to reduce racial differences in police brutality, or estimating whether the value of a home appraisal depends on the race of the homeowner, or reducing bias in bail decisions by using artificial intelligence. I could go on, like the conversations stuck to those slipcovers. The solution isn’t to look away from discrimination. It does exist. But we also can’t point at every gap in outcomes and instantly conclude it’s racism. Prejudice must be measured rigorously. Statistically. Disparity doesn’t necessarily imply racism. It may feel omnipresent, but it isn’t all-powerful. Skills matter most.

Mr. Fryer is a professor of economics at Harvard, a fellow at the Manhattan Institute, and founder of Equal Opportunity Ventures.