The output, however, is much more vexed. On the face of it (no pun intended), it seems like a simple binary (yes/no) classification problem: Is this person a criminal? But criminality is not an inherent property of a person. The category of “criminal” is instead produced by an interaction of at least three things: the person’s behavior (very much constrained by their circumstances), social norms around what is considered criminal behavior, and the legal regime the person finds themselves in. So there cannot possibly be sufficient information in the input (an image of a face) to determine the output. You can’t tell if someone is a “criminal” by looking at them. … Moreover, the claim that the system’s output lacks racial bias is preposterous, given everything we know about racial discrimination in policing and how poorly facial analysis systems perform with racial minorities.

Emily M. Bender, Alex Hanna, The AI Con- How to Fight Bi…, loc. 436