Before I begin my comments on WMDs, I would like to share with you that on my way home from class last week, I passed a sign outside of a Bank of America advertising their mobile “assistant”. Her name was Erica.
O’Neil, in her conclusion, says that “big data processes codify the past. They do not invent the future.” This neatly sums up the arguments she’s made. The examples are clear. In schools, the “codifying of the past” is done wildly inaccurately. The performance indicators of teachers simply do not measure what they are meant to. This is the first type of problem introduced by WMDs. The response to the inaccuracies are not surprising. In the name of ease or of streamlining, or most likely in the name of cost minimization, teachers are held to standards that bring inherent contradiction. How can a school measure the value added by a teacher of underperforms with the same algorithm that it measures teachers of overachievers? The outcomes are not important here, only the seemingly priceless impact of essentially digitizing employee review. While pretending that taking humans out of the judgement process will level the playing field, it actually codifies the human error.
The example of teacher evaluation is the least threatening of the examples given by O’Neil in the assigned reading. Worse is the outright and blatant codification of existing systems and structures. Where value-added educator evaluation is an original model of measurement with new flaws, in the case of the use of WMDs in the financial industry is the codification of unoriginal, existing models that have unfairness baked deep within already. By using existing data, choices are made about the value of individuals without consideration of the data that has not already been collected – like using the zip code as a weapon despite an unmeasured propensity toward frugality, for example. This, arguably more dangerous, form of WMD highlights O’Neil’s point about “codifying the past”.
Reading these chapters, I thought about our prior conversations about digitization and datafication. The data had already been collected; vast swaths of information exists about individual insurance risk, policing patterns, or political motivations. The use of WMDs seems to me a type of digitization of our existing social structures and patterns. This begs a new perspective. Why are we looking at the success of data systems to fix the world when we cannot even create data systems that properly express the world as it is? O’Neil’s answer is this: the mathematical tools discussed can be used for good or for evil, for equity or inequality, to codify or to “create” our society. It is the human component that decides how to use these tools. Unfortunately, it appears that the same players involved in codifying, datafying, and digitizing our reality have very little interest in the human component at all – likely underestimating or even devaluing their roles.
