My last post here, the descriptively-titled “Big Data’s Biggest Challenge? Convincing People NOT to Trust Their Judgment,” generated a fair amount of commentary. So I think it’s worthwhile to devote a couple follow-on posts to the reactions, questions, and objections raised in response to my contention, which was (and is) that we should generally be relying a lot less on the judgments, diagnoses, and forecasts of human ‘experts,’ and a lot more on the outputs of cold, hard, data-driven algorithms.
A good place to start is with the simple question of where this contention comes from — why am I so convinced that we should be relying less on experts and more on algorithms? The simple answer is that both the theory and the data support this conviction.