When a new, powerful tool comes along, there’s a tendency to think it can solve more problems than it actually can. Computers have not made offices paperless, for example, and Predator drones haven’t made a significant dent in the annual number of terrorist acts.
To judge by the number of requests for proposals coming out of agencies and departments, the current power tool of choice is one that can apply analytic tools to massive amounts of data to find underlying, meaningful patterns. Government agencies, for example, are using big data tools to detect crime patterns and Medicaid fraud. The Department of Homeland Security is using big data tools to scan social media for signs of terrorists, and private companies are using similar tools to detect insider threats.
While big data tools are, indeed, very powerful, the results they deliver tend to be only as good as the strategy behind their deployment. A closer look at successful big data projects offers clues as to why they are successful … and why others fall short of the mark.
One of the most effective big data projects I’ve covered in recent years is the USDA Risk Management Agency’s program to set crop insurance rates and to detect fraudulent claims of crop losses. The agency starts with FCI-33 rate maps created in ESRI ArcView that combine data from a variety of sources, including soil data from the Natural Resources Conservation Service, floodplain data from the Federal Emergency Management Agency, farm location and crop data from the Farm Services Agency, historic weather data and satellite imagery.