Resources

Big Data vs Intuition: Finding a middle path to success

18th Jun `14, 03:05 PM in Resources

In this era of Big Data, everyone is becoming adherents of the data-driven life and data underlies everything…

Baiju-NT
Baiju NT Contributor
Follow

In this era of Big Data, everyone is becoming adherents of the data-driven life and data underlies everything we do…. Our business decisions, arguments, choices… But there was a time when entrepreneurs even at larger companies had no data to back their decisions; they had no computers, devices or applications that we have today to access/store/transfer information in decision making. Still, they made smart decisions through observation, common sense, experience and most importantly intuition.

Let’s take Steve Jobs as an example. Jobs was smart and genius. But he didn’t study data or crunch numbers, but he could sniff the winds and sense what lay ahead. His imaginative leaps were instinctive, unexpected, and at times magical. They were sparked by intuition, not analytic rigor.  Well known as the “champion of intuition”, Jobs knew intuition is a powerful tool and it had a big impact on his work.

Trained in Zen Buddhism, Jobs began to appreciate the power of intuition, in contrast to what he called “Western rational thought,” when he wandered around India after dropping out of college.

He said: “The people in the Indian countryside don’t use their intellect like we do, they use their intuition instead, and their intuition is far more developed than in the rest of the world… Western rational thought is not an innate human characteristic; it is learned and is the great achievement of Western civilization. In the villages of India, they never learned it. They learned something else, which is in some ways just as valuable but in other ways is not. That’s the power of intuition and experiential wisdom. Coming back after seven weeks in Indian villages, I saw the craziness of the Western world as well as its capacity for rational thought. If you just sit and observe, you will see how restless your mind is. If you try to calm it, it only makes it worse, but over time it does calm, and when it does, there’s room to hear more subtle things–that’s when your intuition starts to blossom and you start to see things more clearly and be in the present more. Your mind just slows down, and you see a tremendous expanse in the moment. You see so much more than you could see before. It’s a discipline; you have to practice it.”

Jobs’s intuition was based not on conventional learning but on experiential wisdom. He also had a lot of imagination and knew how to apply it. Intuition and personal judgments are necessary to set priorities for what to look for, what matters to the customer, what the competitive landscape looks like, how behaviours are changing over time and ultimately what to do with the data. Tools can help the analyst’s brain, but shall not replace it.

But, we have a problem. We know that intuition is important, but what if intuition goes wrong? Will flipping a coin always work? Experts say that under certain circumstances, intuition can lead us to incorrect decisions and judgments.

For example, every person who plays basketball and nearly every person who watches it believes that players go through hot streaks, when they are in the groove, and cold streaks, when they are just not feeling it. But researchers have found that a player who has made six consecutive foul shots has the same chance of making his seventh as if he had missed the previous six foul shots. When a player has hit six shots in a row, we imagine that he has tapped into some elevated performance groove. In fact, it’s just random statistical noise, like having a coin flip come up tails repeatedly. Each individual shot’s success rate will still devolve back to the player’s career shooting. Besides, there is a data processing problem too. There is no way human brain can even dream to process massive data of large companies without machines and algorithms that tackle personalised problems. Too much information is way too complex for an objective analysis in very short time. So, it’s a practical problem.

So, what’s our take on Big Data vs Intuition? Are they mutually complimenting? Yes, they are. Big data and intuition should not be seen in silos. They are to be in a perfect harmony — like mind-body relationship. Analysis of data helps to bring down the risk involved in decision making, as an intuition can go horribly wrong and it happens most of the time. In Steve Jobs’ case, it is extremely tempting to call/project him as “a champion of intuition”, but I believe that his ideas must have been based on/influenced by some sort of personal analysis (not necessarily a big data analysis), based on technology trends, recommendations of his associates and understanding the market and its demands. So, can we say there is something called “Pure Intuition”? Nope. In a nutshell, there is no “Big Data vs Intuition” scenario, rather it’s Big Data plus Intuition. Big Data actually supports decisions based on intuition. If you want your organization to take better decisions, your decisions will need to be based on both explicit information (data) and implicit information (intuition).

Now, what’s next? Let’s look into some possible questions in “Big Data vs Intuition” situation! We’ve seen they are complimentary, but how do we combine big data and intuition in a situation, especially when they give contradictory output? Let me put the question in this way: How do you respond to the instances where the data view is a 100% contradiction of your gut feeling and passion? You would love to chase your passion/ intuition because that’s what had been deeply-rooted into your DNA over the years, but you are not backed up by data. What will you do? Will you take risks and tend to write your own destiny? Second, how do you manage conflicting intuitions in a company/situation, when there is no data to prove which one is correct/wrong? Leaving these questions open for all to reflect on… Thanks for reading.

MORE FROM BIG DATA MADE SIMPLE