Why Big Data is Failing?

Well as soon as I say this, I am sure several persons will be ready to contradict me. While I am sure Big Data as a Technology has earned its place in history and with Cloud and IoE (Internet of Everything - Living, Non Living and Digital) and with the ever increasing awareness of the same, it is poised to become an essential component of software solutions.

But while Big Data Technologies make foray into mainstream, it is not clear yet what could be the real logically and fundamentally different about Big Data other than well, it is a large database. Of the numerous IT people I have interacted over the past few years, nearly everyone considers Big Data as a replacement of their existing storage as operational data stores, considering the scale and cost when compared to traditional databases. Hardly anyone I saw is too much excited about the computing part - the ability to crunch large amount of data in parallel.

Why is this?

While no one denies the requirement of cheap reliable storage systems for Big Data, we have not whole heartedly moved into the realm of 'What to do with this data'. Big Data is often marketed and found practical use in storing myriad kind of data, such as in files and then build engines which could interpret those files. In many cases we don't even care or come to stage of caring about those engines because we don't know what data is there in the first place.

The one problem what Big Data has solved effectively is that now you are able to capture and store 'All' The Data and if you are smart enough, you can explore that and find out what can be done with it. And once you have found what data you need, you really don't need Big Data because you already got what you want. The Small Data. And we are already very experienced and familiar on dealing with Small Data. see my earlier post.

So, as it remains in the case with Big Data now we are in a stage of exploring the data in a way that was never possible before. We are paying more importance to tools which can extract, explore, dig, analyze data - in multiple forms - and help us to get some insights which will help to create better services or business cases. 

Classic use cases have already been designed and deployed in case of extremely personalized advertisements, fraud prevention, opportunity detection and so on. But the personal profiles created by correlating multiple data sources and behavior of a person across multiple channels, times and contexts, the fraud flags generated by similar methods or by applying sophisticated machine learning methods, finally are small sized fragments which contribute to decision making.

As long as we are not aware of what is the small data we are looking to compute and 'HOW', Big Data will remain elusive. Its value will only be perceived as a storage system to dump useless data with a hope that some day some one will enter the ocean and explore new islands.