Big Data Analytics

I am sure by now most of you have heard about Big Data and the advent of a host of newly formed companies who provide services relating to analyzing Big Data. They refer to this new “Industry” as Big Data Analytics. There are now companies claiming they can predict just about anything after analyzing all items in the enormous universe of data which is held deep in the internet. This explosion is witnessed in every major paper, including NY Times, WS Journal, and Financial Times where a day doesn’t go by where a prognosticator or CEO doesn’t pontificate the merits of predicting outcomes based on these analytic tools exposing predictable outcomes living in Big Data.

So what is this all about? Is this just hocus pocus? Is it just another internet based fad? My belief based on my diligence is it’s very real and will be the next big “Thing”. Fundamentally, here is how it works, a very sophisticated real time search engine (think Google on Steroids) goes out into the internet including the most obscure places in the deep recesses, not just the easy to find place (including twitter, facebook, etc) and within seconds pulls all the data on a particular subject. Let’s say a Company like Apple, all this Data is then cataloged into a separate database. This database is filtered through various algorithms, the most sophisticated of which will tag each word with a numerical value from one to ten. These values are to denote the importance of this word in predicting an outcome. Some of the greatest Linguistic minds in the world have studied the importance of what words people use to determine how people feel. Finally, these words which are weighted with a numerical value are put through a mathematical based algorithm which will predict an outcome based on SENTIMENT. This all takes place within milliseconds. Yes, milliseconds. What you get at the end is the computer can tell you if Apple will go up or down in the next minute, five minutes or whatever period of time you chose. Of course, the closer the timeframe, the better the correlation. Think about what this would mean for portfolio management! Let’s say the Gap try’s out a new advertising campaign and wants to know how its customers or potential customers or anybody thinks about this new ad in real time  not two weeks after gallop or some other dinosaur takes a poll. How about this coming election and the effect of messaging from the parties, what is working or resonating and what is falling flat, in real time!

You get the picture, Big Data Analytics is for real and will impact your life in ways you  can’t understand today. I recently talked to a Company here in Mass, Relevant Data Corp ( This Company is creating products for day traders, fantasy sports and consulting services. It is an amazing Company; it is one of the few company’s in this space who has a patent on its real time search engine. Take a look at their advisory board. You should check it out.

Bottom line is with analytic tools like this the information coming our way will be micro tailored to our predictable inclinations. Like it or not it is here.