The year was 2006, George W was in the white house and Google was the king of search. If you wanted a restaurant review you probably either looked at a copy of Zagat (the book, not online!) or you did a Google search. Fast forward to 2010 and what’s changed? Beyond the obvious change inside 1600 Pennsylvania Avenue, now if you want to research a restaurant you do a Yelp search – what a difference 4 years makes! What else has changed? The biggest change is the growth of data – mobile interactions, Google searches, Twitter tweets, Yelp reviews, Facebook friends and pokes and pictures …. it’s a data deluge out there!
I”m increasingly being convinced that the next big tech opportunity lies in being able to do something with the data deluge. A recent column in Gigaom mentions that Facebook’s greatest asset is it’s social graph (i.e., the connections between people and their friends, pokes, pictures etc.). Facebook is working very hard at extracting value from this data. Similarly, Yelp is trying to mine its own user-generated restaurant reviews.
The key to solving this data-mining challenge is to engage in “non-linear thinking.” It’s important to keep in mind that there is more to data mining than the IT-focused steps such as data collection, aggregration, modeling, validation etc. One of the most difficult parts of data mining social media information is that the data is mostly unstructured (i.e., text, pictures etc.).
I’m really excited about the data challenges – this is the age of big data – for more on big data read this .
What do you think? Do you see an opportunity in the data deluge?