Big Data, Part 2
Last week we looked at what Big Data (BD) is and some of its potential uses while this week we will explore some of the challenges facing its use and some of the potential fears of BD. Gartner, Inc., widely recognized as the worldwide leader in IT research and advisory services, started speaking to BD in 2001 and formulated the now famous “3 V” model of the challenges of handling BD: volume, velocity and variety. Below we will demonstrate that the volume of data currently available is nearly incomprehensible, the velocity at which it is increasing is almost frightening and the variety of its sources is increasing dramatically.
Just how much data exists?
There is currently enough data available to equal 200 BILLION high definition movies (which would take over 47 MILLION years to watch!). EBay says that the current volume of business intelligence data is DOUBLING every 1.2 years, and that interval is quickly contracting, as there are 450 billion internet business transactions every day. Wal-Mart conducts over 1 million transactions per HOUR and YouTube adds 48 hours of new video per MINUTE (much of which has participants tagged, thus identifiable). BD has become a runaway train and there is no slow-down in sight. With corporate ERP systems and E-Commerce firmly taking root, the volume and velocity of data available for analysis has exploded. With the number of new apps and websites currently online, the variety of data sources continues to grow at an unprecedented pace.
Upside of Big Data
One obvious upside to this is the magnitude of job creation in the field of BD analysis. McKinsey & Company estimates 1.5 million workers will be needed by 2018 with knowledge of data analysis, and more importantly, how to use the interpretations to formulate business decisions. Harvard Business Review warns that BD must be used in conjunction with “Big Judgment” as only about 1/3 of current data analysts are actually capable of correctly interpreting BD. They also predict that these jobs will be the “hottest field of the 21st century” and equate it to the dotcom explosion of the 1990’s.
Big Data Concerns
It is easy to conclude that one of the greatest fears of BD is the Big Brother feeling it invokes. Just as video cameras are seemingly everywhere, data collection filters abound at every turn. It is virtually impossible to conduct any business transactions or to be able to log in to the cyber-world without leaving a littered trail of unintended fingerprints along the way. Someone may have a legal matter resolved and have the record sealed, but BD may provide insight into the particulars of the case through the analysis of online search history or the lawyer hired. Youthful indiscretions may come back to haunt many years later, job offers, club memberships or credit decisions may be made or withheld due to predictors of the likelihood of someone’s future; the potential list is endless.
Another significant and well-warranted worry deals with security breeches. As companies collect and compile eerily accurate (and comprehensive) profiles of people, a single breech of their systems can give cyber criminals much more info about someone than they could ever have obtained previously. Equally concerning is that there is no way to know exactly what is “known” about an individual (read: YOU), so there is no way of correcting inaccurate information. Finally, of concern to companies is that as more and more is learned about BD, it becomes much easier for con artists and criminals to game the system and intentionally provide bogus data for the sole purpose of inflating or orchestrating their profile.
In closing, consider the possibilities of exactly what will be known about you as BD captures all of the data produced as the world moves into the Internet of Things age. There will be an undeniable convenience factor as your preferred vendors offer you reminders and specials for all of your favorite products and services, BEFORE YOU EVEN KNOW that you need them. That said, the concept reminds me of the question posed in Philosophy 101 classes: does man really have freedom of choice if a higher being already knows what he will chose. If Wal-Mart is telling me every single thing I need right this very minute and THEY ARE RIGHT, am I really free to move through life or just an uber-predictable creature of habit?
