To understand how businesses can use Big Data to garner businesses insights, we must first define what it is. Gartner, an IT analyst firm, refers to Big Data as the volume, variety and velocity of structured and unstructured data pouring through networks into processors and storage devices, along with the conversion of such data into business advice for enterprises.
Although a nice definition, it does not address what Big Data really means as far as its business impact. Ken Rosen, Managing Partner at Performance Works thinks that Garnter’s definition misses the real point, suggesting that it is like saying ‘New ideas come from electricity moving among brain cells.’ It’s correct, but the emphasis is wrong. It makes sense that an IT‐oriented firm like Gartner would focus on speeds, feeds, and infrastructure, but executives need a different view.
To be clear, Big Data is not simply dealing with lots of data. For instance, a thousand movies take up a Petabyte of storage but a thousand movies is not a Big Data problem. On the other hand, a business may have a serious Big Data initiative, but the total amount of data fits on a single hard drive.
So what is Big Data? Big Data is where businesses derive new meaning from new data sources. New meaning that was never practical to find before. This could be because of scale, data format, the distribution of data in many locations or the fact that no one thought of looking before. It is easily as much a new mindset as new technology.
Why should businesses care? Businesses must care about Big Data as they can learn what to offer and to whom; when to offer something new and through what channels; which employee can best solve a problem and when to get outside help; which competitor will win and when their stock price will reflect the victory. Big Data will be the one of the most important things for business since the Internet due to the business insights that it can provide.
Big Data is all about delivering new insights to decision makers. As reported in a Forbes article, Walmart wanted to find out what the biggest selling items were that people bought before a hurricane hit. The No. 1 answer—batteries— was not a surprise, but the unexpected No. 2 item was Kellogg’s Pop‐Tarts, as they last a long time, don’t require refrigeration or preparation, and are easy to carry and store. As a result of this intelligence, Walmart can now stock up on Pop‐Tarts in its Gulf Coast stores ahead of storm season. This is where the reach of new‐generation business analytics tools shine by directly helping enterprises make smart decisions.
Historically, data analytics software hasn’t had the capability to take a large data set and use it to compile a complete analysis for a query. Instead, it has relied on representative samplings, or subsets of the information to render results. That approach is changing with the emergence of new Big Data analytics engines, such as the opensource Apache Hadoop. Hadoop processes large caches of data by breaking them into smaller, more accessible batches and distributing them to multiple servers to analyse, much like cutting your food into smaller pieces for easier consumption. Hadoop then processes queries and delivers the requested results in far less time than old‐school analytics software—most often minutes instead of hours or days.
Hadoop and other such systems provide complete looks at big data sets, instead of a team of analysts spending days or weeks preparing the parameters for data subsets, and then taking 1, 2 or 10 percent samplings, all the data can be analysed at one time, in real time. Why bother? Because data sitting in storage arrays and cloud accounts represents unrefined value in its most basic form. If interpreted properly, the stories, guidelines and essential information buried in storage and databases can open the eyes of business executives as they make strategic decisions for their company.
Never mind 20th century focus groups and marketing research surveys. Let the Big Data / Big Analytics games begin, for they are not about computing and databases, they are about a new generation of analytics driving business insights for business innovation.
Big Data analytics provides businesses with customer information that helps derive meaning from customer actions; the difference between what they say they are going to do and what they actually do. Real predictable consumer behavior comes from understanding the attitudes that drive that behavior. Understanding attitude is a second and third order effect, derived from the combined context of multiple Big Data type analysis, but more importantly from true engagement, interaction and understanding of the customer.