Big Data: The Premise and the Promise


Spring is in the air (it was, a few months ago) and so are allergies. You can’t change the channel without running into an ad for the latest allergy medicine strong enough to bring down a woolly mammoth.

I get delayed at JFK. Soon after I post on Facebook about the lack of decent dining options in Terminal 7 (as opposed to, say, Terminal 5), I get an email from Priceline telling me I can get a 4 Star hotel for $60 a night close to the airport. 

What do these examples have in common?

In each instance, based on the weather, your health or my status, someone on the “other side” predicted, guessed or hoped that a certain type of consumer activity would follow – typically involving some kind of monetary transaction. It then follows that if a company spends time and money predicting or guessing what a consumer or user might do next, then it will generally want to make sure that its time and money were wisely spent. 

And in simple but VERY broad terms, Big Data’s premise is exactly that (yes, purists will argue that this is data warehousing or BI and yes, you’re correct – more about that in a second). Big Data is all about collecting, correlating and analyzing large amounts of historical and “real-time” data in order to gain valuable insights into the behavior and performance of both sentient and non-sentient entities. 

What do I mean by that?

Let us go back to our first two examples and see how Big Data changes the game.

a) Delays and Priceline

On a given day, many passengers might get delayed at JFK or any other airport with clusters of hotels close by. Some of those delayed may be delayed for more than a few hours and might actually need to get a room for the night. And of those, some might actually let their friends on Facebook now about their misfortunes. 

By understanding and analyzing in minutes, if not sooner, 

  • Who is delayed at which airport 
  • The number of hotels around that airport with specials and deals
  • The probability of someone stuck in the airport needing a hotel for the night (who also posted something relevant to the delay on Facebook)
  • Historical data that correlated airport delays to increases in room bookings in nearly hotels for the last ten years
Priceline just vastly increased customer satisfaction (meeting an unmet need at just the right time) while increasing revenues. 

b) Woolly Mammoths and Allergies

If allergy medicine makers were to indiscriminately bombard the airwaves with their ads 24 hours a day, they could at best waste millions of dollars and at worst, irritate and turn off a bunch of existing and potential customers. So, in order to determine what to advertise where and how frequently, they could

  • Look at age, income and other demographic data in different parts of the country
  • Understand the relationship between demographic data, different channels and spending patterns
  • Determine which zip codes need or use sinus medication (or other medication that might ALSO be used by allergy sufferers) the most
  • Retrieve Twitter data where their Allergy meds are mentioned positively (or negatively)

And once again, they could put all of these data sets together to analyze which medicine to advertise where and when in order to maximize sales – not just every month, or every quarter, but on a weekly or even a daily basis. My Priceline example takes it one step further. Priceline could potentially run these types of analyses every hour fine-tuning and tailoring its offers based on a myriad of changing, dynamic conditions. 

So is Big Data all about marketing?


Students of marketing will no doubt realize that so far we have talked about Big Data helping with the 4 Ps: product, price, place and promotion, with much better segmentation and targeting. Marketing organizations and Chief Marketing Officers (CMOs) are realizing the power of Big Data and are aggressively either creating or executing (sometimes by the seat of their pants) their Big Data strategies to increase market share, enhance pricing power and optimize their costs. But that’s only part of the story.

Its only part of the story because, while Big Data has numerous applications or “use cases” that can help increase sales and revenues and reduce or optimize costs, depending on the industry in question, Big Data can also help in areas such as public policy, education, healthcare (ex: if you take medicine X for a condition, based on my analysis of millions of other people’s anonymized health records and the meds they take, I could perhaps predict your likelihood of needing an expensive pill in 5 years and then pre-emptively start you on a dramatically cheaper wellness regime?), defense, homeland security, etc. In fact, numerous Big Data applications and products exist today that are successfully helping organizations and companies in these areas be more effective and efficient. 

Summing it all up

If you need better results and outcomes and “all you have” is a large and growing volume of data that’s generated every hour, day, week or month from various disparate sources (internal, external, historical, real-time, macroeconomic, social media), Big Data can help. That’s all there is to the promise of Big Data. 

In future stories, to be published in no specific order, we will examine specific use cases of Big Data and how value can be created there and captured.

Note on Big Data, Data Warehousing and Business Intelligence

Critics of Big Data argue that this is just Date Warehousing and Business Intelligence in new clothing. They also argue that Big Data is overhyped and that in order to truly create value by using Big Data, strong business insights need to be applied. Finally, they argue that industries where Big Data is supposed to create value have always been crunching ginormous data sets. 

There is some truth to all of these arguments.

The promise of DW and BI was also about driving insights based on crunching massive data sets. It is also true that without applied business insights (provided by knowledgable, experienced, “smart” humans), Big Data might only provide weird, nonsensical correlations (ex: every time the pilot’s socks are blue, there’s a 80% chance that the flight will be delayed) derived without any insight into causation. But Big Data’s promise is that we now have the ability to analyze in real time the tera or petabytes of data being generated on a scale (from everything we had before and social media and networked appliances and smart meters and …) and at a pace never seen before – using very low cost processing power and nearly infinite storage capacity, in real-time, to generate actionable insights and enable (near) instant decision making. This was not possible with DW/BI in the past. Also, tantalizingly, while DW and BI were used to test hypotheses to a very large extent, Big Data can (and is) create and uncover causal patterns which are humanly very hard, if not outright impossible, to even hypothesize. But no question that unless “smart” humans armed with knowledge and experience validate these patterns and apply them in a context-sensitive manner, true value will be hard to create. 

Providers of Big Data solutions should be wary…especially those that are in this for the long-run. While we talk about the premise and promise of Big Data in this article in generic terms, over time, intelligent, “applied” Big Data that brings with it strong industry knowledge and awareness will triumph via true value creation and capture for its users, when the dust settles, over “one size fits all” products and solutions. In the interim, in this “foundation” stage (or hype stage, if you prefer), software, appliance and services companies will prosper, spurred by strong and widespread adoption across various industries. 

So what is the take-away if you are a CMO, CIO and even a CEO? Needless to say, you need to be wary of any product, technology or thingummy that promises to usher in world peace and destroy poverty. Pilots, POTs, validated success stories and use cases applicable to your industry, vertical and/or niche, pay for performance models, etc. must be leveraged in order to ensure ROI on any Big Data investments. And like with any technology, having the right people and processes in combination with the right technology will make all the difference.


Technorati Tags: 

Social Media Icons Powered by Acurax Web Design Company
Check Our FeedVisit Us On TwitterVisit Us On FacebookVisit Us On Google PlusVisit Us On Linkedin