Big Data: Understanding How Data Powers Big Business

Introduction

Big data is today's technology hot topic. Such technology hot topics come around every four to five years and become the must have technologies that will lead organizations to the promised land—the silver bullet that solves all of our technology deficiencies and woes. Organizations fight through the confusion and hyperbole that radiate from vendors and analysts alike to grasp what the technology can and cannot do. In some cases, they successfully integrate the technology into the organization's technology landscape—technologies such as relational databases, Enterprise Resource Planning (ERP), client-server architectures, Customer Relationship Management (CRM), data warehousing, e-commerce, Business Intelligence (BI), and open source software.

However, big data feels different, maybe because at its heart big data is not about technology as much as it's about business transformation—transforming the organization from a retrospective, batch, data constrained, monitor the business environment into a predictive, real-time, data hungry, optimize the business environment. Big data isn't about business parity or deploying the same technologies in order to be like everyone else. Instead, big data is about leveraging the unique and actionable insights gleaned about your customers, products, and operations to rewire your value creation processes, optimize your key business initiatives, and uncover new monetization opportunities. Big data is about making money, and that's what this book addresses—how to leverage those unique and actionable insights about your customers, products, and operations to make money.

This book approaches the big data business opportunities from a pragmatic, hands-on perspective. There aren't a lot of theories here, but instead lots of practical advice, techniques, methodologies, downloadable worksheets, and many examples I've gained over the years from working with some of the world's leading organizations. As you work your way through this book, you will do and learn the following:

Educate your organization on a common definition of big data and leverage the Big Data Business Model Maturity Index to communicate to your organization the specific business areas where big data can deliver meaningful business value (Chapter 1).

Review a history lesson about a previous big data event and determine what parts of it you can apply to your current and future big data opportunities (Chapter 2).

Learn a process for leveraging your existing business processes to identify the right metrics against which to focus your big data initiative in order to drive business success (Chapter 3).

Examine some recommendations and learnings for creating a highly efficient and effective organizational structure to support your big data initiative, including the integration of new roles—like the data science and user experience teams, and new Chief Data Office and Chief Analytics Officer roles—into your existing data and analysis organizations (Chapter 4).

Review some common human decision making traps and deficiencies, contemplate the ramifications of the death of why, and understand how to deliver actionable insights that counter these human decision-making flaws (Chapter 5).

Learn a methodology for breaking down, or functionally decomposing, your organization's business strategy and key business initiatives into its key business value drivers, critical success factors, and the supporting data, analysis, and technology requirements (Chapter 6).

Dive deeply into the big data Masters of Business Administration (MBA) by applying the big data business value drivers—underleveraged transactional data, new unstructured data sources, real-time data access, and predictive analytics—against value creation models such as Michael Porter's Five Forces Analysis and Value Chain Analysis to envision where and how big data can optimize your organization's key business processes and uncover new monetization opportunities (Chapter 7).

Understand how the customer and product insights gleaned from new sources of customer behavioral and product usage data, coupled with advanced analytics, can power a more compelling, relevant, and profitable customer experience (Chapter 8).

Learn an envisioning methodology—the Vision Workshop—that drives collaboration between business and IT stakeholders to envision what's possible with big data, uncover examples of how big data can impact key business processes, and ensure agreement on the big data desired end-state and critical success factors (Chapter 9).

Learn a process for pulling together all of the techniques, methodologies, tools, and worksheets around a process for identifying, architecting, and delivering big data-enabled business solutions and applications (Chapter 10).

Review key big data technologies (Hadoop, MapReduce, Hive, etc.) and analytic developments (R, Mahout, MADlib, etc.) that are enabling new data management and advanced analytics approaches, and explore the impact these technologies could have on your existing data warehouse and business intelligence environments (Chapter 11).

Summarize the big data best practices, approaches, and value creation techniques into the Big Data Storymap—a single image that encapsulates the key points and approaches for delivering on the promise of big data to optimize your value creation processes and uncover new monetization opportunities (Chapter 12).

Conclude by reviewing a series of calls to action that will guide you and your organization on your big data journey—from education and awareness, to the identification of where and how to start your big data journey, and through the development and deployment of big data-enabled business solutions and applications (Chapter 13).

We will also provide materials for download on www.wiley.com/go/bigdataforbusiness, including the different envisioning worksheets, the Big Data Storymap, and a training presentation that corresponds with the materials discussed in this book.

The beauty of being in the data and analytics business is that we are only a new technology innovation away from our next big data experience. First, there was point-of-sale, call detail, and credit card data that provided an earlier big data opportunity for consumer packaged goods, retail, financial services, and telecommunications companies. Then web click data powered the online commerce and digital media industries. Now social media, mobile apps, and sensor-based data are fueling today's current big data craze in all industries—both business-to-consumer and business-to-business. And there's always more to come! Data from newer technologies, such as wearable computing, facial recognition, DNA mapping, and virtual reality, will unleash yet another round of big data-driven value creation opportunities.

The organizations that not only survive, but also thrive, during these data upheavals are those that embrace data and analytics as a core organizational capability. These organizations develop an insatiable appetite for data, treating it as an asset to be hoarded, not a business cost to be avoided. Such organizations manage analytics as intellectual property to be captured, nurtured, and sometimes even legally protected.

This book is for just such organizations. It provides a guide containing techniques, tools, and methodologies for feeding that insatiable appetite for data, to build comprehensive data management and analytics capabilities, and to make the necessary organizational adjustments and investments to leverage insights about your customers, products, and operations to optimize key business processes and uncover new monetization opportunities.

Chapter 1

The Big Data Business Opportunity

Every now and then, new sources of data emerge that hold the potential to transform how organizations drive, or derive, business value. In the 1980s, we saw point-of-sale (POS) scanner data change the balance of power between consumer package goods (CPG) manufacturers like Procter & Gamble, Unilever, Frito Lay, and Kraft—and retailers like Walmart, Tesco, and Vons. The advent of detailed sources of data about product sales, soon coupled with customer loyalty data, provided retailers with unique insights about product sales, customer buying patterns, and overall market trends that previously were not available to any player in the CPG-to-retail value chain. The new data sources literally changed the business models of many companies.

Then in the late 1990s, web clicks became the new knowledge currency, enabling online merchants to gain significant competitive advantage over their brick-and-mortar counterparts. The detailed insights buried in the web logs gave online merchants new insights into product sales and customer purchase behaviors, and gave online retailers the ability to manipulate the user experience to influence (through capabilities like recommendation engines) customers' purchase choices and the contents of their electronic shopping carts. Again, companies had to change their business models to survive.

Today, we are in the midst of yet another data-driven business revolution. New sources of social media, mobile, and sensor or machine-generated data hold the potential to rewire an organization's value creation processes. Social media data provide insights into customer interests, passions, affiliations, and associations that can be used to optimize your customer engagement processes (from customer acquisition, activation, maturation, up-sell/cross-sell, retention, through advocacy development). Machine or sensor-generated data provide real-time data feeds at the most granular level of detail that enable predictive maintenance, product performance recommendations, and network optimization. In addition, mobile devices enable location-based insights and drive real-time customer engagement that allow brick-and-mortar retailers to compete directly with online retailers in providing an improved, more engaging customer shopping experience.

The massive volumes (terabytes to petabytes), diversity, and complexity of the data are straining the capabilities of existing technology stacks. Traditional data warehouse and business intelligence architectures were not designed to handle petabytes of structured and unstructured data in real-time. This has resulted in the following challenges to both IT and business organizations:

Rigid business intelligence, data warehouse, and data management architectures are impeding the business from identifying and exploiting fleeting, short-lived business opportunities.

Retrospective reporting using aggregated data in batches can't leverage new analytic capabilities to develop predictive recommendations that guide business decisions.

Social, mobile, or machine-generated data insights are not available in a timely manner in a world where the real-time customer experience is becoming the norm.

Data aggregation and sampling destroys valuable nuances in the data that are key to uncovering new customer, product, operational, and market insights.

This blitz of new data has necessitated and driven technology innovation, much of it being powered by open source initiatives at digital media companies like Google (Big Table), Yahoo! (Hadoop), and Facebook (Hive and HBase), as well as universities (like Stanford, UC Irvine, and MIT). All of these big data developments hold the potential to paralyze businesses if they wait until the technology dust settles before moving forward. For those that wait, only bad things can happen:

Competitors innovate more quickly and are able to realize compelling cost structure advantages.

Profits and margins degenerate because competitors are able to identify, capture, and retain the most valuable customers.

Market share declines result from not being able to get the right products to market at the right time for the right customers.

Missed business opportunities occur because competitors have real-time listening devices rolling up real-time customer sentiment, product performance problems, and immediately-available monetization opportunities.

The time to move is now, because the risks of not moving can be devastating.

The Business Transformation Imperative

The big data movement is fueling a business transformation. Companies that are embracing big data as business transformational are moving from a retrospective, rearview mirror view of the business that uses partial slices of aggregated or sampled data in batch to monitor the business to a forward-looking, predictive view of operations that leverages all available data—including structured and unstructured data that may sit outside the four walls of the organization—in real-time to optimize business performance (see Table 1.1).

Table 1.1 Big Data Is About Business Transformation

Think of this as the advent of the real-time, predictive enterprise!

In the end, it's all about the data. Insight-hungry organizations are liberating the data that is buried deep inside their transactional and operational systems, and integrating that data with data that resides outside the organization's four walls (such as social media, mobile, service providers, and publicly available data). These organizations are discovering that data—and the key insights buried inside the data—has the power to transform how organizations understand their customers, partners, suppliers, products, operations, and markets. In the process, leading organizations are transforming their thinking on data, transitioning from treating data as an operational cost to be minimized to a mentality that nurtures data as a strategic asset that needs to be acquired, cleansed, transformed, enriched, and analyzed to yield actionable insights. Bottom-line: companies are seeking ways to acquire even more data that they can leverage throughout the organization's value creation processes.

Walmart Case Study

Data can transform both companies and industries. Walmart is famous for their use of data to transform their business model.

The cornerstone of his [Sam Walton's] company's success ultimately lay in selling goods at the lowest possible price, something he was able to do by pushing aside the middlemen and directly haggling with manufacturers to bring costs down. The idea to buy it low, stack it high, and sell it cheap became a sustainable business model largely because Walton, at the behest of David Glass, his eventual successor, heavily invested in software that could track consumer behavior in real time from the bar codes read at Walmart's checkout counters.

He shared the real-time data with suppliers to create partnerships that allowed Walmart to exert significant pressure on manufacturers to improve their productivity and become ever more efficient. As Walmart's influence grew, so did its power to nearly dictate the price, volume, delivery, packaging, and quality of many of its suppliers' products. The upshot: Walton flipped the supplier-retailer relationship upside down.¹

Walmart up-ended the balance of power in the CPG-to-retailer value chain. Before they had access to detailed POS scanner data, the CPG manufacturers (such as Procter & Gamble, Unilever, Kimberley Clark, and General Mills,) dictated to the retailers how much product they would be allowed to sell, at what prices, and using what promotions. But with access to customer insights that could be gleaned from POS data, the retailers were now in a position where they knew more about their customers' behaviors—what products they bought, what prices they were willing to pay, what promotions worked the most effectively, and what products they tended to buy in the same market basket. Add to this information the advent of the customer loyalty card, and the retailers knew in detail what products at what prices under what promotions appealed to which customers. Soon, the retailers were dictating terms to the CPG manufacturers—how much product they wanted to sell (demand-based forecasting), at what prices (yield and price optimization), and what promotions they wanted (promotional effectiveness). Some of these retailers even went one step further and figured out how to monetize their POS data by selling it back to the CPG manufacturers. For example, Walmart provides a data service to their CPG manufacturer partners, called Retail Link, which provides sales and inventory data on the manufacturer's products sold through Walmart.

Across almost all organizations, we are seeing multitudes of examples where data coupled with advanced analytics can transform key organizational business processes, such as:

Procurement: Identify which suppliers are most cost-effective in delivering products on-time and without damages.

Product Development: Uncover product usage insights to speed product development processes and improve new product launch effectiveness.

Manufacturing: Flag machinery and process variances that might be indicators of quality problems.

Distribution: Quantify optimal inventory levels and optimize supply chain activities based on external factors such as weather, holidays, and economic conditions.

Marketing: Identify which marketing promotions and campaigns are most effective in driving customer traffic, engagement, and sales, or use attribution analysis to optimize marketing mixes given marketing goals, customer behaviors, and channel behaviors.

Pricing and Yield Management: Optimize prices for perishable goods such as groceries, airline seats, concert tickets and fashion merchandise.

Merchandising: Optimize merchandise markdown based on current buying patterns, inventory levels, and product interest insights gleaned from social media data.

Sales: Optimize sales resource assignments, product mix, commissions modeling, and account assignments.

Store Operations: Optimize inventory levels given predicted buying patterns coupled with local demographic, weather, and events data.

Human Resources: Identify the characteristics and behaviors of your most successful and effective employees.

The Big Data Business Model Maturity Index

Customers often ask me:

How far can big data take us from a business perspective?

What could the ultimate endpoint look like?

How do I compare to others with respect to my organization's adoption of big data as a business enabler?

How far can I push big data to power—or even transform—my value creation processes?

To help address these types of questions, I've created the Big Data Business Model Maturity Index. This index provides a benchmark against which organizations can measure themselves as they look at what big data-enabled opportunities may lay ahead. Organizations can use this index to:

Get an idea of where they stand with respect to exploiting big data and advanced analytics to power their value creation processes and business models (their current state).

Identify where they want to be in the future (their desired state).

Organizations are moving at different paces with respect to how they are adopting big data and advanced analytics to create competitive advantages for themselves. Some organizations are moving very cautiously because they are unclear where and how to start, and which of the bevy of new technology innovations they need to deploy in order to start their big data journeys. Others are moving at a more aggressive pace to integrate big data and advanced analytics into their existing business processes in order to improve their organizational decision-making capabilities.

However, a select few are looking well beyond just improving their existing business processes with big data. These organizations are aggressively looking to identify and exploit new data monetization opportunities. That is, they are seeking out business opportunities where they can either sell their data (coupled with analytic insights) to others, integrate advanced analytics into their products to create intelligent products, or leverage the insights from big data to transform their customer relationships and customer experience.

Let's use the Big Data Business Model Maturity Index depicted in Figure 1.1 as a framework against which you can not only measure where your organization stands today, but also get some ideas on how far you can push the big data opportunity within your organization.

Figure 1.1 Big Data Business Model Maturity Index

Business Monitoring

In the Business Monitoring phase, you deploy Business Intelligence (BI) and traditional data warehouse capabilities to monitor, or report on, on-going business performance. Sometimes called business performance management, business monitoring uses basic analytics to flag under- or over-performing areas of the business, and automates sending alerts with pertinent information to concerned parties whenever such a situation occurs. The Business Monitoring phase leverages the following basic analytics to identify areas of the business requiring more investigation:

Trending, such as time series, moving averages, or seasonality

Comparisons to previous periods (weeks, months, etc.), events, or campaigns (for example, a back-to-school campaign)

Benchmarks against previous periods, previous campaigns, and industry benchmarks

Indices such as brand development, customer satisfaction, product performance, and financials

Shares, such as