As Hadoop technology evolves, from initially solving massive data storage problems for companies like Google and Facebook, to now being adopted by more and more enterprises for processing big data, Hadoop has had a profound impact on the enterprise data landscape. Understanding its common usage patterns can greatly reduce complexity.
Just a few weeks ago, Apache Hadoop 2.0 was officially released, marking a monumental milestone in the Hadoop ecosystem, as it ushered in an unprecedented revolution in data storage paradigms. Hadoop retains its quintessential “big data” foundational technology, but is it suited for today’s databases and data warehousing approaches? And is there a general pattern that can practically reduce its inherent complexity?
Common Hadoop Usage Patterns
Hadoop was originally conceived to solve massive data storage problems for companies like Yahoo, Google, and Facebook at very low cost. Now, it is increasingly being introduced into enterprise environments to process new and diverse data types. Machine-generated data, sensor data, social data, web logs, and other data types are growing exponentially, and this data is often (though not always) unstructured. It is precisely this type of data that elevates the human-machine dialogue from “data analysis” to “big data analytics”: because mining this data can yield competitive business advantages.
Analytical applications have proliferated in various forms, most importantly those that can address the needs of a specific vertical industry. At first glance, they may seem unrelated across industries and verticals, but in reality, when observed at the infrastructure level, some very clear patterns emerge—namely, the following three patterns:
Pattern 1: Data Refinery
The Hadoop “Data Refinery” pattern enables organizations to incorporate these new data sources into their conventional BI and analytical applications. For example, I might have an application that can view customer data built on top of ERP and CRM systems. But how can I discover their interests from their web sessions (based on our website)? The “Data Refinery” usage pattern is exactly what customers expect.
The key concept here is that Hadoop is used to extract and process large volumes of data to make it more manageable. The resulting data is then loaded into existing data systems, where it can be accessed using traditional tools—but keep in mind, these operations are all built upon a richer dataset. In some respects, this is the simplest use case, because without requiring major modifications to traditional approaches, enterprises can clearly benefit from Hadoop. Regardless of the vertical, the refinery concept remains applicable. In financial services, we see organizations refining transaction data to better understand markets, analyze, and discover value from complex portfolios. Energy companies use big data to analyze consumption levels across different regions to better forecast production levels. Retail businesses (and any consumer-facing organizations) frequently use the refinery pattern to gain insights into online popularity. Telecom companies use the refinery pattern to mine call detail records and extract useful information for optimizing billing approaches. Finally, in expensive, mission-critical vertical equipment, we often find Hadoop being used for predictive analytics and proactive fault identification. In communications technology, this could be a network base station. In franchise restaurants, it could be used to monitor data from refrigeration units.
Pattern 2: Data Exploration with Apache Hadoop
The second most common use case we call “Data Exploration.” In this scenario, organizations ingest and store large volumes of new data on Hadoop, and then explore this data directly. So rather than using Hadoop as a staging area for processing and then moving data into an enterprise data warehouse (as in the refinery use case), the data is kept on Hadoop and explored directly.
The data exploration use case typically begins when enterprises start to explore data that was previously discarded (such as web logs, social media data, and so on) and build entirely new analytical applications that work directly with this data. Nearly every vertical system can benefit from the exploration use case. In financial services, we can use the exploration use case to perform forensics or identify fraud. Professional sports teams leverage data science to analyze trades and annual drafts, much like what we saw in the movie Moneyball. In summary, data science and exploration can be used to discover new business opportunities or new insights that were simply impossible before Hadoop.
Pattern 3: Application Enrichment
The third and final use case is “Application Enrichment.” In this scenario, the data stored in Hadoop determines the application’s purpose. For example, by mining all stored web session data, we can deliver personalized experiences to users when they return to the website. By mining this data stored in Hadoop, we can extract tremendous value from session history—such as providing timely feedback based on a user’s historical records.
This use case is the foundation of business operations for many of the world’s largest websites, such as Yahoo and Facebook. Through customized user experiences, they can effectively differentiate themselves from their competitors. This was Yahoo’s second Hadoop use case, just as they realized Hadoop could help improve ad placement. This concept has transformed large websites and is also enabling traditional enterprises to improve sales, while even some smaller organizations are using these concepts to implement dynamic pricing at retail outlets.
As you might expect, as organizations become more familiar with refining and exploring data on Hadoop, this last and most archetypal use case is increasingly being adopted and accepted. But at the same time, it also hints at what Hadoop can do going forward, and over time, traditional database applications will gradually be replaced by Hadoop applications.
Of course, any new platform technology brings a certain degree of complexity when introduced into an IT enterprise environment, and Hadoop is no exception. Whether you are using Hadoop to improve, explore, or enrich your data, compatibility with existing IT infrastructure will be key. This is why the Hadoop ecosystem and solutions that integrate across different vendors have seen significant growth. Hadoop has the potential to have a profound impact on the enterprise data landscape, and by understanding common usage patterns, you can greatly reduce its complexity.
Via: http://www.linuxeden.com/html/news/20131228/146962.html