Deploying Analytics to Take Advantage of Big DataBy Bob Violino | Posted 2014-04-29 Email Print
Some companies are finding innovative ways to leverage their information resources to improve customer service and create new opportunities for business growth.
"Every day, Edo tracks stores, blends and analyzes over 15 billion financial data transactions, with an outlook of daily growth by the millions," Sippel reports. "So we needed an analytics solution that could handle the immense volume of data flowing into our organization daily, while simultaneously extracting, transforming and loading that data in an efficient way to run queries."
The company wanted to provide better ROI for marketers and merchants. "Hadoop, complemented with Pentaho, helps us do just that," Sippel says, adding that data visualization improved the company's analytical capabilities so it can better serve its customers. Reporting capabilities in the system enable managers to create real-time and ad-hoc reports for customers.
The extract, transform and load (ETL) process time decreased by 70 percent almost immediately after implementing the Pentaho product, Sippel reports.
"Processes that originally took 30 hours decreased to four, depending on data load," he says. "This allows our analysts and management teams to make informed decisions about strategy significantly faster and to meet daily SLAs [service level agreements]. Additionally, we've seen an increased ease of use for reporting and analysis."
The main challenge the company faced was the time-consuming and costly need to manage the ETL process of 15 billion consumer transactions. "Our legacy data integration and analysis system took more than 30 hours to run, so meeting daily service level agreements was nearly impossible," Sippel says.
"After only a few weeks of implementing a data distribution on Hadoop, with Pentaho for data integration, we were able to reduce processing time to less than four hours, and often as little as two hours." To ensure that the database can't be accidentally modified, the system locks down file permissions through Hadoop.
With the growing use of social media by many businesses, the need for effective data management and analysis will likely increase.
"Say a firm wants to leverage Twitter and Facebook—unstructured data sources—to gain information on how a product is being received in the marketplace," says Eliot Arnold, managing partner at Crunch Data, an analytics managed services provider based in Denver. "The big data problem here is how you associate all that 'chatter' to a specific SKU [stock-keeping unit] or product group— structured data—that you'd like to analyze.
"Effective data management plays a critical part in tying together structured and unstructured data. When this is done right, the firm can benefit from both cost savings and sales growth." A data-driven business strategy is essential, he says.