Big Data and Analytics Are an Ideal Match

By Tony Kontzer  |  Posted 2013-05-21
big data and analytics

By Tony Kontzer

Long before the concept of big data took hold at iconic clothing retailer Guess?, the company considered itself a sort of business intelligence innovator.

Armed with MicroStrategy's business intelligence (BI) application backed by an Oracle database, the Los Angeles-based company was collecting an abundance of sales and inventory data, and was using it to generate informative reports. But only a handful of power users were taking advantage of the tool and driving most of the reporting.

The company needed to figure out a way to get the increasingly valuable data into the BI environment and then into the hands of the merchants who were deciding which products went to which stores and in what quantities.

"They're touchy-feely people who like products and visuals, and getting them to drill down into business intelligence is always a challenge," Michael Relich, executive vice president and CIO at Guess?, says of the company's merchants. "We'd have analysts dump things into spreadsheets and then cut and paste pictures. It was crazy."

Adding to the challenge was the fact that the database couldn't process the mushrooming data volume fast enough to keep up with merchants' growing taste for answers. For instance, if merchants wanted to figure out what sizes had been selling at the company's 1,500-plus retail locations over the previous six months, the related queries of the BI system would run for hours before timing out.

The path to an answer started four years ago, when Relich and his team decided to look for a solution better suited to processing big data, eventually choosing HP's Vertica analytics platform. Relich was attracted by Vertica's use of massive parallel processing to split queries into multiple boxes, and its ability to scale inexpensively with commodity hardware. Even so, he wasn't prepared for performance that was as much as 100 times better than the database provided.

"When we did the first queries, they were done so fast, we thought they were broken," Relich recalls. "Queries that would run in minutes in Oracle run in seconds in Vertica."

The experience at Guess? taught it what many companies are learning today, namely that big data and business intelligence and business analytics (BA) don't merely feed each other. When used in tandem, they take data analysis to a whole new level. Having a good BI/BA application in place makes it easier for users to tap big data, and having big data technologies in place can fuel the value of a BI/BA system.

"Augmenting your business intelligence practice with big data is a very intelligent thing to do," says Mike Matchett, senior analyst with the Taneja Group. "The two go together very strongly."

It's a practice that should become much more commonplace now that more affordable analytics services and appliances are joining open-source big data tools in the market, says Dana Gardner, principal analyst at Interarbor Solutions.

The one-two punch of big data and business intelligence enabled Guess? to develop an  iPad application that won an innovation award from The Data Warehousing Institute. By combining images from its e-commerce system with the flow of information coming from the data warehouse, Guess? is now able to deliver sales and inventory data to merchants fast and with a visual element to boot.

"It's the equivalent of about 18 different dashboards combined into one app," says Relich.

Now, instead of arriving at stores armed with binders filled with dated information, merchants have real-time insight into sales trends, store data and product availability at their fingertips. Although the app was designed specifically for merchants, its 150 regular users include district managers attracted by its ability to help them with store planning.

The impact of the app has been huge: Product markdowns have been reduced, allocations have improved, and merchants and district managers have a better idea of what, how much and when product is needed.

"Retail is all about having the right quantity of the right product in the right locations," says Relich. "We're able to identify store issues much quicker and respond to them."

That, according to Interarbor's Gardner, is the kind of result companies aspire to achieve with big data initiatives. "They want to get all the data possible in order to make a decision with the highest order of likelihood of being correct," he says.

Big Data Analytics Drives Success

At Ford Motor, big data-enabled analytics has been tied to $100 million in annual profits, a figure that led to a recent analytics award from the Institute for Operations Research and the Management Sciences. Part of that success is attributable to the efforts of Michael Cavaretta, technical leader for predictive analytics and data mining for the automaker's research and advanced engineering group, who is focused on using data to improve Dearborn, Mich.-based Ford's internal business processes.

Cavaretta's team is using a combination of big data tools and business analytics applications in a number of interesting ways. They're creating data mashups of previously siloed information by linking business processes to warranty and marketing data and the like; crunching internal and external social media posts and figuring out how to link them with and inform business processes; and capturing huge amounts of data generated by vehicles—not only to refine vehicle design, but also to determine what additional types of data could be collected.

The latter of these, in particular, has huge implications as automakers add more sensors to vehicles so they can monitor performance, crank up customer service levels and improve future designs.

For instance, Ford's Fusion Energi plug-in hybrid generates and stores 25 gigabytes of data per hour on everything from engine temperature, speed and vehicle load to road conditions and general operating efficiency. That data flow can increase to as much as 4 terabytes per hour when running tests with special instruments—instruments that Cavaretta says could easily become standard equipment in a few years. Being able to capture, store and analyze that data, and then apply the insight to the right processes in real time will require finely tuned big data and analytics platforms.

Along those lines, Ford has been experimenting with a gamut of open-source and commercial technologies. Cavaretta says his team has worked with big data tools such as Hadoop, HIVE and Pig on the big data side; traditional databases such as SQLServer, MySQL, Oracle and Teradata; BI and BA software such as IBM's PASW Statistics and R; and specialized data mining tools like Weka, RapidMiner and KNIME.

It's an assortment that flies in the face of predictions that big data was essentially a replacement for business intelligence.

"The initial impression a lot of people had was that this was going to be a whole new thing: Put in big data and business intelligence goes away," says Cavaretta. "I don't think that's the case. There are a lot of BI initiatives that would be greatly helped by big data."

That said, Taneja Group's Matchett believes one of the big data mistakes that companies can make is to jump the gun before a viable business intelligence or analytics solution is in place. "If I just invest in big data without an application for it, I'm not going to get much of a return," he says.

As Ford works to refine its big data-business intelligence/analytics intersection, there's little doubt in Cavaretta's mind that the combination of the two is powerful.

"The biggest thing about big data is that it changes the value of analytics," he says. "People have been focusing BI/BA on large data sets, but not at the level where big data needed to play.

"Now, new tools are giving them the ability to analyze data in new ways. Soon, technology will make things relatively easy, and what you'll be left with is analytics that can give the company value."