Responding to Local Conditions Rapidly

By Kevin Fogarty Print this article Print

Out-of-band, near-real-time data warehousing allows companies that could never afford real-time data before find good reasons to pay for it.

That’s normally the kind of calculation that’s made over days or weeks in the grocery business, and one that’s difficult to justify with hard return-on-investment numbers, Lewis says. But it does give individual stores a greater ability to respond to local conditions at a pace competitors can’t match, he says.

Lewis uses real-time data integration software from GoldenGate Software, which uses a centralized data warehouse as a repository and distribution engine for online transaction processing data. Rather than process online transaction processing (OLTP) data in batches nightly or hourly, GoldenGate processes it continually, trickling transactions a few at a time into the data warehouse and disseminating it from there to other applications.

That way, according to Sami Akbay, vice president of product management and marketing for GoldenGate, the real-time system doesn’t burden the production system by polling it continually. The OLTP system operates normally, and all the business intelligence and other real-time applications pull data from the warehouse, which also cleanses the data to make it usable by other applications.

GoldenGate is far from the only vendor taking a similar approach. HiT Software, for example, pulls data off a production server and replicates it into an XML repository, allowing customers to run any analyses they need. Ascential Software (acquired by IBM in 2005), which began life as Informix software, also offers near real-time implementations. 

Software in the whole category works the same way historical reporting and data mining does, according to reports from IDC business analytics analyst Dan Vesset.

A market survey released last month found that customers not only continue to buy more sophisticated analytics, but that they’re investing in both technology and human skills to expand the range of sophistication of their analyses and shorten the time it takes to get answers.

Near-real-time data analysis using data warehouses and other out-of-band approaches are a popular way to address the problem, the report says. The configuration is relatively simple, but it’s extremely fast, very reliable and far cheaper than systems with comparable results, according to Curt Miller, CTO of SinglePoint, which manages TV-Web marketing campaigns for customers such as Black Entertainment Television, Bravo, NBC and Fox.

“In TV-land, in many cases, especially on a live show, producers will do things that drive traffic to their Web site or through cell phones during the show,” Miller says. “What we had was a whole system geared for post-delivery reconciliation all of a sudden getting demands for immediate results.”

Results that SinglePoint had collated nightly suddenly had to be assembled and distributed continually, without trashing the Cognos data manager and Sybase database that formed the core of SinglePoint’s system.

“We jiggered our system with triggers so it would run every two hours, but it wasn’t something that would answer a question in five minutes,” Miller says. “We were able to point a mirror at that data and take the data out to the archive server and run the results against the mirror, so we aren’t hurting our production systems.”

Miller also declined to discuss cost, but says the GoldenGate software and integration services were comparable to any other kind of database work, not the highest-of-the-high costs that would be attached to traditional real-time data systems.

“Right now, when you respond to one of these shows, you vote and see a message go through your phone, you hear the beep and 15 seconds later it’s in the archive server,” Miller says. “Producers can see how they were doing not two hours ago; now they know how they were doing 10 seconds ago.”

This article was originally published on 2008-05-02
eWeek eWeek

Have the latest technology news and resources emailed to you everyday.