Migrating a Big Data Warehouse to the Cloud

By Guest Author Print this article Print
migrating data warehouse to cloud

When migrating to a cloud, be sure to investigate third-party solutions, automate data migration and syncing, do a proof-of-concept and check default settings.

Our move to Amazon Redshift allowed us to thin out our production server. Our overall data store grows by about 10GB a day, and our goal is to keep the SQL server system at about 500GB. Everything else is now stored in Redshift. This reduction in on-premises storage has helped us save thousands of dollars per year, while greatly improving the performance of our production server.

Don’t assume default settings on third-party solutions will work well for your environment. Chances are they won’t.

Vendor products typically have default settings that cater to a broad spectrum of environments. These settings may or may not be optimal for your environment. It is essential to work with the vendor to learn how to correctly tune the product so the tool fits your specific environment.  

Attunity CloudBeam has a good set of default options in its change data capture (CDC) feature. However, in our case, the default settings hindered our ability to replicate data as quickly as we wanted. So we worked with Attunity to adjust the CDC batch-tuning default value to reduce replication latency to the desired level.

The best advice I can give you is to test, test and then test some more. You’ll be glad later that you took the time to do it.

Now that we’re up and running with Amazon Redshift, I can vouch for the numerous benefits it offers. We are extremely happy with our implementation, and I’m now going to reserve my long-awaited vacations.

Adam Haines, director of systems engineering at Federated Sample, has more than 10 years of industry experience. He is a Microsoft Certified Master (MCM), a Microsoft Certified Solutions Master (MCSM) and a former Microsoft Most Valuable Professional (MVP).

This article was originally published on 2014-05-01
eWeek eWeek

Have the latest technology news and resources emailed to you everyday.