Big Data Governance: 5 Lessons Learned From PRISMPosted 2013-07-08 Email Print
Regardless of what you think of the NSA’s efforts to collect information on Americans, one fact is clear: Big data is real, it’s here to stay and it’s dangerous.
Lesson 4: Figure out what to do with all your data.
The fourth lesson involves what we at ZapThink like to call the big data corollary to Parkinson’s Law, which states that the amount of work you have will expand to fill the available time. The big data corollary says that the amount of data you collect will expand to consume your ability to store and process it. In other words, if it’s possible to collect big data, somebody will. The question is not whether to collect big data—it's what to do with it.
Lesson 5: Find value in historical data, not just current data.
Finally, the fifth lesson is actually a lesson from something the NSA is not doing, because for the agency, current data is more valuable than historical data. The NSA’s paramount concern is to mine current intelligence: what terrorists are doing right now.
However, your company might find value in using historical data along with current data to solve problems. If some of your business issues do deal with historical trends, then your data sets have ballooned again, as have your data governance challenges.
The NSA was collectingonly phone call metadata because that metadata met its needs. But what about the data itself—the audio portion of the calls? Perhaps the NSA is currently unable to collect such vast quantities of data. But if that's the case, it’s only a matter of time. The question is, once the NSA is able to collect all call audio, will it? I believe it will. After all, that’s the corollary to Parkinson’s Law in action.
In fact, we might as well assume that somewhere in the federal government, agents are collecting all the data—all the phone calls, emails, tweets, text messages, blog posts, forum comments, log files, everything. Because even if they aren’t able to amass the whole shebang yet, it’s just a matter of time till they can.
While this scenario may seem like a page out of Orwell’s 1984, the most important lesson here is that data governance is now critically important. It’s no longer a question of whether we can collect big data. The question is: What should we do with big data once we have it?
Jason Bloomberg is president of ZapThink, a Dovel Technologies Co. His fourth book, The Agile Architecture Revolution: How Cloud Computing, REST-based SOA, and Mobile Computing Are Changing Enterprise IT (John Wiley & Sons), was published in March 2013.