Few organizations manage more data than the National Oceanic and Atmospheric Administration (NOAA). The agency, which oversees the National Weather Service and other federal entities, gathers more than 20 terabytes of data every day—more than twice the data of the entire printed collection of the U.S. Department of Commerce.
This environmental intelligence streams in from numerous sources, including Doppler radar systems, weather satellites, buoy networks and stations, tide gauges and real-time weather stations, as well as data from ships and aircraft.
“Unfortunately, only a small percentage of data has historically been available outside the walls of NOAA,” says Zachary Goldstein, CIO and director of High Performance Computing and Communications at NOAA. What’s more, “The demand for this data is increasing rapidly. It’s critical that the agency makes it available in a form that other organizations can use.”
The list of organizations and groups includes a diverse array—from emergency responders and private weather forecasting services to the reinsurance and seafood industries. “The availability of the data opens up all sorts of possibilities and opportunities,” he says.
A Collaborative Cloud Initiative
In April 2015, the U.S. Department of Commerce announced a new big data initiative that it hopes will catapult NOAA into the digital age. The agency will collaborate with a consortium of technology and cloud providers—including Amazon Web Services, Google Cloud Platform, IBM, Microsoft and the Open Cloud Consortium—to build an open data framework.
The goal is to unlock resources and transform data capabilities to support a data-driven economy, Goldstein reports. The project was established through Cooperative Research and Development Agreements (CRADA), which provide the foundation for a set of data alliances led by each of the anchor companies.
“Cloud computing is an ideal way to unlock the value of all the data,” Goldstein says. “It allows us to make it available without investing in expensive upgrades and incurring a huge budget increase.”
In fact, much of the cost of the initiative will not be borne by government. Among other things, CRADA will examine how NOAA can better serve the private sector through data sharing; what systems are required to support data sharing; what standards are necessary; and how best to collaborate moving forward. “It’s an entirely different approach and mindset,” Goldstein says.
The initiative, which will tap into the technology strengths of the various partners, will not only allow NOAA to make data more readily available, it will also help the agency store, manage and consolidate data sets for maximum value to the agency and users.
Goldstein says the agency is rolling out the initiative at a controlled pace so that it can prove the business case and build on results. “We want to make sure the lessons learned are available for other agencies,” he explains.
NOAA also hopes that the project will introduce entirely new ways to use the data. These innovations could result in new business models, as well as new products and services, in the private sector.