New York City Tranforms Technology Infrastructure

By Eileen Feretic  |  Posted 2010-04-08

The CIO of New York City has one of the most demanding IT jobs in the world—one that involves providing technology services or support to more than 50 city agencies, approximately 300,000 employees, about 225,000 businesses and some 8.2 million residents across the five boroughs.

That’s the challenge facing Carole Post, the recently appointed CIO and commissioner of the Department of Information Technology and Telecommunications (DoITT). In January, she succeeded Paul Cosgrave, who retired as commissioner at the end of 2009, and began crafting her vision of how DoITT can most efficiently and cost-effectively service its constituents.

“When I started as commissioner in January, Mayor [Michael] Bloomberg charged me with developing a 30-day report that would be a road map for IT during the mayor’s third term,” Post said. “That vision is centered on three concepts: service delivery, accountability and core competencies.”

The goal was to create a comprehensive technology strategy designed to provide more coordinated, effective and efficient citywide IT services, building on the foundation that Cosgrave started. Bloomberg has made it clear that “government should constantly be looking for new and better ways to provide information and services,” and that’s the mandate he’s given to DoITT.

The first part of Post’s road map, service delivery, involves providing top-notch IT services to the city agencies and the public. “We want to be the finest IT delivery service that we can be,” Post said. “Our goal is to be on the leading edge of new technology, as opposed to playing catch-up, as governments tend to do.”

The second area, accountability, involves performance metrics and accountability. “We plan to augment what we already have in terms of performance metrics on the delivery of IT services, filling in where we have gaps,” she said. “We also plan to implement new levels of accountability for our vendors, partners and service providers.”

To execute on the third concept, core competencies, DoITT will articulate what those competencies are. “We’ll determine the areas in which we should be best in class, as well as the areas that are not moving us forward,” Post explained. “We’ll figure out how to do some things differently and where we should do less of other things. We don’t need to be all things to all people. We’ll focus our efforts on modernizing the back-office applications, such as data center consolidation.”

In fact, citywide data center consolidation is one of the primary recommendations of Post’s report to the mayor. By modernizing and consolidating its outdated data infrastructure, the city expects to save up to $100 million over five years.

At a press conference held on March 1, Bloomberg stated: “Today, city agencies are embracing new technology, constantly adding hardware and software to improve services and make information more readily available to the public. Instead of building these systems on the often-outdated and varied IT systems that exist at individual agencies, we will consolidate them in state-of-the-art data centers that can support the growing needs of forward-thinking agencies while saving the city tens of million of dollars.”

Consolidating to Save Taxpayer Dollars

Beyond providing world-class service delivery, DoITT aims to reduce costs and make the city more sustainable by consolidating its 50 data centers, which serve approximately four dozen agencies.

In his State of the City speech, Bloomberg pledged “to achieve cost savings by reducing redundancies within city agencies, including information technology resources.” One initiative is the Citywide IT Infrastructure Services program (CITIServ), whereby DoITT will build a standardized infrastructure that will give agencies the same secure data centers they currently have, but through a shared structure that will provide a lower cost of operation, reduced energy consumption, stronger security and improved services.

CITIServ offerings include help desk, hosting, storage, e-mail, virtualization and network services. DoITT plans to start migrating agencies to these offerings later this year. The first to be migrated will be the departments of Education, Buildings, Housing Preservation and Development, Sanitation and Finance.

These migrations will build on the consolidation efforts that have been going on in the city for years, with DoITT centralizing mainframes, servers, networks and e-mail systems, as well as virtualizing servers.

“We started out with close to 1,000 physical boxes, and we virtualized down to about 300 boxes, for a current total of about 800 virtual machines,” said Michael Bimonte, deputy commissioner, IT Services, DoITT. “We’ve converted them to virtual systems and have between a 15-1 and a 20-1 ratio. We have signed enterprise license agreements with the primary vendor, VMware, for the virtualization, and that’s enabled a lot of the city agencies to take advantage of virtualization.”

The fact that New York City has been in the virtualization business for at least two years gives it a leg up because virtualization is the technology driving cloud computing.

“We’ve moved from the planning stage to the operational stage on this initiative,” Post said, adding that DoITT is far along in its negotiations to acquire leases for the first of two new data centers.

Leaving a Legacy

Another major DoITT initiative revolves around legacy systems. When former CIO Cosgrave was interviewed last December, he reported that the city “had a lot of growth of non-mainframe hardware over the last 10 years, but the heart of computer processing in the city is the mainframe. The issue is that we’ve got every database management system known to man running on the mainframes, and we need to consolidate them. That’s the only way we’re going to make substantial cost reductions.”

The city agencies will have to make these decisions, since they own the applications. They have to decide whether they want to switch from one mainframe to another mainframe environment, or take the opportunity to rewrite the application to run on a lower-cost platform. That decision has to be made for each application.

“Our primary data center grew by having one of everything,” Bimonte added. “Give us your tired, your weak and your broken down, and we’ll take care of it. That worked initially, but in order to realize the full benefits and cost savings of consolidation, we need to build a clean, green, standard architecture.”

But that doesn’t mean there will be just one platform. The data centers will encompass mainframe and Wintel technology, as well as some combination of Linux. Within each platform, it will get down to the level of which hardware, database management system, storage and software will be used.

Cutting Costs, Not Service

DoITT will also provide the city agencies with an enhanced online self-service request catalog enabling them to go online and choose a technology service. That service launches a form, which is prepopulated based on the questions the agency representative asks, and then generates a request-for-service ticket in the city’s BMC Remedy IT Service Management system. The user will get an e-mail with a ticket number, so he or she can track the progress of the request through its life cycle, and will get another e-mail when the request has been fulfilled by the service desk.

“We’ve implemented service desk, change management, problem, incident and asset management,” Bimonte explained. “Now we’re moving to full asset life cycle management, and configuration and patch management.”

The results are impressive. “Since we started, we’ve reduced our cost per service call from $42 or $43 down to between $10 and $11,” he said.

“We built in a lot of automation and integrated many of our point products and our monitoring with the Remedy tool,” Bimonte added. “Monitoring is based on rules we’ve written, and that automatically generates tickets. Based on the criticality of a particular ticket, the system automatically routes it to a first-, second- or third-level engineer for troubleshooting. So we’ve been able to reduce reassignments of customer queries by 80 percent to 90 percent.

“The system enabled us to build a 24/7 citywide service desk, which is used by almost all the agencies. We’re now in the process of rolling it out to the rest of the agencies in a consolidated fashion.”

Despite budget cuts of between 25 percent and 30 percent over the last three years, the service desk has been able to increase the number of user contacts it handles from around 4,000 up to 16,000 a month. “Yet, we’ve reduced our mean time to repair by 300 percent, due to the automation and advanced routing of the tools we’re using,” Bimonte said. “That’s a huge cost saving.”

When any problems do crop up, DoITT handles them quickly and efficiently. For example, the department constantly monitors the network, and if a component starts to act up, it can be dealt with long before it fails. This enables DoITT to provide much better overall service to the agencies.

Networking the City

In response to another recommendation in Post’s 30-day plan, DoITT will create a new Office of Telecommunications and Broadband Policy. This office will be responsible for developing, maintaining and implementing the city’s telecommunications strategies. It will also coordinate efforts “to increase public access to broadband technologies through public computer centers, school programs and the expansion of WiFi in parks.”

Communications technologies play an important role in the life of the city. The NYC Wireless network (NYCWiN) is a high-speed mobile data network that is operational across the city’s more than 300 square miles. DoITT operates NYCWiN, launched in May 2009 in conjunction with Northrop Grumman, as an extension of CityNet, a fully redundant fiber-optic network.

“We’ve been running the fiber network for a number of years and have been upgrading it over time,” Bimonte said. “We now need to make the next big upgrade, WDM [wavelength-division multiplexing] capacity, and that’s currently in the works.”

Though most agencies use the city’s network, some still run their own. However, DoITT is in the process of moving some of these agencies to CityNet, which will save them a significant amount of money. At this time, an analysis of the move is taking place at the Department of Education.

The wireless network was developed primarily to support public safety, and it gives first responders high-speed access to large data files. DoITT is planning a major upgrade in police cars and fire engines that will enable the police to transmit warrant photographs and streaming videos to police cars, and the fire department to send maps and building layouts to fire fighters on their way to a fire.

But the city’s broadband initiative goes far beyond the agencies. “We’ve applied for two grants on the broadband stimulus from the federal government,” Post said. One recently resulted in a $22 million award to provide free computers, discounted broadband service and support services to more than 18,000 low-income sixth-grade students and their families. This is part of a program to increase broadband adoption among underserved populations. The second grant is for a program to expand the availability of public computer centers in the city’s highest poverty areas.

New York City has a number of partners involved in this initiative, including Verizon and the cable companies. However, the initiative depends on getting the broadband stimulus funds from the federal government.

Another major networking goal is to make WiFi available as widely as possible. Later this spring, DoITT will release an RFP to expand the availability of WiFi in public spaces.

Mobile technology is another communications arena managed by DoITT. It currently hosts 20 to 25 mobile solutions, which are a boon to agencies that have a lot of field workers. These employees can handle all their work remotely, without having to go to the office, and this has resulted in a huge increase in productivity for the agencies.

Securing New York

Information security is a huge challenge and an onerous responsibility for every CIO, but when you’re responsible for the security of New York City’s IT infrastructure and data, the task can seem Herculean in scope.

“We’ve made some great strides is this area,” Bimonte said. “We’ve developed a number of IT security policies, including acceptable-use policies for employees and application accreditation policies. And the IT security office sends out cyber-security awareness materials on a regular basis.

“In addition, we’re entering into an enterprise license agreement with a top security provider to make it easier for agencies to procure a standard set of security products. Rather than having each agency go through the procurement process on its own, we could do that once for the whole city, which would save a lot of time and money.

“We’ve also introduced some standard procedures so that a new application has to have a thorough security accreditation before it’s put into production. All our Internet-facing applications are set up in a way that protects our internal systems, so no one can use an application to get into our systems.”

Another growing security challenge is social networking, which has been “growing organically throughout the city,” according to Post. Until recently, there hadn’t been any central oversight of these efforts, but that’s about to change.

“In partnership with the city’s Law Department and policymakers, we are going to encourage the use of creative networking sites, with appropriate guidelines and structures around them,” she explained. “We will serve as something of a moderator for these sites, creating policies for the agencies to follow. We are not looking to police them or limit them, but we don’t want agencies to unilaterally develop social networking policies.”

The city agencies will be asked to register their social networking efforts with DoITT so that it can centrally track these sites and accounts.

The massive technology changes taking place in New York City government are requiring an equally huge cultural change. To facilitate this change, DoITT is working with agency CIOs to relieve them of technology obligations that are not core to their mission.

Post put it this way: “If the agencies let us manage their IT infrastructure, we will save them money and time, and enable them to focus more effectively on their core missions.”