Merrill Lynch & Co.: Web Services, Millions of Transactions, All Good

By Mel Duvall  |  Posted 2006-02-07

Jim Crew could be considered an All-Star in the fledgling field of service oriented architecture. Over the past four years, Crew, as the former head of database infrastructure at Wall Street brokerage Merrill Lynch & Co., had spearheaded an effort to modernize the firm's multibillion-dollar investment in mainframe technology by building a platform to easily integrate many of the company's decades-old legacy applications with new software.

Since 2001, his team has helped generate as many as 420 Web services—essentially using a collection of Internet technologies like eXtenstible Markup Language (XML) and Simple Object Access Protocol (SOAP) to make operations currently performed on the mainframe available to programmers creating new applications in modern environments such as Sun Microsystems' Java or Microsoft's .NET.

In the process, he has helped Merrill Lynch save as much as $42 million in application development by being able to leverage business-hardened code on the mainframe and by making new hardware purchases unnecessary. In getting information directly from the mainframe and reusing existing legacy applications, Merrill has avoided having to copy information to more modern database platforms like Oracle and Sybase or rewriting software to run on application servers.

But much like skiing's Bode Miller, this all-star is also a rebel.

Unlike many other corporations that are embracing Web services, Merrill decided to do it without the help of a traditional vendor of Web services platforms, such as webMethods, Neon Systems or BEA Systems. Instead, it chose to develop and implement an entirely new Web services architecture on its own.

"We took a very careful look at what was available in the industry at the time. It would have been a lot easier to buy what we needed than build it," says Crew, who headed up Merrill's database organization in Hopewell, N.J. It would have also allowed Merrill to get up and running faster. "But we couldn't find anything that met our requirements," Crew explains. "In our opinion the [existing] SOA vendors were going about the problem the wrong way."

The crux of the problem, as Crew saw it, was that most of the Web services development platforms being pitched by the vendors had been built from the perspective of the distributed programmer, or programmers using modern development tools such as Java and .NET.

That just didn't make sense for Merrill because the firm would have to retrain many of its 1,200 programmers and purchase more powerful workstations to develop the Web services platform.

In fact, Merrill has a huge IBM mainframe installation—one of the largest in the world—with 1,200 programmers supporting some 23,000 mainframe programs that process 80 million Customer Information Control System (CICS) transactions per day. Each time a person logs on to a Merrill Lynch account online, or enters his password at a bank machine, could be considered one transaction, although the action of signing in at a bank machine or online would trigger many other transactions.

The company's mainframe programmers were the guardians of that "treasure." They knew what work was performed by the legacy programs and what information was stored in the databases, and how they might be leveraged by other applications in development. Training those programmers in Java or .NET would have been an expensive undertaking.

According to research firm Gartner, it can take an enterprise 12 months and as much as $67,000 to retrain a COBOL developer in Java.

In addition, it would have required Merrill to purchase new, more powerful desktops for its mainframe programmers to run the SOA application development software. As a result, Merrill opted to build its own Web development platform from scratch.

QUESTION: What's the biggest headache your tech team can expect to get from deploying Web services? Let us know at

Story Guide:

Merrill Lynch & Co.: Web Services, Millions of Transactions;, All Good

  • Unlocking Access to Customer-Service Functions
  • Despite Complexity, Merrill Skipped Middleware
  • Mainframe Remains Strategic
  • Base Case: Merrill Lynch By the Numbers

    Next page: Unlocking Access to Customer-Service Functions

    -Service Functions">

    The actual case for developing Web services was a no-brainer. As reliable and robust as Merrill's mainframe infrastructure was, it was also fraught with limitations. As new Internet-based applications, such as self-help credit card balance checks, were being developed, programmers needed to access data locked in Merrill's mainframe vaults. It was difficult to tap directly into those vaults using non-mainframe-based software.

    In fact, it's estimated that 90% of all new development costs involve the plumbing to integrate mainframe applications with newer platforms. Merrill had instead taken the approach of copying that data into Oracle, Sybase or Microsoft SQL databases, which could more easily integrate with server-based applications.

    But the copying of the data created two problems: Copying tends to be unreliable because of disk errors, read errors and running-out-of-space errors. Second, the data could be out-of-date as soon as it is copied.

    For example, a client making several trades would have to wait until the following day to see an accurate balance in his account. As a result, a client might make a trade believing there were adequate funds, but that trade would have to be rejected because, in fact, the funds were not available.

    "A lot of money was being spent copying data to Oracle or SQL servers, and the frustrating thing was this information was available and up-to-date on the mainframe—it was just difficult to get at," Crew says.

    Web services seemed the ideal solution to the challenge. Through a set of widely accepted standards, such as XML, which uses tags to describe data and which can in turn be understood by any XML-aware application; and SOAP, which provides a means for a program in one kind of operating system to communicate with a program in another kind of operating system, it was possible to open up Merrill's mainframe vault to modern applications.

    In a simple scenario, a client using an online application to look up his credit card balance submits a request via a Web browser. The mainframe runs the requested operation (obtain credit card balance) and sends the information back to the client via SOAP. XML makes it easier for the mainframe to interpret the online application, and in turn assists in allowing the online program to understand the response delivered by the mainframe.

    In 2000, Crew formed a team within the database group to begin exploring the relatively new CICS Transport Control Protocol/Internet Protocol (TCP/IP) capabilities. As a result, the group wrote a tool, using CICS TCP/IP, to replace an existing mainframe tool used for getting real-time market data to the mainframe.

    The existing tool employed the more traditional Systems Network Architecture (SNA) protocols, a set of network protocols developed by IBM, and was used to allow a Merrill Lynch customer accessing his account online to get pricing of a traded security directly from the mainframe. The existing SNA-based tool was quite complex and consumed more than 200 mainframe million instructions per second (MIPS). The new TCP/IP-based tool was faster, more accurate and consumed fewer than 3 mainframe MIPS. This reduction saves Merrill Lynch approximately $2 million per year in operating costs.

    The success of this project, and the experience the team gained with TCP/IP, prompted the team to explore ways to expose this, and other mainframe data and services, to the distributed world. This led directly to the exploration of Web services standards and the development of the X4ML product.

    Crew says the organization got a lot of credit from senior executives for the real-time quoting application, and just as important, the company's mainframe programmers recognized that Web services could give the mainframe new life in the Web world. It was during a period when Merrill was laying off some 24,000 workers globally, and mainframe programmers were chief among the casualties. "I went back and issued a challenge," Crew says. "I said, 'OK, we just saved $2 million; now let's go see if we can save the company $20 million.'"

    In late 2000, Crew formed a strategy team to develop the Web services initiative. The group set out five requirements:

    1. Mainframe programmers would not need to learn a new programming language like Java to create Web services.

    2. The Web services development environment, where Web services would be built, analyzed and tested, would not require the purchase of software tools that had to be loaded onto powerful and expensive workstations. Instead, the group wanted development tools that could be accessed via a Web browser.

    3. There had to be a centralized directory for listing Web services, so that programmers could reuse and combine services that had been developed.

    4. Any Web service developed had to both maintain the security features that were already built into a mainframe application, and incorporate new Web security standards such as Web Services Policy Framework (WS-Policy) for areas including encryption, authentication and authorization.

    5. The Web services architecture had to incorporate emerging Web services standards to allow future advancements.

    Crew brought together a team that included programmers with experience in mainframe, Java and .NET environments to create the overall architecture for the Web services from scratch, as well as the development environment where Web services would be built, analyzed and tested.

    The initiative was dubbed X4ML for "XML for Modernizing Legacy." Venkat Pillay, a database administrator who learned Java for the effort, says a large portion of the development time was spent strategizing what the architecture would look like. Of vital importance was that the platform not require changing application code on the mainframe or impede its performance in any way.

    Those applications were tried and tested, and tinkering with mainframe applications can be like opening a can of worms, because of their complexity and the likelihood that the programmer and the knowledge behind the code are no longer around.

    Story Guide:

    Merrill Lynch & Co.: Web Services, Millions of Transactions;, All Good

  • Unlocking Access to Customer-Service Functions
  • Despite Complexity, Merrill Skipped Middleware
  • Mainframe Remains Strategic
  • Base Case: Merrill Lynch By the Numbers

    Next page: Despite Complexity, Merrill Skipped Middleware


    The team considered using a middleware server for the key job of translating requests in one language, such as Java, into instructions that could be understood by the mainframe, but opted to implement the translation piece directly on the mainframe. The reason: speed and simplicity. Using fewer moving parts—by eliminating the server—means higher reliability and writing the translation code in Assembler, a programming language for the mainframe, delivering higher performance than a Java parser.

    The X4ML development environment where Web services are created, analyzed and tested sits on a Java server—not on a mainframe—and was written in Java 2 Enterprise Edition, which was selected because of its ability to work with multiplatform architectures. The Java server can be accessed by any browser-equipped computer, satisfying the requirement to allow Merrill's mainframe programmers to develop Web services without the need for high-end workstations.

    The key thing, Crew says, is that there is no middleware between the requesting application and the mainframe. For example, a Merrill financial adviser may submit a request from his desktop to find all of his clients with shares in ExxonMobil, perhaps due to a sudden dip in the stock. The request is submitted from the adviser's desktop application, and the search is run directly on the mainframe via a SOAP request. SOAP tells the mainframe which operation to perform, and XML does the translating.

    The X4ML Web services development environment is where the magic is created. A mainframe programmer accesses the X4ML tool via his desktop browser. He assigns the new service a name, such as "Client Finder," then tells the tool where to find the application by providing its mainframe listing.

    The programmer then clicks on an Import button, and X4ML learns the program's "signature"—that is, the program's variables and how they are used.

    Next, the programmer uses an analyzer tool to essentially pull the program apart and determine each of its inputs and outputs. By doing this, a programmer can create a Web service tailored to perform all or just parts of an operation contained in a legacy application. For example, a client finder application on the mainframe may produce results that list the name, address, phone number, birthdate and Social Security number of a client.

    However, the programmer may want to produce a Web service that just retrieves name and address but not the other data, due to privacy concerns. Increased security policies, such as requiring authentication or encryption, can also be assigned. Alternatively, the programmer can click on a One Click button to automatically create a Web service based on the signature of the program that was determined by the Import tool.

    Either option creates the Web service and automatically lists the new service in a Universal Description, Discovery and Integration (USSI) directory, where it can be accessed and utilized by other programmers.

    A testing function within the X4ML tool also lets the programmer test the operation and evaluate errors before it is used. Creating new applications from the Web services then becomes a matter of accessing and combining operations in the directory.

    The value of Web services was demonstrated over the last three years when Merrill launched a $1 billion effort to create a new suite of applications for wealth management that are available for use by Merrill's 14,000 financial advisers. The dual-screen desktop essentially provides these advisers with access to research, tools and account information to serve clients.

    The wealth management project involved the creation of a wide range of new programs, in partnership with Thomson Financial, which brought its range of financial data and research services to the table, and Siebel Systems, for its client management expertise. Where possible, Merrill leveraged functions and data already available on the mainframe.

    There was no need to create a complicated system that would allow a financial adviser to search through 1,200 clients to find just the ones owning ExxonMobil shares, when such a search could easily be run on the mainframe and be enacted by a Web service.

    QUESTION: What's the biggest headache your tech team can expect to get from deploying Web services? Let us know at

    Story Guide:

    Merrill Lynch & Co.: Web Services, Millions of Transactions;, All Good

  • Unlocking Access to Customer-Service Functions
  • Despite Complexity, Merrill Skipped Middleware
  • Mainframe Remains Strategic
  • Base Case: Merrill Lynch By the Numbers

    Next page: Mainframe Remains Strategic

    Andrew Brown, Merrill's chief technology architect, says what shouldn't be lost in the excitement around Web services is what the technology means for mainframe applications overall.

    "We are making the mainframe strategic to our future. It's a fundamental decision," he says. "Back in the 1990s, it was trendy to say you were migrating off the mainframe. Now, I couldn't predict when the mainframe will go away. If anything, technologies like X4ML have given it new life."

    In December, Merrill decided to give X4ML a chance to prove itself in the larger marketplace. The technology was sold for an undisclosed sum to SOA Software, a Los Angeles-based vendor of service oriented architecture systems. Four members of the development team—Crew; Michael Card, a database administrator and one of the chief developers of X4ML; Pillay; and Bruno Vitro, a Java expert who was hired to help develop X4ML—have transferred to SOA Software to shepherd the continued development. X4ML has since been renamed Service Oriented Legacy Architecture (SOLA).

    According to Brown, Merrill has a long history of selling off technology that it has developed internally, and that it feels will not threaten its competitive advantage. In return, Brown hopes SOA will be able to further enhance the newly named SOLA, particularly in the area of registry services and by continuing to adopt and implement new security standards.

    "We spend a lot of money on technology—billions of dollars a year," he says. "So, we're always looking for ways we can optimize that investment."

    For now, Crew and the rest of the X4ML team continue to work from Merrill's Hopewell campus, not far from the stately grounds of Princeton University. Looking out his third-floor window, across a still-unfrozen pond toward Merrill's credit card operations, Crew feels a sense of satisfaction and excitement about the future.

    "In many ways, this is the culmination of a life's work," he says. "People like myself have spent their entire careers building mainframe applications, testing and refining them, making them work faster and absolutely reliably. Mainframers see this [Web services] as a way to leverage and build on all of that hard work."

    QUESTION: What's the biggest headache your tech team can expect to get from deploying Web services? Let us know at

    Story Guide:

    Merrill Lynch & Co.: Web Services, Millions of Transactions;, All Good

  • Unlocking Access to Customer-Service Functions
  • Despite Complexity, Merrill Skipped Middleware
  • Mainframe Remains Strategic
  • Base Case: Merrill Lynch By the Numbers

    Next page: Base Case: Merrill Lynch By the Numbers

    : Merrill Lynch By the Numbers">

    Merrill Lynch & Co. Base Case

    Headquarters: 4 World Financial Center, New York, NY 10080
    Phone: (212) 449-1000
    Business: Provides an array of investment banking, financing, wealth management, insurance and related services on a global basis.
    Chief Technology Architect: Andrew Brown
    Financials in 2005: $47.8 billion in revenue, $5.2 billion in profits.
    Challenge: Leverage the company's heavy investment in mainframe applications and hardware by making functions in legacy applications available as Web services to other applications.
  • Increase profit margin from 17% in 2001 to 28.5% in 2005.
  • Use Web services technology to cut the cost of developing new applications by as much as 90%.
  • Invest $1 billion over five years to develop and implement a new wealth management platform for financial advisers, supporting about 14,000 desktops.

    How It Works: Modernizing Merrill Lynch
    Merrill Lynch created a service oriented architecture platform that it named X4ML (XML for Modernizing Legacy) to leverage its heavy investment in mainframe applications and hardware. X4ML enables mainframe legacy programs and the functions contained within them to be exposed as Web services. Here is the process that Merrill Lynch uses:
    A mainframe programmer identifies a function in a legacy application that could perform an operation for a new application under development or that could be leveraged by other applications as a Web service. Such functions might include name or account searches, credit checks or commission calculations. Once a function(s) within an existing legacy program is identified, it can quickly be exposed using the X4ML environment, without the programmer having specialized knowledge of Web services, Java or eXtensible Markup Language (XML).
    The next step is to import the program into the X4ML development environment. The programmer goes to an internal Web site used by Merrill Lynch programmers, and assigns the program a service name. This name is how the program will be known to the distributed environments. Once the program is imported, X4ML learns the program's "signature"—that is, it knows what variables constitute the program's application program interface (API) and how those variables are used.
    In the case of programs running on IBM's z/OS operating system and displayed on 3270 terminals, this process involves running the 3270 application within the X4ML tool to teach the tool how to navigate the application's screens. X4ML learns how the various fields on the screens are used—which fields are input, which are output, which ones are used for displaying error messages, etc.
    Next up is creating one or more operations based on the functions contained within the program. There are many options for tailoring a Web service, such as exposing the program's variables by using different names than those originally chosen by the programmer. For example, WS-P-CLI, the legacy name of a field, could be exposed as "PrimaryClient" to make the Web service for searching client names more intuitive to use. Additionally, program outputs can be excluded if desired, defaults assigned to inputs, etc. If the program has only one function to expose and the program's variable names are acceptable, the entire process can be completed with a single click of a button.
    All Web services created through this process are automatically registered in X4ML's Universal Description, Discovery and Integration (UDDI) directory, a Web-based directory that enables software to automatically discover Web services and interact with them.
    X4ML's run-time environment is contained completely within the mainframe environment. Simple Object Access Protocol (SOAP) requests, a means for a program running in one kind of operating systems such as Windows 2000 to communicate with mainframe or other programs, arrive at the mainframe via HyperText Transfer Protocol (HTTP) or IBM's MQSeries messaging middleware. These requests are received by X4ML, which parses them and invokes the appropriate legacy program on their behalf. Once the legacy program is completed, X4ML will format the output as a SOAP response and deliver it back to the requestor.
    X4ML incorporates Web services security standards, including Web Services Security (WS-Security), Secure Sockets Layer (SSL) and custom security rules managed via the Web Services Policy Framework (WS-Policy).

    It's unusual for a large mainframe-centric company to go it alone, as Merrill Lynch did, and build its own Web services development platform, says Michael Kuhbock, chairman of the Integration Consortium, a non-profit industry group based in Calgary, Alberta, that is coordinating Web services development efforts.
    It is usually a safer bet to outsource the project to a partner with experience in integrating mainframes with modern applications and to work with an established Web services platform software vendor. Developing the software in-house could be expensive and possibly delay the implementation of new applications based on Web services.
    But for companies that can't find what they're looking for out on the market and don't want to take the risk of developing Web services on their own, there's another option—a hybrid approach.
    With a hybrid approach, a company works with an established vendor but still utilizes its in-house expertise, particularly on the mainframe. Bank of America, for one, is involved in a massive Web services development project using this strategy. The bank has teamed its in-house staff with several outsourcing firms and vendors, including IBM and webMethods, to implement a service oriented architecture.
    But no matter what approach a mainframe-centric company takes, the key challenge remains the same—getting the mainframe side of the organization onboard with the distributed side, or with programmers developing applications in newer technologies like Java and .NET. "Web services can be viewed as friend or foe," Kuhbock says. "The challenge for the [technology executive] is in getting all of the stakeholders onboard and helping them understand how they can both benefit." —M.D.

    QUESTION: What's the biggest headache your tech team can expect to get from deploying Web services? Let us know at

    Story Guide:

    Merrill Lynch & Co.: Web Services, Millions of Transactions;, All Good

  • Unlocking Access to Customer-Service Functions
  • Despite Complexity, Merrill Skipped Middleware
  • Mainframe Remains Strategic
  • Base Case: Merrill Lynch By the Numbers