Joshua Levine retired last year from his job as chief technology and operations officer of E*Trade. Before joining the online brokerage, he was the managing director and global head of equity technology at Deutsche Bank and, before that, a managing director at Morgan Stanley, where he served in a number of technology management roles.
Today, Levine is the managing director of Kita Capital Management, an investment management firm; sits on the boards of several technology companies, including Securify, a computer monitoring and security company; and is a member of the National Counterterrorism Center’s Industry Advisory Board.
Levine spoke with Baseline editor-in-chief John McCormick on Aug. 29 about how he handled computer security at E*Trade, which, during Levine’s six-year tenure as technology chief, was never disrupted by a computer security problem.
The morning they spoke, news broke that AT&T’s computer systems had been hacked.
It’s been an interesting morning with the hack into AT&T. When you look at E*Trade, that was a kind of online store—it was the company’s online face to its customers.
It was the business. There was no way you would have lax controls.
Everyone can talk about franchise risk when it comes to theft, but hacking a financial Web site is true franchise risk. If E*Trade or ING Direct or Ameritrade or any one of the online financial services companies were ever to be hacked, I think that would be sector risk as well. Everyone in the sector would be hurt.
When you were at E*Trade, there were no security breaches.
That’s right. [And] there’ve been no breaches.
Security in general is all about something you know, something that you have, something that you are.
Straightforward, it’s really cost that holds everybody back from doing the real thing. You balance the fraud versus the cost. [Even] if you know for a fact that every ATM, every credit card, everything could be tight, really tightly secured—[a company] is not going to do it until the cost of fraud exceeds the cost of the solutions.
What percentage of E*Trade’s budget was spent on security when you were there?
If you define security broadly, in terms of policy, engineering and operations—the three real components of security—we probably spent 2% to 2_ % of the I.T. budget.
And what was the budget—ballpark?
$300 million.
It’s always interesting when people talk about the return on their security technology investments. When you were at E*Trade, the company started to offer customers handheld security tokens—key-chain-size devices that display a new individual password every minute or so. Customers with big accounts were given the tokens at no charge, but customers with small accounts who wanted the devices were charged a couple of bucks. So, there were probably some discussions over who received the tokens for free and who had to pay. I guess that all gets weighed into some kind of ROI calculation.
Absolutely.
You know, the tokens are not to prevent a loss of personal information; they’re really [to prevent] the loss of financial assets—someone who buys stocks or wire-transfers money and so on. So, if you don’t really have any money, it’s probably not a great concern.
[And] the tokens were so low [priced] . . . . Would you pay $12 for the added benefit that no one could ever hack into your account through a key logger? I would.
And I say this to everyone: How could you do business without tokens?
The tokens illustrate the point that while you were at E*Trade, the company seemed to be on the leading edge of security technology.
We felt if we were going to present this space of online stock trading, a fundamental premise [would be] that you could put your money there.
In 1999 and beyond [our philosophy was], “This is the way everybody is going to do things.” [Security technology] became even more important. [But in 1999–2000, we were hit with a] denial-of-service attack. And while we felt that we could protect the perimeter, it just didn’t dawn on us that someone would try to bombard us with [this kind of] attack.
What do you do about denial-of-service attacks?
The big problem is that they become dangling sessions. What happens is, you open a session and the way TCP/IP is architected, it takes six minutes for that session to break. So, let’s say on one box you get 15,000 connections . . . and they don’t do anything. It takes six minutes for that box to be released. And if you did that across the board, you would shut the service down.
But plenty of solutions came out for that—to break the sessions. So now, it’s much more sophisticated. Now [systems] interrogate the packets to decide [if it’s] one of our “gets” or “puts.” [If it’s not, it] breaks the session.
What were the two or three biggest security worries you had while you were at E*Trade?
That we overlooked something . . . or some kind of carelessness that we didn’t have a verification procedure for. But [those were] the early days.
[W}e were moving so quickly that we were very cognizant [that] we could make a mistake, and that would be a disaster. But over time, we built enough verification procedures that we never had any issues. Plus we had monitoring sites outside—robots essentially. We used different vendors, like Keynote, and other monitoring services. And we would watch the alerts in real time. So, if we do a change and someone—one of these robots—immediately starts seeing poor response time, that sets off an alarm. And everybody would rush to the occasion.
But that would be the only thing I lost sleep over—that we overlooked something.
What did you do to cover yourself—to make sure you didn’t overlook something? Were there a lot of sessions in which you just thought about the worst case? Did you run disaster scenarios?
We did a lot of that. We also phased everything in. We didn’t do any gigantic “let’s flip the switch” changes. And we mitigated our risks—constantly.
And you can’t manage what you don’t measure. We constantly measured everything from the external customer standpoint. It used to be that we would actually wait until customers called in before we knew there was an external problem. Things could look like they’re working great—you may say, “Well, it doesn’t look like we have a big user load today,” or what have you. But if you don’t measure everything, you really can’t tell what’s happening.
[In] a dashboard that we ended up creating, you can see in real time the number of transactions that are occurring, the response time customer are getting, and other measures, market measures. So, you can line everything up and you can see if you’re having a problem.
And if you set those baselines, you can easily tell if you’re having a denial-of-service attack or other event.
Exactly. And with all the monitoring sites, you know what the customer is seeing. You don’t have to wait for a customer to call. Because within milliseconds of any problem occurring, you’re going to detect it.
From your years of experience, what advice would you offer to other CIOs and CSOs?
I think one of the nicest things we did, organizationally, [was] splitting things up [into policy, engineering and operations]. It put some checks and balances into the equation.
What we found was all the network engineers were moonlighting as security engineers. And all our network operators were moonlighting as security operators.
And what we found was that when we actually broke up all functions, we got more efficient and we got more effective. People started to focus on things.
We removed the policy from the engineers, because engineers have all sorts of views on how things should be done and what should be done. But it’s much better if someone else comes up with the view, and engineers have to implement it and operators have to run it. And if you split it up that way, it really works very nicely. And it doesn’t add any overhead. It certainly makes everyone more focused on what they need to be doing.
This business of moonlighting, adjunct to your role, [is] awful. Especially the network folks. They love security. But they don’t get a chance to work on it.
You need people to focus.
Right, their original job still exists.
One of my problems with forensics and the lack of real-time tools and monitoring tools in most companies is that you end up taking your best people and putting them on these forensics. Dropping everything. And it’s debilitating.
For one thing, the engineers and operators love dropping everything and working on these things. On the other hand, they loathe it—they say, “I couldn’t get my work done, it was a total distraction, I got no credit for it.”
So, I think that for companies to really be effective, they really need to split up the functions.
The other problem I always had [is with the role of the] CSO. For example, as long as they have a policy role—and they know that’s what their role is—I find that they’re content.
I think where it breaks down is when they’re standing outside the house just throwing rocks in—[saying] “I wouldn’t do it this way” and so on. I think that’s debilitating. [It’s] very difficult when you have a CSO who sits outside of I.T. and basically just takes potshots.
Any other advice?
Measure everything. Analyze everything. And do it in real time. If you’re not doing it in real time, you’re not doing it.
