Go back to ETrade UK

ETrade Beats the Clock

ETrade Financial Corp. and other online stock brokerages are at war over how fast they can execute a trade. Ameritrade Holding Corp. kicked off the execution promise game in 2001 by guaranteeing a turnaround of 10 seconds or less. ETrade countered with a nine-second guarantee, and since then it has all been downhill.

Last year, Ameritrade promised to complete trades within five seconds, and in March, ETrade lowered its pledge to two seconds.

ETrade declines to say how many of its trades fail to complete in two seconds, but a spokeswoman says it will forgo less than $1 million in trading commissions this year as a result of the guarantee. The company collected $191 million in commissions in the first six months of this year.

ETrade has 3.5 million customer accounts and completes more than 100,000 trades each day, on average. The company says most of its trades are completed in less than a second, a remarkable achievement for a complex operation that spans multiple computers, routers and applications, not all of them controlled by ETrade.

"ETrade has a sophisticated infrastructure," says Tim Carpenter, senior brokerage analyst at GomezPro, an Internet performance management company recently acquired by Watchfire Corp. inWaltham, Mass. "They are not bogged down by trying to integrate with a legacy infrastructure, like an older broker that moved from the off-line world to the online world. They didn't have to make concessions."

Excess Capacity Cheap

The company has gone through several major IT overhauls since 1996, when it began Internet-based trading. The most important transition, one that is still going on, has been a cutover from proprietary products to open-source components -- principally Apache Web server software, Tomcat application server software and the Linux operating system (see diagram). The move from Sun Solaris/Sparc to Linux/Intel at the top two layers of the three-tiered architecture, in particular, made for a whole new economic ballgame for ETrade.

In 2001, the cost of proprietary systems was rising while the cost of open systems running on Intel processors was falling, says Joshua S. Levine, ETrade's chief technology and administrative officer. "And it wasn't just the price of the chip; it was memory, peripherals, everything. We said, 'If we stay on a proprietary architecture, we'll always be trapped by the rising price of proprietary vendor equipment.'"

With the savings from open-source products came the luxury of being able to overprovision by buying spare capacity for every conceivable hardware failure and spike in demand. "When you are buying servers so cheaply, the concept of 'capacity' drops away," Levine says. "We buy a machine for $3,900, and we don't even buy maintenance on it. When it fails, we throw it away."

ETrade's three-tiered architecture includes a server farm, which it continues to build out horizontally, with 700 Linux/Intel boxes running most of the time at some small fraction of their capacity. Making that work, and making it easy to add more boxes, requires software that can move the workload around and the ability to swap out failed servers, in real time. ETrade uses load-balancing software from Resonate Inc. but is migrating to a similar tool from NetScaler Inc. that it says is faster.

Those products load-balance within an application, so if trade routing needs more horsepower or encounters a hardware failure, for example, that's addressed automatically and immediately. ETrade is now looking at ways to create a virtualized server farm for use by multiple applications. "How do you jump-start or repurpose a server?" says Hartley Caldwell, vice president of re-engineering. "We are thinking about that in the future."

The Right Routes

An ETrade customer may connect to the Internet over one path. But ETrade then finds the best route back to the customer using Adaptive Networking Software from San Mateo, Calif.-based RouteScience Technologies Inc., which can adjust traffic in subseconds to meet user-specified policies about application availability, performance and network usage. Route adjustments are made using open-standard application programming interfaces on routers and other network devices.

"It's a common myth that you can't manage the Internet," says Lloyd Taylor, technology vice president at Keynote Systems Corp. "Using the proper measurements, and understanding how routing works and working on your providers, you can actually control a good portion of the performance."

The move to Linux/Intel servers reduced trade times 30% and enabled the nine-second trade guarantee. Getting down to two seconds required other moves, including bringing ETrade's system for routing -- moving trade orders to the proper exchange and bringing back confirmations -- in-house.

ETrade was using a third-party service, but it saw that it could improve reliability and speed by developing its own routing software. Its internal system, RoutX, eliminated route hops and optimized the software, Levine says.

Extreme Measures

Gomez Inc. and Keynote both monitor ETrade Internet activity continuously. Keynote sends test transaction suites every few minutes from each of 20 locations around the U.S. and reports the results to ETrade every 15 minutes.

Says Caldwell, "Every morning we review the previous day's operations and the coming day's operations with the operational staff around the world. What was the customer's experience, how many million hits did we have, how many Web errors did we have, were they customer Web errors, did our load-balancing solution remove them in time, and so on."

Greg Framke, an executive vice president and head of IT at ETrade, says, "We have developed a rigor around performance and availability that has really made a big difference. Measuring reliability and performance is almost a cult with us."

At the middle layer of the three-tiered trading system, ETrade runs BEA Systems Inc.'s Tuxedo transaction manager, and that could be the next target for the company's move to open-source, Levine says. "We are very interested in Java Message Service -- JMS," he says.

The move to JMS would be straightforward, and its benefits would be primarily economic, according to Caldwell. "Tuxedo is fairly expensive on a per-CPU basis," he explains.

While the top two tiers of the trading and routing systems run on Linux, the Sybase database layer remains on Sun Sparc systems running Solaris. A move to Sybase on Linux is possible, but that would require a significant system redesign and have an uncertain operational impact, Caldwell says.

Rather than making an intermediate move to Sybase on Linux, Framke says ETrade could go to a grid database concept in which data is cached at various points up and down the application stack. "If I could put data closer to the user, I could serve it up faster," he says.

That could improve availability as well. "If you look at the way we have architected our front end, it is rare -- a very rare event -- to have a user who couldn't on the second click get to where they were going," Framke adds. "And that's what I want in the database."

Watchfire's Carpenter says that many ETrade customers are not concerned about a few seconds of trade execution time. But for "active traders" -- those making hundreds or thousands of trades a year -- "execution quality and speed are of the utmost importance."

Says Levine, "What I'm most proud of is that we operate at 100% every day, 24 hours a day. And it's out there for everyone to see if you have a failure. I mean, my career could end right now with a three-hour outage."

 Also see E*Trade Financial Corporation

 Even more on ETrade including History and Locations

 Meet the staff at ETrade UK