The Server CrunchBy David F. Carr | Posted 2007-03-01 Email Print
WEBINAR: Available On-Demand
Innovate and Thrive: How to Compete in the API Economy REGISTER >
Real-world companies such as American Apparel, IBM, Starwood Hotels and Toyota are exploring whether 3D virtual communities can be adapted to serve businessand whether they are an effective place to do market research and collaborate on projects, an
The Server Crunch
The growth of the Second Life virtual universe depends heavily on Linden Lab's server infrastructure and the software that runs within it. The equivalent of the Web server, or Web application server, in this architecture is the simulator. A simulator is a software process that tracks activities within 16 acres of virtual real estate, and Linden typically runs four simulators per server (machines with two dual-core processors, allowing each simulator to have one processor's worth of computing power to itself), according to technology VP Miller.
The simulators work in conjunction with MySQL databases, along with a network storage system that contains some 33 terabytes of images, object geometry files and other data, Miller says. The majority of this gear is hosted by data center operator 365 Main at a facility in San Francisco.
The simulators determine what each avatar should be seeing at any given moment, and transmit instructions and imagery across the Internet to each user's viewer. The simulators also communicate with other simulators that represent neighboring parcels of land, allowing users to see across the border into the next region. In addition, the system incorporates physics simulation software, licensed from Havok.com of Dublin, Ireland, which allows objects to behave as if they have weight, fall to the ground if they are dropped, and bounce back upon collision with other objects.
When things went haywire during the Jan. 9 Town Hall meeting, and flying avatars who tried to touch down in the auditorium found themselves falling through the floor, the likely reason was a communications breakdown between the simulator for that area and the viewers for those users, says Andrew Meadows, a Linden Lab senior developer, who was employee No. 2 at the young company. Normally, the simulator should have detected that an avatar had reached floor level and directed it to stop. But if the message from the simulator fails to reach the user's viewer, perhaps because of an interruption on an overloaded Internet connection, the viewer tries to extrapolate motion along the last known path, sometimes with comic results.
So, strengthening the simulators, and the reliability of their communications with the viewer software, is one of Linden's many technical challenges.
The simulator is also responsible for executing the user-written LSL scripts that make airplanes fly, doors open and close, and all sorts of other objects behave according to their programming. Linden is currently working to exchange its homegrown bytecode interpreter, the software that executes those scripts, for an open-source product called Mono that in tests has been able to run the same scripts 20 to 150 times faster.
The Mono project was started by prominent open-source programmer Miguel de Icaza and is backed by Novell Corp., where he now works as vice president of developer platforms. Its goal is to give Linux and Unix platforms an equivalent to the Microsoft .NET runtime environment, which among other things includes a bytecode interpreter that supports multiple programming languages (in contrast with the Java Virtual Machine, which targets only Java). So, in addition to delivering a performance boost, a Mono-enabled version of Second Life should be able to execute scripts written in programming languages other than LSL, such as Python or Microsoft's C#.
"The choice of Mono was mostly driven by dissatisfaction with our own bytecode interpreter," Meadows says, as well as the fact that it was open source "so we could bend it to our will if we had to."
Linden is also making some more basic tweaks, such as switching to using the Web's HyperText Transfer Protocol (HTTP) for downloads of object geometries and textures (binary images). Linden originally created its own download protocol, which was built on the assumption that geometries would make up the majority of the data transmissions. In fact, texture downloads turned out to have a bigger impact on performance. So, by switching to HTTP, Linden can employ the same kind of caching technologies that are used to speed downloads on the Web, Meadows says.
The actual geometric representations, textures and scripts for each object are recorded as files using storage systems from Isilon Systems of Seattle, according to Linden. Each object is also assigned a unique identification number, and those object IDs are recorded in MySQL, the popular open-source database backed by Swedish company MySQL AB. The database tracks the ownership and position of each object, as well as properties such as whether it can be legally copied, modified or transferred to another resident.
Second Life started out with one central MySQL database, but has gradually moved to a more distributed architecture with, for example, multiple database servers responsible for tracking each resident's possessions, or "inventory."
"Your inventory could be in one database, and your friend's inventory could be in another," Meadows explains. So, if you give your friend something of yours, the system has to delete it from one database and add it to another.
However, there is still one centralized database for tracking user accounts and account balances, and for executing the transactions to transfer Lindens from one account to another. That remains one of the major chokepoints in the system, according to Meadows. The database team wants to break it up into smaller components, so that the workload can be spread across more servers, but they're still figuring out the best way to accomplish that, he says.
"We know this is a problem companies like eBay and Amazon have faced and have actually succeeded in fixing," Meadows says, so Linden should be able to overcome it, too. Although Second Life isn't handling anything like the dollar volume of eBay, the challenge in terms of database transactions may be nearly as great because of the number of micropayments the Second Life system must handle when tens of thousands of residents go shopping. More than 6 billion Lindens changed hands in January, for example. And although some technically knowledgeable residents have argued that Linden should switch to a commercial database such as Oracle, Meadows replies, "MySQL's been working great for us, overall, and at this point the work required to switch would be far more work than it would take for us to just fix our problems."
From a user perspective, database overload can mean that purchases don't go through, the inventory system that tracks each avatar's possessions breaks down, and the search function grinds to a halt. One posting on the official Second Life blog from mid-January even blamed database glitches for a problem that prevented users from teleporting to new locations which is the equivalent of having hyperlinks stop working on the Web.
Second Life residents have also had to tolerate frequent scheduled and unscheduled outages, sometimes several times a week, when Linden takes the system down for maintenance. According to Miller, the firm's technologists recognize that has to stop, particularly now that more organizations are starting to do serious business in Second Life. For example, several universities are starting to teach courses in the virtual world, and they're going to expect the system to be up when it's time for class, he says.
"We're moving rapidly to a model that basically allows us to deploy new code to the grid without bringing it down at all," Miller points out. And that should be good news to the people trying to make a living on Second Life.