Building BlocksBy Mel Duvall | Posted 2003-02-01 Email Print
Their market research system mines the collective thoughts of netizens. But can the Rochester, NY company accurately interpret its polls to represent the world at large?
There were five major pieces to the technology infrastructure that had to be built:
The first was a high-speed, customized e-mail system that could create, format and deliver more than 1 million e-mails an hour. E-mails are sent to pre-registered panel members, inviting them to take part in a survey. Participants are given a unique password and URL, which takes them to an online questionnaire. For this Bayer used a buy-and-build strategy. An SMTP (simple mail-transfer protocol) e-mail engine was licensed from L-Soft International, which then had to be highly customized. Bayer largely relied on internal developers for coding work, although some work was outsourced to Rochester-area developers and at various stages of the development cycle, coding work was outsourced to developers in Romania.
The second major piece was a survey engine, designed to accommodate more than 200,000 custom, five-minute surveys per hour. For this, Bayer partnered with a relatively young company called Survey Craft. A complication arose when Survey Craft was purchased by SPSS in November 1988. However, Milla, who had been hired by Bayer partly because of his past experiences with Survey Craft, was able to negotiate a deal to buy the source code.
Another major piece was a survey dispatch system. The dispatch system coordinates and tracks the movements of respondents, and balances loads across the company's servers. It tracks how long a person takes to fill out a survey, and if someone decides to suspend answers, it can place the person back at the exact question when he or she logs in later. This system was built entirely in-house, on top of a SQL Server database.
A multilanguage polling and registration system also was designed in-house, with coding work outsourced. This system was needed to add new members, and to allow existing panel members to add, delete or update their registration information online. The system can then recognize each panel member's language preference and deliver surveys in that language.
The final, majorand most challengingpiece was a propensity weighting system. This allows Harris to accommodate anomalies in the data. Not only does this take into account the cultural makeup of panel members, but also differences that occur when people fill in surveys online or offline. For example, it has been known for some time that when people answer a telephone survey, they tend to select answers based on the order in which they are read. Propensity weightings have been developed and applied to accommodate those biases. With an online survey, Harris didn't know whether the same biases exist. It also didn't know whether something as simple as the color of the survey's background affects people's answers. Black says close to $6 million was spent on some 400 studies to understand and account for factors that would influence respondents' answers online.
Harris also invested time and money in determining how to attract and keep panel members. The company gained most of its members in the first three years through a partnership with Excite@Home. It has since dropped that partnership and now gets more than half its panel participants through Yahoo.
The company set a goal of achieving at least 2 million panel members by the end of 1999. It achieved that goal, but not without headaches. In the summer of 2000, Harris was placed on the Realtime Blackhole List (RBL) by Redwood City, Calif.-based Mail Abuse Prevention Systems (MAPS), after MAPS claimed that Harris' method of e-mailing panelists was simply a form of spam. As a result, Microsoft, AOL and other Internet providers blocked Harris from e-mailing its members.
"It was a very serious problem for us," says Bayer. "We couldn't reach a large section of our panel, and that meant we couldn't fulfill our contracts." At first Harris fought back by launching a lawsuit against MAPS, but as that battle dragged on, the company negotiated for time to convert its entire registration system to a double opt-in program. In the past, new panel members clicked on a banner that would take them to a registration page. The problem as MAPS saw it, was that it was possible to sign someone up as a panel member without the person's knowledge. With the double opt-in system, new panel members receive an e-mail stating that they have registered to become a member of the Harris survey network, and must then confirm their participation. In the end, Bayer says the company should have used the double opt-in system anyway.
Other technology issues confounded the information technology staff and led to customer defections. For example, the company had difficulties pushing high volumes of e-mail out to panelists. That problem was later attributed to bugs in early versions of the L-Soft e-mail engine and the use of unstable hardware. Harris gradually resolved those conflicts by installing newer releases of the software and upgrading the hardware. Also, the survey engine crashed frequently. Bayer blamed unreliable software largely built from scratch, and unstable "garage-built" hardware. To resolve those problems, the hardware was upgraded to servers from tier-one suppliers such as Compaq. And slowly, but steadily, the software was debugged.