Sawtooth Software: The Survey Software of Choice

500,000 CBC Surveys in Less than Three Months

The following story was submitted by Peter Kurz of TNS Infratest. It recounts how TNS Infratest conducted an online social research study in Germany from October to December, 2003. With 500,000 respondents collected using SSI Web, this certainly represents one of the world’s largest conjoint studies.

Objective

The objective of the survey was to provide the German public with a forum to voice their opinions about social and political topics. The survey was a joint initiative by a large German management consulting firm, the internet version of a large-circulation weekly magazine, and one of the largest internet providers in Germany.

The huge number of participants in the study owes to a broad media campaign. Numerous advertisements in print media and the internet advertised the survey as a way for Germans to express their opinion about economic reforms that would address the country’s pressing growth and unemployment problems. The media campaign stressed that the results would be broadly communicated, would be brought to the attention of key decision makers, and would be used to spark public debate. Therefore, we believe, many participants saw the survey as a unique opportunity to express their opinion about the most pressing public policy issues in Germany.

Study Design

The online survey was split into a core block that was presented to all participants and four theme blocks that were randomly assigned to participants. The four theme blocks covered the following areas: (1) work and leisure, (2) education, (3) family, and (4) civil engagement. The random-block design was chosen for two reasons. First, this design reduced average completion time for the online survey. Second, the random-block design reduced the risk of strategic self-selection into blocks that were of special interest to participants.

Technology Challenges and Testing

TNS Infratest faced the problem of conducting a conjoint study with five separate conjoint designs where the main block would have 500,000 respondents and the four theme blocks would have about 125,000 respondents each. In addition, the client wanted assurance that 8,000 interviews could be conducted simultaneously. Unfortunately, we found no one who had ever run such a large online survey and could tell us if the different software modules would handle such large numbers of respondents. It seems that we were pioneers in the field, conducting such a large conjoint study with half a million interviews over a relatively short time.

Our first decision was to split the study into four surveys covering the four theme blocks and also to run the main conjoint as well in four replicated surveys to avoid the risk running into some limitation of the data collection module. Extensive tests with Sawtooth Software’s SSI Web conjoint module for CBC encountered no problems managing such large samples. Also the export function to prepare the CHO files for utility calculation showed no problems with more than 500,000 test datasets.

The second problem was to find out if it was possible to handle 8,000 requested interviews at the same time. The IT department at TNS Infratest determined that a server cluster with six Pentium 2.8 GHz computers in combination with special load balancing software would be able to handle the load of 8,000 simultaneous CBC interviews.

Crossing Our Fingers

In October 2003 fieldwork started, and we weren’t quite sure what would happen over the following weeks—what aspects we forgot to test, what pitfalls were on the way. But it seems that we had good luck and all the hardware and software worked quit well. By the end of December 2003, over 870,000 respondents had started the conjoint in the main block and more than 500,000 had completed the survey.

During this time, we conducted several tests to figure out how many splits and what computers would be needed to run the hierarchical Bayes estimation. We thought the main conjoint block could be an HB run with all 500,000 respondents together. Several tests had shown that Sawtooth Software’s CBC/HB module would have absolutely no problem computing utility values for 500,000 respondents at the same time (from one enormous CHO file). Tests with different computers showed that an AMD Dual Opteron PC with a workspace of 2GB and the beta version of Microsoft Windows XP Professional x64 Edition performed best. The Sawtooth code doesn’t use the multiprocessor capabilities, but enabled us to simultaneously run the four theme block conjoints on one Opteron CPU while the other CPU was running the main conjoint. Due to that parallel processing, the utility scores for the main conjoint and the four theme blocks all finished after 128 hours 13 minutes and 46 seconds (a little over five days). We used 15,000 burn-in iterations, followed by 5,000 “used” draws per respondent, skipping every 10th draw, for a total of 65,000 iterations. Our overall conclusion is that we are satisfied with the software and are very impressed with the stability and capabilities of Sawtooth’s programs, especially the CBC/HB module.

Some of the results were expected—for instance, the large gap in the quality of life between the former East and West Germany—but the survey also uncovered some important new insights. Most striking was the large proportion of respondents willing to reform Germany’s social and political system. For instance, the results suggest that Germans may be more ready than their politicians for economic reforms that could solve some of the country’s economic problems, such as high unemployment and unsustainable pensions. Germans also seem to be more willing to contemplate lower levels of government support than politicians generally acknowledge. These and many other results received considerable media attention, not the least because of the large number online participants.