Sawtooth Software: The Survey Software of Choice

Rich Johnson Honored in Latest JMR Issue

The latest issue of Journal of Marketing Research (JMR, August 2005) honored our chairman and founder, Rich Johnson, through a series of articles. Rich was invited to write his own short history, covering his early schooling and training, and subsequent career successes, “A Career between Theory and Practice.” Joel Huber of Duke University wrote a short article entitled, “What Has Marketing Learned from Richard Johnson?” Paul Green wrote a short tribute: “Theory and Practice Go Hand in Hand: A Tribute to Richard Johnson’s Contributions to Marketing Research Methodology.” Finally, Rich wrote a short piece encouraging greater cooperation between the academic and practitioner community, entitled “In Favor of Closer Ties.”

With permission from JMR’s editor, this article is entirely composed of highlights from these articles. Interested readers may refer to the full texts from the August 2005 issue of JMR.

Richard M. Johnson: I have had an interesting and rewarding career in marketing research that has been somewhere on the boundary between the academic and the practical. A lesson from my experience is that young people should keep their minds open about career choices and avoid foreclosing any possible paths of development. They may not learn until later in life what their most satisfying calling is.

In the fourth grade, I encountered long division, which was taught as a rote procedure that consisted of repetitive steps with no apparent rhyme or reason to them. I was completely baffled. In later years, I realized that though I am reasonably good at figuring out what to do if I understand the underlying principles, I am no good at memorizing what appear to be arbitrary facts or procedures. I recall that I was convinced that I should henceforth avoid all unnecessary contact with numbers.

I pursued a premedical program (at Haverford College) with a major in psychology. In my senior year, I took freshman math, which was a stunning experience. The course, which Cletus Oakley taught, was different from anything I had ever encountered. The basic principles were clear. The details followed naturally from the principles. I realized that I had misperceived the basic nature of mathematics, and I had come close to missing out on a subject for which I had a deep appreciation.

A few months later, I was accepted at medical school, but I decided to pursue a graduate education in psychology. In 1956, I entered the doctoral program at the University of Washington. I was attracted to the psychometric program offered by Paul Horst. Horst had been a student of Thurstone. My most useful graduate instruction was a series of courses that Horst taught in matrix algebra: these courses provided me some of the most valuable tools that I have used throughout my career.

Graduate students had no access to computers, but I became adept at using hand-operated mechanical calculating machines. For a summer project, I used such a machine to compute a 30 x 30 correlation matrix for an assortment of variables on patients in a mental hospital, and then I conducted a factor analysis of those correlations. Today, a personal computer could perform these calculations in a few seconds, but it took hundreds of hours of manual labor. Although this was tedious, it gave me an understanding of the basic processes involved in factor analysis, and it proved useful in later years.

One career path for students of psychometrics was the development and validation of tests to aid in the selection of employees for particular types of industrial jobs. Horst suggested to his former colleagues at P&G that they take me on as a personnel psychologist.

I arrived at P&G at about the same time as the first large-scale computer, an IBM 705. The computer was intended for accounting work rather than for psychometric research. However, there was hope that it might be available occasionally for statistical analyses, and several employees with statistical training became involved in trying to harness its capabilities. Among my first activities at P&G was the design of programs for regressions analysis, factor analysis, and other statistical procedures that could be run on the IBM 705.

The early 1960s were an exciting time because the rapid expansion of computing capabilities made many things possible that had previously been unimaginable. I worked on many interesting problems in personnel psychology, but marketing research soon captured my interest. Now, I was able to use multivariate techniques such as perceptual mapping and cluster analysis with moderately large numbers of variables.

Although I treasured my relationships with many excellent colleagues at P&G, I became frustrated with the sheer inertia of such a massive organization. I decided to try my hand at consulting, and in the late 1960s, I resigned from P&G. I found a firm that could provide computer access, and I busied myself as an applied statistician, computer programmer, and model developer. One of my clients was Chicago-based Market Facts, Inc. (MFI). My work with MFI expanded so rapidly that the company believed that it was uneconomic to continue the trend. I began a period of employment with MFI that would include the most productive years of my professional life.

One of the problems that interested me was how to make use of product-ratings data to produce maps of product categories that described not only customer perceptions of products but also densities of customer ideal points in the same space. MFI conducted dozens of large projects using this approach in a wide variety of product categories, and there were many instances of repeat business from the same clients.

Another thread of development work led eventually to the conjoint method known as adaptive conjoint analysis (ACA). Market Facts had a client in a durable goods business. Whenever this company seriously contemplated a new or modified product, a concept test was performed. The client was responsible for conducting the concept tests and answered to a product manager who commissioned them. However, before the client could report the results, the product manager would say, “Sorry, we did not have time to tell you about this; instead of two handles, it is going to have one, and instead of 20 units per minute, it will produce 22. Can you test that one in the next three weeks?”

The client found there was never time to do the required concept tests quickly enough to affect the product design cycle, and thus the client came to MFI with what was considered an urgent problem—that is, the need to find a way to test all future product modifications at once. The client wanted to be able to tell the product manager, “Oh you say it is going to have one handle, with 22 units per minute, weight 30 pounds, and be green?” Well, that product would receive X share points. Any other questions?” Today, this is recognizable as a conjoint analysis problem, but Green and Rao had not yet published their historic article (1971). An answer to this problem is to consider a product a collection of separate attributes, each with a specified level. A new method of questioning was required to elicit information about values of attribute levels, and a new estimation procedure was required for converting that information into “utilities.” The solution came to be known as “trade-off analysis.”

To collect data, respondents were presented with several empty tables, each crossing the levels of two attributes, and they were asked to rank the cells in each table in terms of preference. To estimate what are now called partworths, a nonmetric regression algorithm was used to find a set of values for each respondent. Although much was learned about how to improve the technique for future applications, this first study, which was conducted in 1970, was a success.

In the early days, there was less communication between practitioners and academics than that which is enjoyed today. My early work at MFI was done almost in a vacuum and without the knowledge that a similar stream of development was taking place with Paul Green and his colleagues. Green and Srinivasan called such procedures conjoint analysis, and as time passed, it became clear that trade-off analysis was just a different variety of conjoint analysis.

When time sharing and CRT (cathode ray tube) terminals first became available, I was excited about the possibility of using them to enhance the quality of market research interviews. I still remember an experience at MFI when I arranged a meeting with the company’s management to demonstrate the radical idea of computer-assisted interviewing. I had borrowed the most cutting-edge CRT terminal of the time, which consisted of a tiny three-inch screen in a large cabinet. I had shrouded the CRT with a cloth so I could introduce the idea of computer-assisted interviewing without distraction. The meeting went well until the unveiling, when, with a flourish, I removed the cloth to reveal the CRT. When management saw the tiny screen in the enormous cabinet, everyone in the room began to laugh, and they continued laughing until I ended the meeting. Fortunately, CRT terminals improved rapidly, and it was not long before computer-assisted interviewing became entirely feasible.

By the mid-1970s, computer technology had advanced sufficiently that it became feasible to conduct computer-assisted trade-off analysis using pairwise questioning. The questioning format was dramatically easier for respondents than filling out trade-off matrices. A U.S. military service branch commissioned a project to study various recruiting incentives. We purchased what was then described as a “minicomputer”; it filled only a small room rather than a large one. Respondents used CRT terminals at interviewing sites around the United States that were connected to a central computer by telephone lines. The data turned out to be of high quality, and the study was judged to be a complete success. That study marked the end for the trade-off matrix.

After ten years with MFI, I left the firm in 1978. Curt Jones and I founded the John Morton Company. Our plan was to apply emerging analytic techniques in a strategic marketing consulting practice. We purchased several dozen Apple computers, and this began a fascinating adventure of using them all over the world in many languages and in product categories of almost every description.

In 1982, I retired from the John Morton Company, and moved with my wife Judy to Sun Valley, Idaho. Judy and I formed Sawtooth Software, a company named after the nearby Sawtooth mountains, to provide personal computer software to the market research industry and to make available some of the analytic techniques I had developed or used in earlier years.

Sawtooth Software’s first product was Ci2, which permitted researchers with no previous programming experience to compose and administer computer-based interviews. The second product was ACA.

In the early 1990s, it seemed that choice studies were likely to become more popular. Sawtooth Software was able to offer its CBC System for “choice-based conjoint” analysis. In 1995, Sawtooth moved to Sequim, Wash., which offered employees a comparable quality of life while avoiding the increasing cost of living in Sun Valley.

I believe that I have had a singularly satisfying career, though completely different from what I might have expected as a youth. I have worked as an applied mathematician, yet throughout my early schooling I was terrified of mathematics. I have flirted several times with the idea of trying academic life, but on each occasion, I have decided to remain a practitioner. I will probably always wonder whether I could have succeeded as a professor and whether such a path would have been as interesting as the one I pursued.

What success I have had has been due to my being positioned between theory and practice. Not being in an academic setting and being unaware of much prior academic work, I have had to invent my own approaches, many of which had already been invented by others. In retrospect, I might have accomplished more if I had been more of a scholar and less of an inventor, but that would not have been as much fun!

Joel Huber: Thanks to the efforts of Richard Johnson, the way marketing research is thought about and done has changed. Rich has made contributions in basic theory and applied knowledge, but most important, he has been a critical catalyst in bringing so many ideas into general use in the market research community. Sawtooth’s offering for choice-based conjoint (CBC) analysis made choice experiments accessible in ways that were possible only with state-of-the-art customized procedures. Perhaps the cleverest aspect of that product was the way it finessed the issue of choice design by selecting random but balanced choice sets for each individual. The net result is a program that is classic Johnson: It is efficient, robust, and perhaps most important, accessible to a broad range of researchers.

The other element that made the CBC system particularly valuable to the marketing research community was the availability of hierarchical Bayesian analysis. Richard Johnson had no formal training in Bayesian estimation, but he gained expertise on the procedure from pioneer researchers Greg Allenby and Peter Lenk. Richard Johnson took one particular Bayesian model and developed it to estimate the results of CBC. By limiting its applicability, he was able to increase the speed of convergence by an order of magnitude. In addition, the Sawtooth community performed many simulations and tests that have enabled the development of standards and procedures that transformed Bayesian estimation from a black art available only to the tutored few to a standard technique available to all.

Part of what is different about Rich is his fearlessness. The other part comes from his strategy of having a solid base that supports his risk taking. This support base includes both consulting clients and Rich’s intensely loyal and talented coworkers. It also comes from a group of academic researchers that include John Hauser, Dick Wittink, Peter Lenk, Greg Allenby, Paul Green and myself.

Perhaps the most critical supportive base is that of Sawtooth Software and its loyal users. Their bonding and mutual support play out in meetings and on their Web site. The open and sharing culture of that organization also contributes to its effectiveness. Sawtooth’s culture is unlike that of many market research companies, which are understandably reluctant to limit returns from their investments in methodology by making it easy for others to copy or criticize. Using a different, more open model, Sawtooth has become the dominant worldwide supplier of conjoint software to marketing researchers. The Sawtooth Software meetings share this transparency by inviting as speakers competitors or people with findings that are unfavorable to Sawtooth.

Paul Green: One of the most interesting features of Rich’s work is his ability to find new ways to solve “old” problems, whether they are in multidimensional scaling, conjoint analysis, or other areas of interest. I first noted examples of this skill in his early work in perceptual mapping and clustering. There are often twists that come from his fertile mind and have pragmatic value for applications. With his widely diverse skills and interests, it was only natural that he put these talents to work. Not content with his already important contributions to psychometrics, Rich and his associates started Sawtooth Software, a unique firm that specializes in psychometrics and related marketing research tools.

Rich remains an active and novel thinker and doer. (It would be difficult to imagine him as placid and content the role of country squire.) Rich is a rare blend of scientist, entrepreneur, risk taker, and sportsman. Rich and I are also both amateur piano players. Unfortunately, he is too shy about this nonrugged talent. P.S. I’m still waiting for a piano duet.

Richard M. Johnson: I would like to thank Paul Green and Joel Huber for their generous comments. There are no colleagues whose approval I value more.

I have heard academic colleagues describe particularly brilliant students, and when I have proposed that these students consider careers in applied research, my academic friends have shuddered at the thought. They believe that such students would be wasted in applied settings. Paul Green’s productivity is admirable, and I believe that he was aided by previous experience in an applied setting. Joel Huber observes that the academic side is becoming more specialized, and the practitioner side is becoming more proprietary. That presents a problem for both sides, and I believe something should be done about it.

I would like to encourage my academic colleagues to suggest that their best students plan to spend at least some time as practitioners. I believe that experience would give these students valuable perspectives and inventories of methodological challenges that could guide them in future academic work. I would also suggest that qualified practitioners be encouraged to spend parts of their careers in academe. This should not only provide them with increased knowledge but also build relationships from which both sides could benefit in the future. If such opportunities had existed years ago, I would surely have been interested in them, and I believe that these different kinds of experiences would have made me more productive.