At the 2007 Sawtooth Software Conference, Well Howell (Harris Interactive) presented a paper detailing a comparison of software to perform HB estimation, including our own CBC/HB v4. While we were quite pleased that CBC/HB had the fastest runtime (Well, your check is in the mail), we are always looking to further improve performance.
In the last two years the computing industry has seen the introduction of 64-bit (x64) desktop computing. AMD began releasing processors supporting x64 in 2003, with Intel following suit in 2004. Over the last few years more chip lines have begun to support x64, and currently all new processors support x64.
While these new processors run 32-bit Windows as well, Microsoft released an x64 version of Windows XP back in April 2005. Consumers could order a new PC with it if they requested it at purchase time, but it wasn’t available in retail outlets and thus the x64 market didn’t grow very fast. In January 2007, Microsoft released Windows Vista which gave consumers two discs for 32-bit and x64, allowing users to choose which to install. Since that time, the number of users running under x64 has steadily increased.
We’ve often been asked if an x64 version of HB is available. Currently CBC/HB v4 is compiled as a 32-bit application. It will run just fine under x64 Windows. The question becomes whether an x64 version of HB would perform better than the 32-bit version. This applies only to x64 Windows since x64 HB will not run on 32-bit Windows.
For the truly geek at heart, let me describe the hardware used for our x64 experiment. We used a Dell Precision 490 with an Intel Xeon Quad-Core processor running at 1.86GHz, two 80 GB 10K RPM drives using RAID 0, running Windows Vista Business x64. While the processor has 4 cores, CBC/HB is a single-thread application (we are also researching multi-threading). We performed utility estimation initially with 2GB of RAM, which would be fairly typical. However, x64 bit Windows allows the ability to use more than 3GB, so we upgraded the system to 6GB and reran the computations to see what could be gained by having more memory.
Now I’ll describe the experiment. We selected a dataset with 1019 respondents who answered 24 choice tasks each regarding 5 attributes involving 3-15 levels each. Under main-effects (base case), CBC/HB would estimate 34 parameters for each respondent. We would consider this a moderately sized dataset and fairly common for CBC/HB. Our performance measure is the iteration rate, defined as seconds per iteration. We compared the 32-bit iteration rate to our x64 test engine, which I will describe next.
The CBC/HB v4 engine is a 32-bit component written in the programming language C++. To create the x64 version, we made no changes to the existing code and simply compiled it as x64. While not part of the experiment, we tested the engine to make sure that conversion to x64 did not lead to different utilities from the 32-bit version. In a side-by-side test given the same random seed, both produced identical results. Because we did not need to change the code, we are confident that the results reflect only the difference between 32-bit and x64.
Using CBC/HB v4, we created the base case along with five variations shown in the table below. To run the estimation, we created a special application (using C#, a .NET language) that would run both the 32-bit and x64 engines, to further ensure there was no bias towards one or the other. We then performed each utility estimation run in 32-bit and x64. The results are listed below:
In each case, we noted how many iterations we ran. Although the utilities had not likely converged in most cases, the iteration rate was stable enough to estimate performance, which was the target of the experiment. We also note the build file size, which relates to whether the data files would reside in memory or were too big and were used off the disk (which is also noted).
In the first three utility runs, we used the original 1019 respondents and increased the number of parameters by adding interaction terms. In the next three cases, we repeated the first three runs, but we replicated each respondent 10 times, resulting in 10190 respondents. In all cases, we note the percent less time that the x64 version required to perform iterations.
The first thing to note is that the x64 version was faster than the 32-bit version in every case. The gains in the 2GB block appear to be more pronounced (with one exception) than with 6GB, but looking carefully we see that the 32-bit rates increase significantly more between 2GB and 6GB than the x64 rates do. This is likely related to the way that Windows manages its own resources. Under 6GB it appears to be able to manage 32-bit applications much more efficiently than it did with 2GB.
The exception occurred with the last case under 2GB. This is an extreme case where the build file size actually exceeded the RAM by over 1.5x. In this case, the machine spent most of its time in resource management (called thrashing), and so the iteration rate was very slow in both 32-bit and x64.
In both the 2GB and 6GB blocks, the last two cases ran the build file from disk instead of memory, although it would appear that (except the previously mentioned case) they should run in memory. While there might have been enough memory to do so, Windows rejected the request to run them in memory. Unfortunately, that can’t be changed, but the operating system was able to cache parts of the files in memory, and under 6GB it was able to do enough of that to make the iteration rates reasonable.
Probably the most encouraging aspect of this experiment is that you can expect a significant benefit running an x64 bit HB on an x64 machine with generous amounts of RAM. We expect an x64 version of CBC/HB sometime in 2008, and we hope to increase performance in other ways as well.