Hypothesis

Our hypothesis is that participants would be able to more consistently identify the location of the Gulf Stream in an image using the Samsel Nested colormap than in other comparison colormaps.

Methodology

For this study, the POP data was used to produce an image of the Gulf Stream, stretching from the east coast of the US to the west coast of Europe/Africa. Four colormaps were compared: the Samsel Nested colormap, the Samsel Blue/Green Divergent colormap, the traditional Rainbow colormap and the traditional Cool/Warm colormap. Each image had a temperature scale with a tick mark at 20degC to identify the color associated with the Gulf Stream. Additionally, a set of crosshairs off the east coast of the US identified the start of the Gulf Stream and was a second method by which participants could identify the color associated with the Gulf Stream. Three narrow vertical boxes were shown in each image.

A Qualtrics study was created to run this study. Participants were first shown an example image to describe the task and explain the temperature scale, the crosshairs and the vertical boxes. The four different colormap images were randomly presented using the Qualtrics Heat Map question type and the participants were asked to click once in each vertical box to identify the location of the Gulf Stream within that box. After viewing each image, a validation question required participants to select the color they had identified as the 20degC Gulf Stream temperature.
Participants were obtained through Amazon Mechanical Turk a crowdsourcing site. Participants were screened out if they were colorblind. Basic demographic information was collected (age, gender, education level).

Data Analysis

In all, we had 43 valid participants in this study. Many participants self-selected out of the study after viewing the instructions. Additionally, each response was manually checked to verify that the participant understood the task and correctly clicked once in each box. Finally, we required that a participant had to correctly identify the color of the Gulf Stream in all four of the images to be considered a valid response.
The output from a Qualtrics Heat Map question is a set of (X,Y) values for each click where the (X,Y) are with respect to the upper left hand corner. Because the vertical boxes are narrow, the X values are constrained to a small range. The Y values indicate the vertical position where the participant identified the 20degC point within each box.
We want to know if the colormap made a difference in the ability of a participant to identify the location of the Gulf Stream (at 20degC). The first level analysis looks simply at the distribution of Y values for the three boxes (Y1, Y2, Y3) for each of the colormaps. As can be seen, the standard deviations of the Y values using the nested colormaps are much smaller than the standard deviations of the Y values using any of the other colormaps. There was much more consistent agreement within the participants on the location of the Gulf Stream when the nested colormap was used.

Nested Colormap Y1 Y2 Y3
Average 276.14 258.21 273.58
StDev 2.98 3.23 4.15

BlueGreen Divergent Y1 Y2 Y3
Average 276.31 266.57 298.10
StDev 18.03 17.74 43.73

CoolWarm Y1 Y2 Y3
Average 293.51 274.84 295.95
StDev 26.60 34.27 55.30

Rainbow Y1 Y2 Y3
Average 271.24 261.43 287.67
StDev 27.46 31.23 38.23

There are other avenues for analysis to pursue in the study and further analysis is ongoing.

Conclusions

Preliminary analysis indicates that participants found the task of identifying the Gulf Stream much easier when the nested colormap was used. We also found that using Qualtrics Survey software, in conjunction with Amazon Mechanical Turk, provides a powerful tool for creating user studies with quick turn-around on data collection.

Acknowledgements
Francesca Samsel – Center for Agile Technology, University of Texas at Austin
Mark Peterson – COSIM, Los Alamos National Laboratory
James Ahrens – Data Science at Scale Team Lead, Los Alamos National Laboratory
Terece Geld – Center for Agile Technology, University of Texas at Austin
Greg Abram – Texas Advanced Computing Center, University of Texas at Austin
Joanne Wendelberger – Statistical Sciences Group Lead, Los Alamos National Laboratory