HypothesisOur hypothesis is that participants would be able to more consistently identify the location of the Gulf Stream in an image using the Samsel Nested colormap than in other comparison colormaps.
MethodologyFor this study, the POP data was used to produce an image of the Gulf Stream, stretching from the east coast of the US to the west coast of Europe/Africa. Four colormaps were compared: the Samsel Nested colormap, the Samsel Blue/Green Divergent colormap, the traditional Rainbow colormap and the traditional Cool/Warm colormap. Each image had a temperature scale with a tick mark at 20degC to identify the color associated with the Gulf Stream. Additionally, a set of crosshairs off the east coast of the US identified the start of the Gulf Stream and was a second method by which participants could identify the color associated with the Gulf Stream. Three narrow vertical boxes were shown in each image.
Participants were obtained through Amazon Mechanical Turk a crowdsourcing site. Participants were screened out if they were colorblind. Basic demographic information was collected (age, gender, education level).
Data AnalysisIn all, we had 43 valid participants in this study. Many participants self-selected out of the study after viewing the instructions. Additionally, each response was manually checked to verify that the participant understood the task and correctly clicked once in each box. Finally, we required that a participant had to correctly identify the color of the Gulf Stream in all four of the images to be considered a valid response.
The output from a Qualtrics Heat Map question is a set of (X,Y) values for each click where the (X,Y) are with respect to the upper left hand corner. Because the vertical boxes are narrow, the X values are constrained to a small range. The Y values indicate the vertical position where the participant identified the 20degC point within each box.
We want to know if the colormap made a difference in the ability of a participant to identify the location of the Gulf Stream (at 20degC). The first level analysis looks simply at the distribution of Y values for the three boxes (Y1, Y2, Y3) for each of the colormaps. As can be seen, the standard deviations of the Y values using the nested colormaps are much smaller than the standard deviations of the Y values using any of the other colormaps. There was much more consistent agreement within the participants on the location of the Gulf Stream when the nested colormap was used.
There are other avenues for analysis to pursue in the study and further analysis is ongoing.
ConclusionsPreliminary analysis indicates that participants found the task of identifying the Gulf Stream much easier when the nested colormap was used. We also found that using Qualtrics Survey software, in conjunction with Amazon Mechanical Turk, provides a powerful tool for creating user studies with quick turn-around on data collection.
AcknowledgementsFrancesca Samsel – Center for Agile Technology, University of Texas at Austin
Mark Peterson – COSIM, Los Alamos National Laboratory
James Ahrens – Data Science at Scale Team Lead, Los Alamos National Laboratory
Terece Geld – Center for Agile Technology, University of Texas at Austin
Greg Abram – Texas Advanced Computing Center, University of Texas at Austin
Joanne Wendelberger – Statistical Sciences Group Lead, Los Alamos National Laboratory