This article was co-authored by Frances Barlas, Ph.D. and Randall Thomas.
Many of us walk around with a high–powered, Internet-enabled computer in hand or nearby at all times of the day. For some, the smartphone is the first thing they reach for when they wake up in the morning and the last thing they put down at night. Increasingly we are turning to our smartphones to fill the downtime as we wait for our turn in a checkout line or kill time in a waiting room before an appointment. It is the “do-everything” device that assists us in dozens of ways every day.
For survey researchers, this means that more and more respondents are looking to take our online surveys on their smartphones. But in many cases, the survey experience is not yet optimized for mobile – respondents take far longer to complete a survey on their smartphones than those who take the same survey on a desktop or laptop computer. As a result, we see lower quality data and higher breakoff rates among smartphone respondents.
As an industry, we need to find creative solutions to help these respondents take our surveys on the devices of their choice, including the small-screened smartphone. This means that long, wordy surveys with intricate, equally long answer options practically guarantee that there will be noise in the data.
Converting survey response scales to a mobile-friendly design
If we think about how to communicate with as few words as possible, what comes to mind? Emojis are a wordless language consisting of small images – the smiley face, of course, but also many variations that express a range of emotions. Their minimal footprint presents an intriguing possibility for helping convert survey response scales to a mobile-friendly design. Similarly, a scale made up of numbers with no corresponding response labels would have a small footprint.
As an example, here’s a question asked with a traditional semantic-labeled response scale:
And that same question and response scaled conveyed with emojis:
Finally, that same response scale conveyed with numerics:
These simple symbols clearly make sense for smartphone communicating. The question becomes – do we get differences in response when we use the different scales? If we were to consider using these alternative response scales, we need to be sure that emojis and numeric scales convey meaning — and gradations in meaning – quickly. Can respondents use them to answer more quickly and as accurately as with the more traditional semantic-labeled scale? With support from the Advertising Research Foundation, we conducted a study to investigate these questions.
Do emojis increase the efficiency of survey responses?
We asked respondents how much they liked or disliked doing a series of five activities and randomly assigned them to a response type – the semantic, emoji, or numeric response labels. In looking at efficiency of response, we found that longer scales with more categories meant longer completion times for semantic, but not emoji or numeric labels. Numeric labels took the least amount of time to complete. So we do see some increased efficiency with these new scale formats over the traditional semantically labeled scales.
Further, these new answer types created a consistency in response across device and response label type; there was no difference in average means across devices, regardless of the type of label a respondent was assigned to see. We also asked respondents how often they engaged in each of the behaviors to help us assess validity; activities that people like to do should also be activities they do more frequently. In looking at the correlations between activity enjoyment and frequency, we saw no difference in validity across the various response-scale types. This means that the emoji and numeric response scales were just as valid as the more traditional semantic labels.
Finding a smartphone-friendly response format
Overall, we found emojis may be a viable, smartphone-friendly response format if they are used with care. However, the use of emojis did limit the types of questions we could ask – we could only ask questions that allowed us to use the most basic and universal emojis (straightforward smiling and frowning faces or thumbs up and down). In addition, we were not able to come up with a clear and easy-to-interpret emoji response scale that could measure importance or frequency. Emojis lend themselves better to evaluative scales (good-bad judgments or agreement scales, for example). Although some topics might not be appropriate for use with emojis, as in the following example:
So perhaps thumbs sideways on emojis for now – if you are going to consider using them, take note of the limitations we mentioned. Numeric labels may be a more efficient label type with equal validity as the other label types. We are currently working to extend these findings across topic areas.
Frances Barlas, Ph.D. is Vice President, Research Methods on the Sampling Statistics team at GfK. She can be reached at frances.barlas@gfk.com.
Randall Thomas is Senior Vice President, Research Methods on the Public Communications and Social Sciences team at GfK. He can be reached at randall.thomas@gfk.com.
