Getting the Most from Your Open-Ends

Open-ended questions are invaluable in research. They allow us gain insight on what someone is thinking and feeling, helping us as researchers to help understand the why. While qualitative research gives the opportunity for open-ended dialogue, we are often limited to the number of open-ended questions that we can ask in quantitative research in order to minimize respondent fatigue. Each open-ended question in quantitative research is an extremely valuable opportunity to hear from our respondents in their own words.

More and more, we have seen market research companies make the shift from manually coding quantitative open-ended responses in-house, to outsourcing to third parties or using AI software. And to an extent – we get it! Manually coding hundreds or thousands of open-ended comments isn’t necessarily a glamorous task. However, at Elevated Insights we still have our internal researchers manually code all open-ended responses in our quantitative surveys. Why?

Artificial intelligence is still in early stages of development.

Despite the improvements in recent years, there are nuances that AI can’t detect. Positive vs. negative sentiment, sarcasm, abbreviations, and typos may not be correctly captured when using artificial intelligence, which can impact the accuracy of coding results. Having our researchers manually code gives the human aspect that AI is missing and leaves less room for misunderstanding and misinterpretation.

It allows us to confirm the quality of responses.

While we thoroughly clean our data after we close fielding – reviewing for speeding, straight-lining, and quality of open-ended responses – we know that professional survey respondents and bots are getting increasingly sophisticated, making them more difficult to detect. When reading each individual response thoroughly for coding, we also review them to make sure we aren’t seeing any red flags in our data (like repeated responses or responses that aren’t answering the question at hand). This double-check helps give us confidence that our final data meets our quality standards and that no ’cheaters’ have slipped through the cracks!

It gives us a deeper understanding of the insights.

For us, the additional day or two it takes to manually code is worth every minute spent. By reading each individual response, we walk away with an invaluable level of knowledge and understanding. And it gives our researchers the opportunity to see patterns and linkages of comments that provide additional context we wouldn’t have otherwise. This gives us the ability to tangibly convey the story we are telling and bring it to life, ultimately helping us elevate the insights for our clients.

So, what is our approach to achieving these elevated insights?

We code the full response.

When reviewing methodology notes from other research firms, we have found that some companies who manually code open-ended responses only code the first comment within the response. For example, if a response reads “I don’t like the color of the product, but I do think it is user-friendly and super affordable!” some companies will just code that this respondent dislikes the color (ignoring the other comments later in the response). With this approach, you risk not capturing the respondent’s full sentiment, which can impact the quality of the data. Instead, we take the time to read each respondent’s full response and assign multiple codes to one response if necessary.

We append the code to include affect.

Clarifying whether the response is positive, negative, surprising, etc. completely changes the intent and likely action associated with a respondent’s comment.

We note patterns and linkages with responses, creating hypotheses to explore for higher-level insights.

Tools, or even humans only focused on counting the frequency of mentions, can miss critical connections that comments have with each other.  For example, in one snack food study, we noticed that many consumers who cited a product had a good/unique texture and also cited satisfaction or calming after-state.

In another study, we noticed that the residents sharing the need to revise the city’s short-term rental policy also referenced the importance of listening vs. asking for opinions.  This clarified specific objections to how the city had responded during a recent time of public outreach.