skip to Main Content
Getting The Most From Your Open-Ends

Getting the Most from Your Open-Ends

Open-ended questions are invaluable in research. They allow us gain insight on what someone is thinking and feeling, helping us as researchers to help understand the why. While qualitative research gives the opportunity for open-ended dialogue, we are often limited to the number of open-ended questions that we can ask in quantitative research in order to minimize respondent fatigue. Each open-ended question in quantitative research is an extremely valuable opportunity to hear from our respondents in their own words.

More and more, we have seen market research companies make the shift from manually coding quantitative open-ended responses in-house, to outsourcing to third parties or using AI software. And to an extent – we get it! Manually coding hundreds or thousands of open-ended comments isn’t necessarily a glamorous task. However, at Elevated Insights we still have our internal researchers manually code all open-ended responses in our quantitative surveys. Why?

Artificial intelligence is still in early stages of development.

Despite the improvements in recent years, there are nuances that AI can’t detect. Positive vs. negative sentiment, sarcasm, abbreviations, and typos may not be correctly captured when using artificial intelligence, which can impact the accuracy of coding results. Having our researchers manually code gives the human aspect that AI is missing and leaves less room for misunderstanding and misinterpretation.

It allows us to confirm the quality of responses.

While we thoroughly clean our data after we close fielding – reviewing for speeding, straight-lining, and quality of open-ended responses – we know that professional survey respondents and bots are getting increasingly sophisticated, making them more difficult to detect. When reading each individual response thoroughly for coding, we also review them to make sure we aren’t seeing any red flags in our data (like repeated responses or responses that aren’t answering the question at hand). This double-check helps give us confidence that our final data meets our quality standards and that no ’cheaters’ have slipped through the cracks!

It gives us a deeper understanding of the insights.

For us, the additional day or two it takes to manually code is worth every minute spent. By reading each individual response, we walk away with an invaluable level of knowledge and understanding. And it gives our researchers the opportunity to see patterns and linkages of comments that provide additional context we wouldn’t have otherwise. This gives us the ability to tangibly convey the story we are telling and bring it to life, ultimately helping us elevate the insights for our clients.

So, what is our approach to achieving these elevated insights?

We code the full response.

When reviewing methodology notes from other research firms, we have found that some companies who manually code open-ended responses only code the first comment within the response. For example, if a response reads “I don’t like the color of the product, but I do think it is user-friendly and super affordable!” some companies will just code that this respondent dislikes the color (ignoring the other comments later in the response). With this approach, you risk not capturing the respondent’s full sentiment, which can impact the quality of the data. Instead, we take the time to read each respondent’s full response and assign multiple codes to one response if necessary.

We append the code to include affect.

Clarifying whether the response is positive, negative, surprising, etc. completely changes the intent and likely action associated with a respondent’s comment.

We note patterns and linkages with responses, creating hypotheses to explore for higher-level insights.

Tools, or even humans only focused on counting the frequency of mentions, can miss critical connections that comments have with each other.  For example, in one snack food study, we noticed that many consumers who cited a product had a good/unique texture and also cited satisfaction or calming after-state.

In another study, we noticed that the residents sharing the need to revise the city’s short-term rental policy also referenced the importance of listening vs. asking for opinions.  This clarified specific objections to how the city had responded during a recent time of public outreach.

We incorporate quotes from the open-ended responses into the report to help bring the story to life.

We pay attention when coding, noting comments that would bring value to multiple areas of the report – not just the open-ended question that they directly pertain to. When picking quotes to highlight, we look for well-written and thoughtful comments that summarize the majority of opinions. This helps provide additional context that may be missing otherwise and gives a quantitative report a qualitative level of rationale.

We present the coded data in a range of ways to ensure it is relevant and tangible.

In addition to sharing the code frame in a table or chart format (with counts/percentages to show frequency of mentions), we typically create a word cloud to visually show what words or phrases were recurring throughout all responses.

Furthermore, when appropriate, we like to provide the raw, open-ended comments to clients. Rather than giving them a Word document with hundreds or thousands of comments, we like to provide our clients with comments split by theme or topic. For example, when working on a community survey, we asked respondents if they had any input they’d like to share with local leaders about improving life in their community. We split the open-ended comments by keyword and were able to provide community leaders with the responses that related directly to the causes that were relevant to them (for example, sharing responses referencing homelessness to an organization that helps the homeless community and responses about housing to an organization working to provide affordable housing solutions).

From our experience, presenting the coded data in a range of ways helps make it easier for our clients to understand, digest, and use internally to take action.

We can’t give away all our secrets, but if you are looking for a partner to help elevate your insights, give us a call or shoot us an email – we’d love to work with you!

 

Erin Bradley
Managing Principal at Elevated Insights
[email protected]

Close
Back To Top