A newsletter on audience research and development for cultural leaders. One reader calls it, "sometimes funny"

Having observed humans for forty years, I’ve noticed that when people can’t find satisfactory answers to whatever is troubling them, they will seek out one another for solace. They spend time together, talk with each other, and maybe try to imagine what things will be like in the future.

It’s healthy and normal for humans to be together in this way.

I saw this poll on Twitter by Ryan Dodge the other day, and it got me thinking about how humans behave in uncertain times:

I’m sharing Ryan’s poll here because I hope it will help illustrate that there’s a sort of continuum to consider.

Museum or casino?

On one side of the continuum, you have healthy speculation and conversation — a kind of social balm — like Twitter polls.

On the other side, you have gambling.

I’ve written about gambling before. When we ask people what they will do in the future and then try to plan around those statements as if they were somehow factual, we’re gambling.

Surveying people to try to find out what will make them feel safe attending cultural organizations once this pandemic subsides or what organizations they are more likely to revisit once the crisis is over is gambling. You’re betting on people knowing what they will actually do in the future and that their responses are accurate.

But what people say they will do is an unreliable measure even in “normal” times.

This is user research 101, so I’m always surprised to see research firms like IMPACTS, which are closely followed by cultural leaders, sharing information like this. (You can find lots of articles on gambling out there. It’s taken for granted in the UX industry that asking about future behavior is unreliable. Or just recall the 2016 election.)

Surveys can help you understand questions related to “how much” or “how many” — They are not well-suited to get an understanding of Why or What Will Happen.

A precise measurement of the wrong question

It’s not that IMPACTS isn’t rigorous in their examination of what people will do post-pandemic — They are.

For example, look at how they put together the categories of possible responses to try to learn what people might need to feel safe visiting cultural organizations (emphasis added):

The data below surveys 3,497 adults in the United States. First, we collected people’s answers to this question using a process called lexical analysis that allows us to broadly categorize responses from people using their own words – the technologies that enable this process help to minimize the risks of unintentional biases that occur when facilitators translate or summarize a respondent’s statements. These categorized responses are thereafter used to populate the response range of a multiple-choice question. In other words, we did not internally brainstorm these options and present them in a survey based on our best guess of what people would say. The options came from survey respondents.

It’s not that their methodology is flawed — It’s that the premise is misguided.

In the 19th century, phrenologists were rigorous and precise in their measurements of the human skull, but that rigor didn’t change the fact that the endeavor was flawed from the beginning.

If we can’t rely on surveys to predict future behavior, what should we do instead?

The most reliable predictor of future behavior is past behavior, so — Look to the past if you want to understand what people will do in the future.

The last time we found ourselves in a situation like this might have been 1918, though, so that recommendation is going to be harder to follow.

But even if these are truly unprecedented times, I think it would be better to embrace the uncertainty and try to focus attention on what’s within our control than it is to try to plan around what thousands of people imagine they will do in the future.

Now more than ever, it’s important to make space for good intentions.

Last week, I wrote to you about two very different approaches to engaging audiences during the pandemic.

Since then, I’ve been thinking about how easy it is to appear tone-deaf or out of touch with what people are going through today. When we’re feeling overwhelmed, it may become harder for us to expend the energy to imagine that others may be in a different space than we are. The idea that some people may feel a responsibility to slow down and think more strategically may seem like a privilege, and any expression of those values may seem somehow tone-deaf to someone who is overwhelmed. The idea that an organization might invest in professional development or research at a time like this might seem unreal or lavish to an organization that is being swept away by the current crisis.


Never miss a post. Get letters like this on in your inbox.

Subscribe now


I bring this up because I want to point out that intentions are important, and firms like IMPACTS — as well as museums that are running similar surveys or focus groups in-house to try to plan around people’s stated preferences or plans — have good intentions. It’s important to acknowledge that.

We’re all trying to find ways to help using the tools that we use best.

So, as much as I feel obliged to point out that it can be dangerous to plan around surveys of what people will do in the future, I also want to be supportive of people who are genuinely trying to help each other.

Just don’t bet too high on what people say they will do in the future.

As always, reply to this email to let me know your thoughts or leave a comment below.

Kyle

P.S. I went on to write more on this — You can read part two here.

What people are saying about this letter

Leave a comment

Subscribe now

You’ve successfully subscribed to SuperHelpful Letters
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Your link has expired
Success! Check your email for a magic link to sign-in.