Let the War on Evaluation Begin
Shots fired from the Department of Generation.
I’m putting together a resource for cultural organizations that combines several frameworks, like Jobs To Be Done (JTBD) theory, for understanding audience motivations. Here’s why.
“Who isn’t coming and why?”
Museum leaders wonder about this question a lot. They’re still asking it, even after years of focus groups and surveys.
Granted, it’s a good question to always be asking. Just as you might continually track visitor sentiment and satisfaction, it’s helpful to always be trying to suss out who isn’t coming and what sort of audience goals the organization may not be adequately addressing.
But many of the organizational leaders I talk with ask this question with the intensity of someone in hell asking for a glass of water — It’s as if they’ve never tasted it before.
That’s because many organizations aren’t regularly practicing the kind of research that would help answer the question, and the attributes they commonly use to describe and understand their audiences are not helpful and are in conflict with many organizations’ values.
Let’s unpack the question.
“Who is not visiting our children’s museum (or art museum, aquarium, arboretum, etc.), and why?”
It’s two questions rolled into one.
To figure out who is not coming — the first question — you can try to learn who is coming. When you know who is coming, you can start to get a sense of who isn’t engaging with your organization. Of/By/ForAll and Slover Linett put together a helpful toolkit for finding out who is coming.
Still, you may only learn about who’s coming in terms of demographics. That can be valuable information, but it doesn’t lead to any answers as to what prevents people from visiting, which is the second question. And I wonder if it’s really two separate questions at all — Are we just asking “who isn’t coming?” so we can get to the all-important “Why?”
Anyway, this resource I’m working on — a piece and version of which you’re reading right now — is meant to:
- Explain this persistent question of “who isn’t coming and why?” as one that has to be addressed through generative research — not the sort of evaluation most cultural organizations are used to doing.
- Help leaders assess the forces of progress and barriers to engagement that are specific to their audiences. (It’s descriptive, not prescriptive.)
- Provide some insights and recommendations as to what your organization can do to fill any potential gaps in your audience research and development practices. (There’s a diagnostic piece.)
Its working title, The Museums-As-Progress Model, comes from a letter I sent to you all a few weeks ago — In short, if organizations can define their audiences less in terms of demographics and more in terms of what progress they can help people achieve, the better positioned they may be to begin answering the question of who isn’t coming and why.
After all, if you learn that millennials are an underrepresented group at your organization, how helpful is it to then ask, “What do millennials want?” Classifying people like that is a start, but it’s not a finish. And would a 35-year-old human find your description of her as a “millennial” to be a bit reductive? We’re not far from thinking in terms of “old” people or “poor” people, and it just goes downhill from there.
On the other hand, if we segment an audience in terms of life goals — or Jobs To Be Done — we may uncover more actionable insights. We can start to understand what people are hoping to achieve over time by engaging with the organization, and we can begin to speak more directly to those goals. Bonus: People may find it more agreeable to be understood in terms of their fundamental goals and motivations rather than in terms of their age, income, or ethnicity, etc.
If you can segment the audience in terms of goals rather than demographics, you can begin to examine what people in your community are doing and where they are going to realize their goals today. By studying their choices and behavior, you can uncover what impediments may be preventing people from engaging with your organization. (My conversation with Paul Kortenaar last week dives into this.)
In the garden of ideas, evaluation is an invasive species that consumes all of the ecosystem’s natural resources.
Ok, I’m being provocative. I don’t think evaluative research is bad, but one reason I’m writing these letters and putting together this resource is because I think there can come a point at which evaluation becomes so dominant that other priorities can suffer and opportunities drift from view.
It’s easy to forget that formative evaluation is still evaluation — We get an idea, which likely came from staff or by observing another organization, and now we’re going to test it. You’re already invested. You already have something to lose.
Generative research, which uncovers new ideas by studying the context of people’s lives, is less common. There are lots of reasons for that, but another question that I sometimes hear — “What should our museum look like in five years?” — is one that can’t readily be answered by pursuing only evaluative research. So, another question that goes unanswered.
The closest thing to generative research I see in cultural organizations is focus groups. But these are often attitudinal studies — they’re less focused on behavior — and they often include questions along the lines of “Would you buy a membership if…” or “What would you like to see us offer next?”
Those types of questions are rarely productive and they’re just a heartbeat away from turning into a sales pitch. Here’s the cartoon version:
Facilitator: “Now that we have you all in this room, our membership director has a question for you.”
Director: “Could you please tell us why you haven’t renewed your membership?”
Participants: “Ok, sign me up — Can I leave now?”
Research is dominated by education and visitor experience departments.
The answer to the question — Who isn’t coming and why? — is rooted in life goals. To understand life goals, it’s helpful to study people’s lives — the context of their lives and the decisions they make outside of the organization. When the organization’s research is dominated by folks in education and operations, it’s natural for that research to be rooted in the organization and the experience goals and end goals that are central to the organization’s immediate interests.
Folks in education want to understand the impact of their work and how they might improve. (And prove impact for funding.) Experience and operations staff want to monitor satisfaction and find ways to improve existing products and services.
Again, that’s important, evaluative work, but it’s focused on optimization. That kind of study is usually geared towards experience goals and end goals rather than life goals.
Many organizations commit to research that is focused on experience goals (sensory, emotional, task-oriented) and end goals (activities) — fewer commit to the kind of generative research that can expose life goals (progress/identity-oriented).
So, the research cultural organizations carry out often originates around educational or operational goals. Marketing and communications teams might brush up against those results, but they’re often not initiating research or involved in its development. That can lead the organization’s marketing department to communicate what is happening (“here’s what’s on the menu for this month”) rather than why it is happening — value propositions.
Misunderstanding JTBD may do more harm than having no knowledge of it at all.
The tipping point for me came as I’ve seen examples of people describing Jobs To Be Done in terms of activities rather than in terms of progress or life goals. While it’s exciting to see people talking about JTBD and how organizations might apply the theory, I worry that a misinterpretation could keep organizations rooted in the study of experience and end goals, which they’re so familiar with.
If leaders interpret JTBD as being activity-based, they may simply continue to study visitors’ end goals. I worry they may not realize the potential of researching life goals — or Jobs — that motivate engagement over time.
So, that’s why I’m writing today.
As always, reply to this email to let me know your thoughts or share your feedback here.
Thanks for reading,