Exploring the world of design research in an imperfect vessel, accompanied by a crew of rogues, idlers, and ne’er-do-wells
(Reading time: 6 glorious minutes, 17 heart-stopping seconds)
List member David Van wrote in with some objections to my survey of decision-makers at cultural organizations, which I’d like to share with you today. (David has kindly given me permission to do so.)
First, for those just joining us: I’m in the midst of a research project studying what motivates constituents of cultural organizations. Interviews of constituents will central to the project, but first I’m surveying and interviewing decision-makers at cultural orgs to check my assumptions about their beliefs about audience motivations and what may inform those beliefs. List members can take the survey here, and here is the survey for non-list members. Feel free to share the latter with your colleagues.
Second, a TLDR on today’s letter: Some people believe that a census is required in all cases for any survey to yield valuable insights. I disagree. I think a convenience sample — in this case, decision-makers who can be reached via LinkedIn — can create value, particularly when combined with other forms of research, like interviews. But underneath this discussion of a particular survey is a more interesting question: At what point should we stop trying to eliminate uncertainty and begin investigating?
Now, David’s email came in response to a question from me:
Do you think museum leaders who can be found on LinkedIn would give meaningfully different answers to the questions I’m asking than those who cannot be reached on LinkedIn?
David replied (shared with his permission, broken into paragraphs for easier reading):
Perhaps if your questions were simpler. Do I think people with LinkedIn profiles give different answers? Absolutely. Do I think people who are active on LinkedIn give different answers than people with profiles but who are not active? Absolutely. And I say that not because I am a Seer, but because it’s what all good research on research concludes: people rarely differ on only one characteristic. We know they’re different. What we don’t know is about what and by how much. Their wants, needs, expectations, and motives differ.
Also, I don’t know what Museum Leaders are or how they’re defined, but we know it is common on LinkedIn for people to overclaim a skill or a role. We do know that people who actively participate on LinkedIn have to find the time somewhere, don’t they? Maybe they’re trying to drum up business. Often they are unemployed. Others have unchallenging jobs that allow them the free time to noodle around on Forums and in Groups.
Yes, sir, your LinkedIn “sample” includes some TBD number of idlers and not only, but self-reported and self-selected participants, also notorious for their lack of representativeness.
Who are “museum leaders”?
By “museum leaders”, I mean people operating at a senior level at cultural organizations, and I’m especially focused on executive directors. I used “museum leaders” as shorthand in my email.
Will executive directors (EDs) on LinkedIn respond in a different way than those who are not on LinkedIn?
I find it hard to believe that people who are active on LinkedIn would consistently give a different sort of answer to a question like, “What motivates people to visit?” when compared to people who aren’t on LinkedIn.
What would it mean for that to be the case?
Maybe EDs who are active on LinkedIn have some common personality traits that differ from those who aren’t on LinkedIn, and those traits color their view of why people visit their organization? I don’t see that as likely, but I suppose it’s possible.
I could put everything on hold and try to find out.
But, then, we could argue that people who voluntarily take any sort of survey are different from people who never take surveys.
Therefore, any survey I create will be flawed.
Therefore, this survey should not exist.
David writes (emphasis his):
Virtually no good investigations begin with surveys. Read that again. Surveys have value only for quantifying. They are not designed for exploring.
This survey of organizational leaders consists of mostly open-ended questions. I’ll be coding the responses to get a sense of frequency, and I’ll be looking for clues as to how respondents mental models around audience research differ from my own. I don’t think a survey is the most in-depth way to explore respondents’ views, but that doesn’t make a survey completely invalid.
As for no good investigations beginning with surveys — I’m not really starting with a survey. Last winter, I interviewed dozens of directors at cultural organizations, asking questions about their experience with design research. This survey is meant to give me a different view before conducting more interviews.
So, David and I might agree that surveys alone aren’t as useful as surveys that are supported by other forms of research.
I’m about the last person to stick up for surveys — I think organizations rely on them far too much… but I’m wary anytime I hear people speaking in absolute terms or suggest that a particular approach has no value under any circumstances. And my natural inclination when anyone tells me I should not do some exploring is to hoist a sail and set to sea.
A more flexible perspective
To be fair, there are some ways in which I think David’s objections are valid.
For example, since there’s no incentive to take the survey, maybe the people who choose to take the survey are generally more curious than those who don’t.
Maybe that curiosity makes them more likely to support audience research within their organization or to work at an organization that supports that kind of curiosity and research.
If that’s true, then responses the question, “Have you invested time or money in studying your audience?” would not be representative of cultural organizations as a whole.
That’s something I definitely need to keep in mind.
But that doesn’t mean the survey won’t give me some insight into how cultural organizations that are doing research go about it. And it doesn’t mean the survey won’t shed some light on how those organizations view their audiences’ motivations.
A world populated by unemployed loafers, scam artists, vagrants, vagabonds — and they can’t wait to take my unpaid survey
What about the argument that people can claim to be other than who they are on LinkedIn, therefore the survey I’m conducting will not be representative?
Here again, I don’t agree, but let’s take a closer look, starting with the tool I use to filter out unqualified candidates.
There is no filter for “idler”, but do we need one?
LinkedIn’s Sales Navigator lets me filter by seniority, and its title search function lets me get far more specific in excluding people who may slip through that filter. My search excludes people with titles that include words like “receptionist”, “consultant”, or “maintenance”. (It’s a long list of exclusions.)
I then take that search and use an automation tool to invite them to the survey.
Will some unqualified people still receive an invitation to the survey? Yes.
Will some of those unqualified people click through to view the survey? Maybe.
Will some of them take the survey? Perhaps.
But those people are likely identifiable in the results based on their answers. If they choose “other, specify” and enter in their title, for example.
Moreover, in reviewing results, I’m primarily focused on responses by executive directors. People who are executive directors are clearly “museum leaders”.
Ok, but who are these people, really?
As David points out, respondents could be lying about who they are.
However, there are identifying characteristics with each submission. Results of the survey will be shared anonymously, but as of this writing 79% of the people who take the survey share their email address and many who take the survey accept the connection request they get on LinkedIn, so I can see who they claim to be.
Yes, someone could create a LinkedIn profile and claim to be an executive director of a museum when they are, in fact, an unemployed person who is… looking for a job? Running a scam where pretending to be the director of a children’s museum makes them lots of money?
In all seriousness, it is possible that someone would claim to be a director who is not.
But many things are possible, and we can try to eliminate all those possibilities till the sun consumes the earth.
There comes a time when action is better than inaction; something is better than nothing; and perfect is the enemy of doing anything at all.
My goal with this survey — which I think many readers may share as they try to better understand the people they serve — is to reduce uncertainty, not to eliminate it altogether, which is unnecessary and all but impossible.
I’ll always be on the lookout for ways to improve my efforts… In the meantime, the work continues.
Thanks for reading,