How Do We Know If We Have Transformed Narrative Oceans?
Here’s how we’re starting to answer the persistent question about narrative change evaluation
By Tracy Van Slyke, Chief Strategy Officer at the Pop Culture Collaborative, and Erica Watson-Currie, PhD, Research Associate at USC/Annenberg’s Norman Lear Center
Before we get into frameworks and explanations, let’s name the big question in the room: When it comes to the work of narrative change, how do you measure narrative impact?
For field members, it’s a question they not only ask themselves in order to understand the impact of their work, but also to prove their work to obtain resources. For funders, it’s the question they are genuinely asking to understand the work better, but that they also need to answer to justify moving those resources to field members.
This question, or variations of it, is necessary — and one that everyone, in their own individual ways, is trying to answer. But the lack of cohesion, alignment, and shared direction around impact evaluation is undermining the ability of field members to measure individual and collective impact and limiting funders from moving the scale of resources necessary to advance transformative narrative change.
This is why in 2020, the Pop Culture Collaborative partnered with the USC Norman Lear Center’s Media Impact Project to lead the research and design for a useable, multi-pronged evaluation framework for field members and funders in the pop culture narrative change field to track, learn from, and evaluate progress in the work to transform narrative oceans in our quest to creating a just and pluralist culture.
And the result is a new beta framework: INCITE — Inspiring Narrative Change Innovation through Tracking and Evaluation.
This new learning and evaluation framework has been developed to equip the pop culture narrative change field — comprised of artists, values-aligned entertainment leaders and companies, movement leaders, cultural strategists, narrative researchers, philanthropic partners, and more — with a shared methodology to unearth learnings and track short and long-term impact, at both the individual and collective levels. This launch of the beta INCITE framework is the first step in a road testing process set to take place over 2024 to make it useful and usable by field members and funders alike. Read on to learn more about its development and how to get involved.
Four Barriers to Effective Narrative Change Evaluation
When the Pop Culture Collaborative publicly launched as a first-of-its-kind pop culture narrative change donor fund and learning community in 2017, we were constantly asked about how we measure narrative change impact. In the ensuing years, we have moved more than $30 million dollars to help scaffold the pop culture narrative change field and build its narrative power; launched the Becoming America Fund, the first multi-year, multi-million dollar initiative to fund and coordinate a narrative network of pop culture narrative change practitioners; and developed and shared foundational narrative change methodologies. These areas of work have required our team to stretch and experiment with strategies for narrative impact measurement within our grantmaking programs and extending to our impact in the field and ecosystem.
In tandem, the Media Impact Project at the University of Southern California’s Norman Lear Center has spent over 10 years collecting, developing, and sharing approaches for measuring the impact of media in order to better understand the role that media plays in changing knowledge, attitudes, and behavior, and ultimately, in shifting culture. The Media Impact Project’s work includes program evaluations investigating audience awareness, reception, and responses to media content, and research into why and how the content achieved its impacts.
The question that continues to plague us both: why is it so hard for all of us — field members, funders, and evaluators alike — to get on the same page about narrative change impact evaluation?
With our collective years of work and bird’s eye view of the practitioner and philanthropic fields, we have identified four key barriers:
1. We’re focused on measuring the drops, not the narrative ocean.
At the Collaborative, we understand that every day we are each immersed in an ocean of narratives, ideas, and cultural norms. This is why, as Pop Culture Collaborative CEO Bridgit Antoinette Evans has written, our work is not to “shift the narrative” on a particular issue, but instead to activate narrative systems — coordinated systems of mental models, narrative archetypes, and immersive story experiences — that work together to normalize pluralist behaviors and values in the U.S. This narrative system approach accelerates our ability to transform the narrative oceans we are all immersed in.
Dimple Abichandani, former Executive Director of General Service Foundation and one of the Collaborative’s “founding mothers,” summarized this contradiction well at ENTERTAINCHANGE: Philanthropy, the Collaborative’s 2019 gathering of 100-plus narrative change funders. “If we’re trying to change the ocean, why are we measuring the drops? We need to measure the ocean.”
The most common evaluation approach is to focus evaluation efforts on measuring the “drops” in the ocean — a singular story, film, television show, or impact campaign. It is extremely important to evaluate and, most importantly, learn from those initiatives in relation to the goals, and scale, of those individual efforts. But to understand if and how enduring narrative change is happening, we need to track and evaluate both the drops as well as how they add up to the transformation of a narrative ocean over time.
2. There isn’t a shared approach that accounts for the diversity of narrative change methodologies and strategies.
There are at least two key reasons for this.
First, while strategic communications and culture change are both types of narrative change work, their evaluation approaches are different (note 1). These two types of work require different types of expertise, strategies, and infrastructure, yet expected outcomes and applied metrics are often conflated. Right now, a mismatch of appropriate impact evaluation metrics is hindering our ability to track and understand the impact of different types of narrative change approaches. If we continue to conflate the two, then we’re setting ourselves up for blurring the learnings, undermining the impact analysis, and limiting the future activation of both narrative change approaches.
Second, culture change narrative indicators don’t match the diversity of approaches. Looking specifically at the pop culture narrative change field, there are a multiplicity of narrative change efforts. For example, a social justice organization’s narrative change work may focus on moving mass audiences around a specific idea, while another initiative is deeply focused on supporting intersectional BIPOC artists to build their creative power and career longevity within the entertainment industry.
The appropriate indicators don’t currently exist, or aren’t accurately applied, to those different sets of strategies and intended outcomes. This doesn’t even account for the stage of the work (For instance: Is it new? Has the work been in place for years?). We need to have clear, shared methodology and metrics that are adaptable and applicable to different narrative strategy approaches and stages.
3. Evaluation efforts aren’t tracking a field’s ability to build and wield narrative power.
Connected to the previous point, evaluation inquiries often start by focusing solely on external narrative outcomes such as reach and impact on audiences. Instead, we need to begin at a different point: evaluating the narrative drivers — field growth and industry transitions — that create the power base to transform a narrative ocean.
Too often skipped are the capacity development, relationship and partnership building, strategy-informing research and narrative intelligence gathering, and experiments and learning that form the foundation of an organization, company, network, and/or field’s ability to have the methodology and capacity to activate narrative change. In addition, we’re not tracking the efforts of what it takes to create transformational power shifts inside a narrative industry, like Hollywood, to create just working conditions and the authentic, needed storytelling that can help transform narrative oceans. This includes the ability to make decisions, move resources, and control the means of production and distribution from within narrative industries.
These narrative drivers are often the determining cause for the advancement or impediment of narrative goals and strategy — and so evaluation efforts need to account for them as actual outcomes to learn about and evaluate over time.
4. If philanthropy wants impact evaluation, we need to help field members pay for it.
Field members do want to evaluate their work, not to just prove its worth for the next grant proposal, but to understand what and how the narrative change strategies they’re designing and/or implementing are working towards their culture change goals.
But as the Media Impact Project well knows, few practitioners in the pop culture narrative change field have the resources — both the funding and the shared knowledge base — to even locate relevant research literature that could inform an impact evaluation, much less conduct consistent, ongoing evaluation of their individual work. Often, field members have defaulted to reporting only reach and engagement metrics and don’t have the resources to engage in broader data-gathering, relying on individual anecdotes or inconsistent participant feedback to attest to their effectiveness. These findings are difficult to generalize or synthesize into systemic impact analysis.
Philanthropy must identify how to match narrative change support with needed evaluation support.
Knowing these barriers have hindered the field’s ability to learn from and evaluate their narrative change work, and for funders to build the needed investment strategies, we went to work.
Meet INCITE: A Beta Framework for Narrative Change Evaluation
In 2020, the Pop Culture Collaborative partnership with the Media Impact Project led to the investigation and co-design of a multi-pronged field and funder learning framework and tools that could start to solve for these barriers.
The first result: Connections and Accomplishments, a first-of-its-kind network mapping (note 2) of the pop culture narrative change field, which set a baseline for understanding the connectivity and strength of the field and its evolving relationships. Some key findings set the stage for evaluating the growth of the pop culture narrative change field over time:
- At the time the survey was conducted, one-fifth of field members who took the survey were connected to one another.
- Many field members indicated that their partnerships had increased their abilities, and desired more connections and collaborations.
- About one-third of respondents indicated that their field relationships provided learning and evaluation support, while more than one-quarter desired this in their future partnerships.
These mapping results provided insights on how field members build and maintain relationships, share resources, and value collaboration. It also underscored the importance of sharing a common set of metrics and methods for assessing the impacts of these partnerships.
The Media Impact Project integrated these lessons into their multi-year information gathering and research review process that included:
- Reviewing a large selection of key articles by field members, narrative change readings, a range of earlier evaluation toolkits, models, and resources from across industries, as well as large portions of the Collaborative’s own archives of evaluation, reports, and more.
- Analyzing the overlaps and unique approaches that field members have towards evaluation efforts.
- Synthesizing evaluation gaps and challenges and identifying opportunities to incorporate metrics from related narrative, cultural, and social change fields.
In tandem, the Collaborative worked closely with Media Impact Project, bringing their expertise about narrative strategy and how the pop culture narrative change field operates to inform and provide feedback around the beta framework design.
During this time, Media Impact Project was also engaged in research projects with other pop culture narrative change field members (e.g., Define American, Caring Across Generations, futurePerfect lab, IllumiNative, National Domestic Workers’ Alliance, Color of Change). These engagements informed knowledge about field members’ interests, activities, and aspirations.
The result of this multi-year investigation and design process is INCITE — Inspiring Narrative Change Innovation through Tracking and Evaluation — an adaptable framework designed to equip both field members and funders with shared definitions, measurable objectives, learning questions, indicators, and metrics to understand and assess the impact of their individual culture change efforts as well as the field’s progress towards seeding the narrative ocean with pluralist ideas, narratives, and norms.
The INCITE framework below reflects our ultimate goal to create the conditions necessary to generate narratives that will ignite pluralist awakenings and a desire for social justice within audience members, which, in turn, lead to long-term culture change.
With this as our goal, there are two major categories for evaluation: Narrative Drivers and the Narrative Ocean. Across these categories are four levers of change within two buckets: Narrative Drivers holds the field and industry levers, and the Narrative Ocean holds the narrative systems and audiences and communities levers.
Let’s dive into the framework.
Narrative Drivers are the people and infrastructure that make narrative immersion in pluralist ideas, cultural norms, and stories possible. These drivers have two key levers for change that merit evaluation: field and industry. When field and industry analysis is conducted together, we can evaluate the pop culture narrative change field’s ability to grow and wield its narrative power, creating the conditions to transform narrative oceans over time.
Within the FIELD, we can evaluate and track both:
- Individual field members’ capacity (e.g., staffing, sustainability) and sophistication (e.g., partnerships, methodology, and strategy development) around their own narrative strategy and implementation.
- A coordinated field’s expansion, intra and external partnerships and networks, resilience, and long-term sustainability, as well as emerging needs and gaps.
Within INDUSTRY, we can track, within a specific narrative industry (e.g., Hollywood) the impact of narrative infrastructure investments, such as the results of:
- Individual talent incubators, networks, and/or new production companies that create new waterways (note 3) of access, career longevity, and/or investment in intersectional BIPOC artists, producers, executives, and strategists.
- The collective ability of specific, coordinated field partners to shift narrative decision-making power towards just working conditions within a particular narrative industry.
- How these mental models are moving into our narrative ocean and catalyzing collective immersion in pluralist ideas by coordinated narrative network through content production and distribution
The other area for evaluation is the Narrative Ocean — the enduring narrative environment itself. This includes the evaluation levers of narrative systems and audiences and communities.
Within NARRATIVE SYSTEMS, we can understand how core pluralist ideas — carried by pop culture stories and experiences through a coordinated network of artists, producers, cultural strategists, and more — are taking hold in our cultural waters, including indicators and metrics that track and analyze:
- How individual artists and projects are incorporating mental models — foundational ideas about the world and how it works — into their work, and how that content is reaching intended audiences.
- How these mental models are moving into our narrative ocean and catalyzing collective immersion in pluralist ideas by coordinated narrative networks.
That’s when we can start to evaluate AUDIENCES AND COMMUNITIES. Evaluation efforts often start and stop at audience reach, but to understand how narrative change leads to enduring cultural shifts, we need to go deeper. We need to understand how audiences engage with, repurpose, take ownership over, and build community around the content seeded by a narrative system. Through this lever, specific indicators will help us:
- Assess how the immersion of specific audiences in pluralist ideas informs and transforms their beliefs, behaviors, and sense of belonging.
- Track how pop culture fandoms and other digital and analog communities create and/or propagate pluralist narratives.
Within each of these levers, the Media Impact Project has identified, synthesized, and organized dozens of quantitative and qualitative indicators and metrics around categories. Here are some topline examples of categories of indicators and how they match to a specific lever of change.
For example, short-term outcomes at the field-level may include organizational capacity metrics, such as the number of team members and the diversity of their expertise dedicated to activating a narrative strategy. Longer-term outcomes might include partnership development and field building indicators, such as the quality, depth, and results of sustained relationships and collaborations over time.
It’s important to note that the INCITE framework is not intended to be read linearly. The activities and levers of change generally build upon one another, but there can be many different pathways to influence various audiences, industries, and culture. For instance, some field members may aim to indirectly influence narratives by way of leveraging the talent pipelines that feed the entertainment industry (i.e., industry-lever with narrative-level outcomes). Other field members may do the reverse, by creating content that sparks power shifts within Hollywood (i.e., narrative-lever with industry-level outcomes).
How Does INCITE Work in Practice?
The first step: As you look at the levers of change — field, industry, narrative systems, and audiences and communities — where do you see yourself, or your grantee partners, operating? Some might fit inside just one lever or across multiple levers. Even just this quick analysis will indicate that a one-size-fits-all evaluation approach doesn’t reflect the complexity and multiple layers for building narrative power and creating narrative immersion.
With INCITE, field members and funders can identify themselves within the lever(s) that authentically reflect their work and apply the corresponding indicators and metrics to learn from and evaluate their impact. We can then avoid the situation of — let’s say, for example — operating within the industry lever and exclusively trying to apply audience metrics to measure your impact.
This framework is only the beginning. Media Impact Project has also started to curate an initial set of project planning resources, suggestions for progress tracking indicators, and learning questions to help connect activities to outcomes or indicators of impact. Alternatively, when applied with an outcomes-orientation, INCITE offers a shorter pathway to options for practitioners and methods to assess types of impact and document project outcomes at different levels (e.g., field, industry, narrative systems, and/or audiences and the changes their work is trying to achieve).
While INCITE has been designed for the pop culture narrative change field and built around what we understand inside the entertainment industry, we hope that this framework expands to become inclusive of other industries working at the intersection of narrative and social change.
Join Us in Testing INCITE
In 2024, the Pop Culture will take INCITE and narrative change evaluation to the next level. We are inspired and motivated by the feedback we have received so far. During the design process, a small group of field members, including culture change strategists and leaders within the entertainment industry and social justice movements, were able to review and provide feedback.
“I have spent a lot of time … reflecting and remarking on the fact that we [are] really far from having any sort of infrastructure in place to actually evaluate this work, so it’s amazing to see a skeleton and a process for that.” — Ishita Srivastava, Chief of Narrative and Culture Change at Caring Across Generations
Over the next year, the Pop Culture Collaborative will organize and support a subset of grantee community and philanthropic partners to both test and then iterate this beta framework and the associated indicators to best support individual and collective learning and impact evaluation needs. Over time, we will work with our field and philanthropic partners to develop tools and resources that will track the evolution of the pop culture narrative change field and transformation of narrative oceans over time.
Our vision: to have shared evaluation vocabulary, methodology, framework, and metrics that will help us in our work of co-creating a just and pluralist culture where everyone inherently belongs.
JOIN US: If you are a field member and/or philanthropic partner interested in testing INCITE in your own narrative change work, please reach out to learning@popcollab.org.
For example, some organizations with whom we shared early drafts of these resources, like the United Kingdom’s Power of PoP Fund, have already begun to adapt the framework for their own purposes. We are in the process of developing testing, use, and attribution guidelines to ensure the integrity of the model while in this beta stage and beyond. We request that you do not use this framework without being in conversation with us.
Article contributions from Dana Weinstein, MA, Ksenia Korobkova, PhD, Erica Rosenthal, PhD, and Johanna Blakley, PhD.
Footnotes:
- Whereas strategic communications is focused on developing strategic, consistent, and targeted messaging to impact opinions and actions over the short and medium-term, a culture change strategy moves at the medium and long-term, working to shape people’s fundamental beliefs and behaviors through various forms of story and immersive experiences.
- A sampling of field members were invited to answer a series of inquiries focused on their priorities, areas of expertise, resources and needs, perceptions of the fields’ accomplishments, and the partnerships they have established within the field.
- Waterways was shared as an alternative to “pipeline” from Vanessa Roanhorse of Roanhorse Consulting.
ABOUT US
The Pop Culture Collaborative organizes and deploys financial and other resources to support the pop culture narrative change field — led by and centering Black and Indigenous peoples, people of color, immigrant and refugees, and Muslim peoples, especially those who are women, queer, trans, non binary, and/or disabled — to transform narrative oceans and build narrative power at scale.
The Norman Lear Center, based at the USC Annenberg School for Communication and Journalism, is a nonpartisan research and public policy center that studies the social, political, economic, and cultural impact of entertainment and works to bridge the gap between the entertainment industry and academia, and between them and the public. For more insights on investing in narrative change, see “A Strategic Framework To Guide Investments in Narrative Change.” With funding from the California Health Care Foundation, this report distills 25 best practices that make narrative change efforts more likely to be successful.
