Imagine you recently attended a public program at your local library about building a community garden. You listened to speakers from nearby towns talk about their experiences acquiring land, working with local officials, and donating produce to food pantries. One after another, they attested to connecting with longtime neighbors with whom they had never before exchanged a greeting. Struck by these testimonials, you start to greet all of your own neighbors. “Good morning!” to the neighbors you see as you set off for work. “Good afternoon!” to the neighbors collecting their mail. “Good evening!” to the neighbors you pass on your after-dinner walk. “Good night!” to the neighbors taking out the trash before going to bed. Within a week, neighbors start to initiate the greetings. Within a month, you know all of your neighbors' names. Within three months, most people in the neighborhood are in the habit of greeting one another. Within a year….
This newfound neighborliness is an obvious, if indirect, impact of your participation in your library’s public program. After all, the foundational topic of the program was connecting with community members, with community gardens serving only as an excuse to do so. Beyond more connected communities, what other kinds of indirect impacts might one expect from the range of public programming available at libraries across the country? How might one assess such impacts? How might one track the route of such indirect impacts from those who attended a library program outward to those with whom they later interact, and further still to their communities, regions, and, eventually, the nation as a whole?
These are burning questions for public programming library workers, programming funders, and their program evaluators. However, before attempting any answers, it would help to first get the lay of the land by asking a preliminary question: what’s going on at the library?
A thumbnail sketch of library public programming
Libraries increasingly function as one-stop providers for a variety of community services. Yes, those services include the circulating collection and reference desk (1.8 billion items, circulated 2.2 billion times, and 220 million questions answered in 2019, but also classes on almost every subject including nutrition, mental health, citizenship, and languages (English and others). Library users can take exercise classes, receive job training, pick up ready-made meals, and get vaccinated—all at their local library.
This broader community mission dates back at least to 1992 when the American Library Association established its Public Programs Office to support an emerging trend in library services. Since then, library public programming has increased dramatically. According to the 2019 Public Libraries Survey (the last survey year not affected by COVID-19 disruptions), almost 125 million people participated in almost 6 million library programs that year.
The trend is likely to continue, with funds for implementing this broader mission coming from taxpayers—via the Institute of Museum and Library Services (IMLS), among other federal, state and local agencies—and from private foundations, including Andrew W. Mellon Foundation and the John S. and James L. Knight Foundation. These funders support library public programming for the sake of goals that go beyond the particular programs, at particular libraries, welcoming particular participants. Of course, it matters that libraries offer meaningful experiences for the people who participate in their programs. But the goal of large-scale support—whether from taxpayers or foundations—is large-scale, long-term, societal change.
The goal of societal change brings us back to the "burning questions" about impact from earlier.
Background on the National Impact of Library Public Programs Assessment
We, the authors, have been grappling with questions about the impact of library public programs since 2014. That is when we and other colleagues at the American Library Association and Knology (a social science research organization) launched a project called the National Impact of Library Public Programs Assessment (NILPPA). The IMLS has supported NILPPA through a planning phase over two research phases:
NILPPA Phase 1 (2017-2019) yielded two frameworks: (1) a conceptual framework for categorizing public programs offered by U.S. libraries and (2) the first-ever competency framework for designing and running public programs at libraries which outlines nine core competencies for library staff and can guide curriculum development for preparing future library professionals.
NILPPA Phase 2 (2021-Present) focuses on two goals: (1) developing a practical framework for characterizing and assessing how libraries work with partner organizations on public programming and (2) generating and validating sets of concrete, observable behaviors and events that could serve as indicators of the different kinds of impacts that libraries are attempting to achieve through their public programming. This second goal is the focus of this article.
What is (indirect) impact?
Specifying what we mean by “impact” is the first step toward answering the "burning questions" about the impact of public programming.
One might hear "impact" used interchangeably with “effect” or “outcome.” We draw distinctions based on scale and complexity. By “impact,” we mean long-term, large-scale social or technological change. While any individual library program aims to achieve circumscribed effects and outcomes, public programs as a whole contribute indirectly to broader impacts. Impact accumulates over time, with many programs, many libraries, and many other community organizations making small contributions toward long-term, large-scale changes.
For example, the imagined program at the start of this article aimed explicitly to inform participants about starting a community garden. Learning that information is the effect of the program. Actually starting a community garden is a possible outcome of the program (and possibly other programs about growing zones and/or navigating local zoning laws). A more connected community is the accumulated impact of participants in various programs acting on what they learned.
In our idealized narrative, we imagined "you" choosing to greet your neighbors. They followed suit, and the behavior spread (many theories of media effects point to similar spreading behavior as an indirect result of repeated exposure). In reality, it usually takes more work.
"You" and other program participants might share what you learned with friends and neighbors, some of whom may choose to help. You might make common cause with people who attended other similar programs (about a variety of other community spaces, such as makerspaces) at a library or another organization working toward similar goals. You and your neighbors might succeed in establishing a community garden.
Others might succeed in establishing a community makerspace or an electronics recycling event or some other community space. As community members connect with one another in these new spaces, the feeling of connection will likely start to leak into their everyday routines. They’ll share a “Good morning!” with the neighbors they see as they set off for work, and you know the rest. Thanks in some small part to the library programs you and others attended, you are all en route to a more connected community — including neighbors who didn’t attend any library program at all. At least, that is the ultimate goal.
What kinds of (indirect) impacts might one expect from the range of public programming available at libraries across the country?
Including “more connected communities,” NILPPA derived nine impact areas toward which public programming at libraries might contribute. Table 1 below lists the nine impact areas, with brief definitions. The NILPPA website offers expanded definitions and examples drawn from programs at libraries around the U.S.
Communities become more... | Meaning that they... | Keywords |
| Connected | Promote social capital by helping their members establish the kinds of social connections, relationships, and networks that facilitate communal trust, cooperation, and reciprocity, along with collective action | Neighborliness, bonding, solidarity, shared values, social integration |
| Knowledgeable | Prioritize learning for its own sake, provide access to reliable information on a wide range of topics, and help their members develop the critical thinking capacities needed to make factual, evidence-based decisions | Thoughtful, deliberate, reasoning, understanding, informed |
| Creative | Provide opportunities and resources for artistic self-expression, enable the pursuit of creative livelihoods, and help their members participate in a manner of cultural productions | Artistic, nurturing, inventive, imaginative, productive |
| Civically Engaged | Promote a vibrant public sphere and encourage their members to play an active role in civic life and community governance — whether through participation in political processes, group and association membership, or community service | Democratic, participatory, transparent, responsible, dutiful |
| Healthy | Promote equitable access to healthcare, the attainment of healthy living and working conditions, and other actions that improve health outcomes and help all of their members lead physically and mentally healthy lives | Active, equitable, robust, safe, secure |
| Economically Vital | Seek to create the conditions that can lead to economic development, increased individual and collective financial wellbeing, and the equitable use of community goods, services, and natural and human resources | Fair, equitable, lively, prosperous, growing |
| Welcoming | Promote the inclusion of newcomers, value cultural diversity, and take actions that help all of their members feel a sense of belonging | Inclusive, nondiscriminatory, accessible, respectful, accepting, diverse |
| Joyful | Create spaces where people can come together to have fun, be entertained, enjoy each other's company, celebrate one another's lives, and affirm both their histories and the futures they are making together | Entertainment, amusement, pleasure, community spirit, pride |
| Caring | Promote prosocial values and behaviors, encourage their members to treat others with compassion and empathy, and help individuals develop a sense of social responsibility | Kind, just, tolerant, considerate, sharing, charitable |
We say that we “derived” these impact areas because they emerged from an iterative, bottom-up process. Through two workshops held with advisors whose expertise covers both library research and practice, we generated a list of 193 concrete, observable behaviors and events that could serve as indicators of the different impacts libraries want their public programming to have.
Knology researchers then sorted these indicators into thematic groups, whose precise meaning and scope were further refined through a review of relevant scholarly literature. Having delineated nine specific impact areas, we next asked programming librarians across the US to scrutinize these (and their related indicators) for descriptive and interpretive validity. Their responses, which we gathered through two surveys and a focus group, were largely positive: libraries agreed that the nine impact areas resonated with their programming goals and that the indicators were appropriate ways to measure progress within each impact area. Moreover, the impact areas derived by NILPPA parallel and expand on those focused on by other projects that aim to assess the impacts of library public programming.
How might one assess the impacts of library public programming?
At present, we know of two major approaches to assessing the impacts of library public programming. The Public Library Association’s Project Outcome exemplifies what one might call a micro-level approach, focusing on the effects and outcomes for program participants. IMLS took a macro-level approach in their study, Understanding the Social Wellbeing Impacts of the Nation’s Libraries and Museums (hereafter the "Social Wellbeing study"), focusing on nationwide trends in public data.
Project Outcome offers public, state, and academic libraries a free toolkit for distributing and analyzing standardized surveys for programs in eight areas that mostly overlap with the NILPPA impact areas. Topics include civic/community engagement, digital learning, early childhood literacy, economic development, education/lifelong learning, health, job skills, and summer reading. Post-program surveys help libraries understand the impact of their programs by asking participants to rate their agreement with four key outcomes: knowledge (did you learn something), confidence (do you feel more confident with the topic), application (will you use what you learned), and awareness (are you more aware of other library resources, programs, and services). Follow-up surveys, distributed to attendees 4-8 weeks after the program, allow participants to report whether they have indeed applied what they learned since participating in a given program. Project Outcome offers a straightforward and low-overhead approach to evaluating the effectiveness of specific programs on attendees. However, attendee surveys do not provide information on impacts to the broader community who did not attend but may still benefit indirectly.
The IMLS Social Wellbeing study assessed three types of social impacts—economic wellbeing, school effectiveness, and community health—that partially overlap with the NILPPA impact areas. Each impact area was measured using an index constructed from public data available at the county level, such as poverty and employment rates, median household and per-capita income, educational attainment, and percentage of households with investment income for Economic Wellbeing. The study reliably predicted the level of impact based on the extent of library and museum resources available in each U.S. county and the rates at which county residents used these institutions. The study also included case studies at twelve libraries and twelve museums around the U.S. The case studies corroborated the statistical findings and confirmed that both types of institutions play an active role in supporting education and public health in their communities. We point the reader to the IMLS report for extensive methodological details. Here we will point out two important aspects of the data. First, one of the variables used to create the library index was the number of programs held by libraries. A future iteration of this study could be redesigned to identify programming-specific effects. Furthermore, the three main datasets used for the Social Wellbeing study—the American Community Survey, the Stanford University Education Data Archive, and the Robert Wood Johnson Foundation’s County Health Rankings—are updated on a yearly basis, meaning that the analysis can be repeated to examine change over time.
How might one track the indirect impacts of library programming at the community, regional, and national levels?
Project Outcome and the Social Wellbeing study offer different views of library impact, one positioned at the micro-level and another that gives a bird’s-eye, macro-level perspective on the changes that libraries are making across the country. However, still missing is a meso-level view that bridges the outcomes for program participants and societal impacts.
Tracking public opinion may offer a meso-level view of public programming impacts. Public opinion is more than what people are thinking. Opinion informs what people are doing and saying. Thus, public opinion can have predictive power. For example, in economics, measures of consumer confidence or sentiment regularly serve as bellwethers for understanding the future direction of people’s spending and borrowing. The same might be true for the impact areas of library public programming. Using the example of our opening scenario: if you think you live in a connected community, your actions are likely to be friendly and prosocial, and will go about making your community more connected. In this way, tracking public opinion serves to track the route of indirect impacts from those who attended a library program outward to those with whom they later interact, and further still to their communities, regions, and, eventually, the nation as a whole.
Unlike our opening scenario, the behavior of one public program attendee, no matter how strong their opinion, would make little impact on the nation as a whole. Luckily, as cited above, 125 million Americans attended one or more library programs in 2019, and one in every 10 people who visited a library did so to attend a program. While all of these people attended a wide assortment of programs, these programs likely sort into one or more of our nine impact domains. That means each impact domain has what the “tipping points” literature calls a “tiny public.” Popularized by Malcolm Gladwell in The Tipping Point (2006), the concept of tipping points dates back to at least the 1950s, when it was first applied to describe the social phenomenon of “white flight” from U.S. cities. When a “critical mass” of people adopt a new behavior or product, then the rest of society eventually follows suit. This critical mass might be smaller than one’s intuitions might predict. Computational simulations found that a relatively small number of committed agents—the so-called “tiny public”—can tip the rest of society (perhaps as small as 10% of a population).
If the 125 million library users who attend library programs are committed to the libraries’ goals and willing to act on them (such as sharing what they learn at programs with other community members), they have the potential to create momentum and inspire change in communities.
Measuring libraries’ impact vis-a-vis public opinion, then, is a way of bridging the micro- and the macro-level perspectives on library-inspired social change. In addition to examining data on populations as a whole (as in the macro-level perspective), assessing library impact on only those individuals who attend programs (as the micro-level approach has done), researchers can investigate how these tiny publics act on what they hear and learn within those programs, thereby shifting the opinions of those around them after they leave the library. While attendees' behavior after a program is difficult to measure directly, the effects of that behavior on the opinions of those around them can be measured through polling.
What Happens Next
At present, we are planning a multilevel approach to tracking the ongoing impact of library public programming. On a first level, we would recruit a rotating cohort of libraries in all 50 of the United States. We would collaborate with the cohort to collect information on the programming these libraries offer. We would ask the libraries to share results from the data they collected through Project Outcome (a second level of tracking). On a third level, we'd conduct opinion polling with a 50-state sample of the general public on a rolling basis throughout the year.
The impact areas and indicators developed and validated as part of NILPPA Phase 2 would serve as a starting point for the development of brief survey instruments to gauge public opinion about library impacts. Integrating these data with public data like that used by the IMLS’ Social Wellbeing study (a fourth level), would bring into view both the state of the nine impacts at each of the four levels of assessment and the extent to which those impacts are growing across those levels—from the libraries implementing programs outward to the nation as a whole.
