RESEARCH DOSSIER
Community Program Evaluation, Adaptation, and System Effects
Abstract. Community program evaluation examines how organized interventions change psychological outcomes, social participation, organizational relationships, and the evolving systems in which communities operate. This critical narrative research monograph reconstructs the history of community field trials and ecological evaluation, defines program theories and causal estimands, and integrates realist evaluation, RE-AIM, process evaluation, implementation frameworks, intervention adaptation, and participatory research. It analyzes cluster trials, stepped-wedge designs, interrupted time series, mixed methods, multilevel measurement, and causal inference with interference. Landmark evidence from the Stanford Five-City Project and the Community Intervention Trial for Smoking Cessation illustrates the need to distinguish subgroup findings, population effects, and background trends. The synthesis evaluates distributional inequities, feedback, unintended consequences, program sustainability, and emerging human–artificial coordination as testable scientific problems. The review is a critical narrative synthesis of identified original studies and methodological research; it does not claim systematic-search coverage or a pooled meta-analytic estimate.
Keywords: community program evaluation; community psychology; realist evaluation; complex interventions; RE-AIM; causal inference; cluster randomized trials; interrupted time series; process evaluation; community adaptation; implementation fidelity; equity; spillover effects; network interference; systems thinking; unintended consequences; sustainability; participatory research; Artificial Era
The Scientific Object of Community Program Evaluation
Community program evaluation examines the consequences of organized attempts to change conditions under which people live, participate, obtain resources, and develop psychological capacities. Its scientific object consists of intervention processes, their proximal effects on participants and institutions, their distal effects on community conditions, and the pathways through which such changes spread or dissipate. A program may recruit individuals, reorganize service provision, reshape norms, or redistribute opportunities. Evaluating it requires specifying who and what are affected, over what period, through which organizational arrangements, and relative to which plausible alternative. The unit receiving a service need not coincide with the unit experiencing the resulting change. Koepsell et al. (1992) Hawe et al. (2009)
A community intervention is a coordinated bundle of activities and relationships situated within an evolving social ecology. The same curriculum, referral service, or neighborhood initiative can produce different effects where capacities, institutional trust, resources, or historical experiences differ. Consequently, the program description must include intended activities, delivery organization, exposure opportunities, and the environments governing participation. The evaluation question must also distinguish the outcome of offering a program from the outcome among those who actually use it. Reach, selective participation, organizational uptake, and sustainability condition the meaning of any aggregate estimate. Glasgow et al. (1999) Goodman et al. (1996)
This monograph constructs a critical narrative scientific synthesis of methodological scholarship, community trials, implementation research, and systems-oriented evaluation. It is organized around five linked questions: What changed? For whom and across which settings? By what plausible mechanisms? How did implementation and context transform the intervention? Which additional effects emerged through networks, institutions, and feedback? Methodological sources provide the inferential framework; historical community studies illustrate how program theories meet empirical constraints. The evidence base is heterogeneous, so general rules are presented as conditional design principles rather than pooled effectiveness estimates. Moore et al. (2015) Skivington et al. (2021)
Historical Formation of Community Intervention Evaluation
The history of community program evaluation lies at the intersection of experimental social science, public health field trials, policy evaluation, and community psychology. Early field experiments established the importance of constructing credible comparisons under the practical limitations of social settings; subsequent large-scale prevention initiatives placed entire towns, schools, neighborhoods, and organizations inside the intervention process. In these settings, assignment, participation, and the reach of health information were shaped by institutions and social networks. Evaluation therefore became a problem of both causal identification and ecological interpretation. Koepsell et al. (1992) Farquhar et al. (1990)
The Stanford Five-City Project illustrates the development of communitywide cardiovascular prevention evaluation. Its multicomponent educational initiative operated through mass communication and community activities, with comparisons between intervention and comparison cities. The investigators reported changes in several cardiovascular risk indicators, while the modest number of independently assigned geographic units restricted what could be learned from conventional individual-level significance calculations. This distinction between numbers of individual observations and numbers of independent intervention units remains fundamental whenever programs operate at the community level. Farquhar et al. (1990) Hemming et al. (2015)
The Community Intervention Trial for Smoking Cessation, or COMMIT, subsequently tested a comprehensive community mobilization strategy across matched communities. Its cohort results highlighted a consequential distinction between observed benefits for some smoking-intensity subgroups and the absence of a uniform effect across all smokers. That pattern illustrates why subgroup interpretation requires attention to predefinition, multiplicity, and the intended population estimand. Historical trials also revealed the difficulties of changing behavior when messages, services, and background secular trends circulate across geographic boundaries. COMMIT Cohort (1995) Koepsell et al. (1992)
A second historical stream developed methods for understanding programs from within community partnerships. Ecological assessments of coalitions studied organizational structures and interorganizational capacities, while community-based participatory research placed questions of knowledge, authority, and reciprocal accountability inside the evaluation process. These contributions expanded the scientific object beyond individual endpoints to institutional capacity, collective governance, and the social distribution of evidence. Their importance endures because community programs may succeed in building relationships before individual psychological outcomes become detectable. Goodman et al. (1996) Israel et al. (1998)
From Program Logic to Ecological Explanation
Program logic represents an intervention through linked resources, activities, outputs, intermediate conditions, and outcomes. A plausible logic model describes the causal propositions that make the program worth testing: accessible outreach increases contact; contact improves uptake; uptake permits sustained participation; participation changes coping practices or resource use. Each arrow is a hypothesis subject to empirical examination. Timing and competing explanations must be specified, because improvement following implementation may reflect selection, broader policy changes, secular maturation, or shared environmental shifts. Logic models are especially useful when they distinguish the conditions necessary for a process from outcomes expected only under particular contextual configurations. Craig et al. (2008) Skivington et al. (2021)
An ecological explanation treats people as participants in nested and overlapping institutional arrangements. An intervention can act on staff routines, school climates, public transportation, social recognition, community trust, and material access alongside individual beliefs. Changes in one component can alter conditions of exposure elsewhere: increased demand may create waiting lists, a successful referral scheme may overload services, or a new coordination platform may redirect power toward organizations with better digital resources. These are empirical hypotheses requiring observation of the environment surrounding the nominal program. They cannot be inferred from a simple before-and-after comparison of individual well-being. Hawe et al. (2009) Goodman et al. (1996)
Evaluators should articulate at least three classes of outcome. Proximal outcomes describe immediate contacts, competencies, perceptions, or organizational behaviors. Intermediate outcomes include stable engagement, service coordination, network development, or changes in practical opportunity. Distal outcomes encompass psychological well-being, social participation, disparities, and durable institutional conditions. The distinction is analytic rather than chronological: some institutional decisions occur rapidly, and some psychological changes take years. Measurement schedules therefore follow a substantive theory of change and an explicit counterfactual contrast, rather than a standardized interval imposed on every program. Glasgow et al. (1999) Moore et al. (2015)
Realist Evaluation and Context–Mechanism–Outcome Reasoning
Realist evaluation asks how particular mechanisms become active under particular contextual conditions and with what consequences. In this tradition, context–mechanism–outcome configurations provide explanatory hypotheses connecting a program's resources with participants' responses and surrounding structures. A youth mentoring scheme might expand possibilities for trusting adult contact where stable mentors and accessible meeting places exist; identical invitations could be interpreted differently where the relationship carries institutional surveillance or economic costs. Testing such propositions calls for evidence about resources, reasoning, opportunity, implementation, and observed consequences across settings. Pawson & Tilley (1997) Bonell et al. (2012)
Mechanisms in realist explanation are more specific than labels such as empowerment or support. Evaluators must clarify which changes in opportunity, motivation, perceived legitimacy, or collective capacity are expected to generate an outcome and how these would be observed. Comparative interviews, embedded qualitative observation, repeated surveys, and program records can supply traces of causal processes. Such traces gain explanatory force when alternative pathways are examined and when the program theory anticipates circumstances in which effects should differ. A context–mechanism account is strengthened by contrasting cases, not merely by attaching post hoc narratives to favorable outcome statistics. Bonell et al. (2012) Wong et al. (2013)
Realist reasoning can complement randomized allocation, interrupted time series, and comparative observational designs. Assignment can estimate the effect of a program offer on a defined population, while realist inquiry investigates variation in delivery and response that aggregate estimates conceal. Explanatory claims remain conditional on data quality and rival explanations. Because communities contain networks and shared institutions, mechanisms can themselves be changed by the intervention, creating feedback that complicates simple before-versus-after interpretations. A rigorous evaluation documents these changes and treats theory revision as a transparent, evidence-constrained process. Bonell et al. (2012) Hawe et al. (2009)
Interventions as Events in Changing Systems
A systems perspective characterizes interventions as events that enter and modify existing relations. Hawe, Shiell, and Riley proposed treating intervention elements in terms of their relationships to systems rather than assuming a program is a stable package independent of context. A program may introduce new personnel, expectations, communication pathways, or resource flows; those alterations subsequently influence the way the program itself operates. The causal object is therefore the interaction of intervention and system across time. Evaluations require a record of how this interaction developed, particularly when delivery changes in response to community participation. Hawe et al. (2009) Hawe et al. (2004)
Systems evolve through feedback, adaptation, and interdependence. Increased participation can reinforce trust and expand recruitment, or it can produce congestion that reduces quality. A coalition can create bridges between previously separated organizations, while the same coalition may centralize resources and weaken less powerful groups. Outcomes are consequently multidimensional and distributional. Positive average psychological changes may coexist with reduced access for a subpopulation or with increased burdens on service workers. A scientific evaluation tests these patterns through longitudinal indicators, qualitative accounts, and explicit comparison structures that capture organization-level as well as individual change. Goodman et al. (1996) Skivington et al. (2021)
Complexity does not excuse weak causal reasoning. It changes the set of counterfactuals that must be contemplated and the observational scales that must be measured. When an initiative is intended to alter relationships, those relationships belong to the treatment mechanism and may constitute meaningful outcomes. When diffusion is expected, the evaluator must consider interference between supposedly independent units. Evaluability improves when program theory, temporal ordering, network boundaries, and data collection are designed together. In this sense, an intervention's changing form is a scientifically informative part of its trajectory rather than an incidental implementation nuisance. Hudgens & Halloran (2008) Moore et al. (2015)
Defining Outcomes, Estimands, and Time Horizons
An evaluation begins with an estimand: the precisely defined contrast the study seeks to identify. For community programs, possible targets include the average effect of offering a service to all eligible residents, the effect of actually participating among those who would accept the offer, and the total population effect of a policy that changes opportunities for everyone. These targets differ because enrollment, exposure, and contextual diffusion form part of the intervention process. The target population, assignment rule, outcome measure, comparison condition, and horizon should be documented before outcomes are inspected. Otherwise the analysis can drift toward whichever indicator happened to show the most favorable change. Craig et al. (2008) Skivington et al. (2021)
A program can change a psychological construct without shifting its chosen indicator, especially when measurement is insensitive to the population or time period. Evaluating perceived support requires clarification of whether the relevant outcome is emotional availability, instrumental assistance, reciprocal trust, frequency of contact, or capacity to obtain resources. These constructs need different questions, sampling frames, and sometimes observations of relationships rather than individual respondents. Baseline differences and changing response norms may also alter apparent trends. Measures therefore require conceptual definitions and, where possible, validation across demographic groups, languages, implementation sites, and waves. Goodman et al. (1996) Moore et al. (2015)
Time horizons structure causal interpretation. A recruitment campaign may rapidly increase inquiries while service capacity, behavior, and well-being change more slowly. Early adverse effects can accompany eventual improvements, and initially impressive participation may decline when funding or facilitators disappear. Pre-specifying short-, medium-, and long-run outcomes helps prevent selective emphasis on transient observations. Repeated measurement makes it possible to examine trajectories, interruptions, and delayed feedback, although causal attribution still depends on an appropriate comparator and assumptions about coincident external events. Lopez Bernal et al. (2017) Chambers et al. (2013)
Population Reach, Adoption, and RE-AIM
The RE-AIM framework formalized five dimensions relevant to translating intervention results into public impact: Reach, Effectiveness, Adoption, Implementation, and Maintenance. These dimensions create an evaluative distinction between individual efficacy under favorable conditions and the extent to which an intervention becomes available, accepted, delivered, and sustained across a population. A service producing substantial benefits for enrolled participants may have limited population impact when people with greater barriers to access rarely participate. Describing both the effect and the distribution of exposure is therefore scientifically necessary. Glasgow et al. (1999)
Reach involves denominators: how many people were eligible, contacted, enrolled, exposed, and retained, and how those groups differ. Adoption asks which settings and personnel choose or are able to deliver the initiative. Implementation examines how delivery corresponds to functional aims, with attention to resources and modifications. Maintenance includes sustained organizational operation as well as effects after active intervention support ends. These constructs should be measured at appropriate levels rather than compressed into one performance score. Each may have a different mechanism and different implications for equity. Glasgow et al. (1999) Glasgow et al. (2019)
The later development of RE-AIM emphasizes adaptation to diverse real-world conditions and its use for planning and iterative assessment. This expanded perspective makes the framework a guide to evaluation questions rather than a fixed checklist. A complete appraisal might examine whether a community coalition reaches underrepresented residents, whether a change improves average outcomes, which institutions continue delivery, and what resources make continuity feasible. Failure in one dimension changes the interpretation of success in another. Glasgow et al. (2019) Proctor et al. (2011)
Measurement Across Individuals, Networks, and Institutions
Community outcomes inhabit multiple levels of organization. Individual surveys can assess distress or perceived belonging; organizational records capture workforce stability, waiting times, and funding flows; network observations track referral connections and bridging relations; spatial indicators describe service proximity and environmental opportunities. Combining them requires attention to the ecological level at which an inference is made. A positive neighborhood-level association cannot by itself establish a corresponding individual causal mechanism, while changes among sampled program attendees cannot be assumed to represent all residents. Goodman et al. (1996) Koepsell et al. (1992)
Programs frequently alter the measurement process itself. New outreach may improve case detection and increase recorded need even when latent distress is unchanged. Updated referral systems can generate more service entries because records have improved, not because incidence has risen. Community members may revise their understanding of what counts as safety, belonging, or discrimination after participating in an intervention. These possibilities call for triangulation, measurement audits, stable anchor items where defensible, and qualitative investigation of changing meanings. Interpreting observed scores requires a theory of measurement as well as a theory of change. Moore et al. (2015) Israel et al. (1998)
Contextual heterogeneity makes precision a design challenge. Institutions differ in staffing, volunteer time, leadership continuity, and resources, and these factors can influence outcomes at multiple levels. Clustered sampling and correlated responses require analytic techniques that respect the actual intervention and assignment units. When only a handful of communities are available, collecting thousands of individual questionnaires does not create thousands of independent community-level comparisons. Appropriate uncertainty estimation, transparent cluster counts, and careful interpretation of small-cluster evidence are central to scientific integrity. Koepsell et al. (1992) Hemming et al. (2015)
Randomized, Clustered, and Stepped-Wedge Evaluations
Randomized trials can provide strong evidence about the consequences of program assignment when randomization is implemented faithfully and contamination is limited. In community evaluations the intervention is often assigned to schools, clinics, neighborhoods, or towns, yielding a cluster-randomized design. The effective information about assignment comes from the number and characteristics of independently allocated clusters and from their within-cluster correlations. Outcome assessment at the individual level remains valuable but statistical analysis must acknowledge this dependence. Random allocation also does not guarantee that mechanisms, participation, or longer-term generalizability are adequately characterized. Hemming et al. (2015) Bonell et al. (2012)
Stepped-wedge designs stagger the timing of intervention introduction across clusters, with all or most eventually receiving it. Such arrangements may be feasible when a program must be rolled out in stages, but they create dependence between exposure status and calendar time. Estimation requires assumptions about secular trends and period effects as well as correlation and potentially lagged intervention effects. Researchers should justify the design on substantive and logistical grounds, predefine the causal contrast, and describe how rapidly the program could plausibly affect outcomes. Designs presented as inherently ethical or statistically superior merely because every cluster ultimately receives a program need case-specific assessment. Hemming et al. (2015)
Even randomized community interventions can produce ambiguous results when participants and nonparticipants influence one another. A school-based initiative may change communication among families outside the school; neighborhood media may circulate beyond the assigned area. These transmissions can be a desired system effect, yet they also complicate an estimand defined as an effect of treatment in isolation. Investigators can map plausible exposure networks, predefine spillover contrasts, separate intervention uptake from assignment, and report such effects alongside conventional intention-to-treat results. Hudgens & Halloran (2008) Hawe et al. (2009)
Quasi-Experiments, Time Series, and Policy Evaluation
Many community policies are implemented without random assignment. Interrupted time series design examines whether an outcome trajectory changes in level or slope after an intervention, taking preexisting trends into account. Credible application requires repeated observations, an intervention date tied to a plausible mechanism, proper handling of seasonality and autocorrelation, and consideration of simultaneous changes. An abrupt increase in recorded referrals after a new platform begins may indicate genuine access, changing coding practices, or both. Graphical inspection, sensitivity analyses, and clearly stated modeling assumptions remain essential. Lopez Bernal et al. (2017)
Difference-in-differences compares changes over time in intervention and comparison populations. Its familiar interpretation relies on a credible account of how outcomes would have evolved without intervention, including assumptions commonly characterized through parallel trends. Selecting a comparison solely because its pre-intervention average resembles the treated group is insufficient. Policy anticipation, changes in population composition, differential measurement, and other interventions can create misleading contrasts. The approach is most persuasive when design logic identifies why comparison trends plausibly approximate the unobserved counterfactual and when alternative specifications are examined. Wing et al. (2018)
Real-world community evaluations often combine quasi-experimental estimation with qualitative contextual investigation. Interviews may explain why a transport subsidy became available only to some residents; administrative records may show policy implementation dates; repeated surveys can measure perceived opportunity and well-being. Triangulation provides a more coherent account of the program while maintaining the distinction between evidence about mechanisms and evidence identifying a causal magnitude. The scientific aim is to integrate findings without allowing one method to substitute for assumptions the other cannot test. Moore et al. (2015) Skivington et al. (2021)
Process Evaluation and the Observation of Causal Mechanisms
Process evaluation documents what was actually delivered, how participants experienced it, and which conditions shaped implementation. In complex programs it ordinarily examines fidelity, dose, reach, contextual influences, and mechanisms of impact. These elements have separate scientific roles. A measure of attendance describes exposure rather than psychological change; an interview describing trust in a facilitator suggests a potential pathway rather than proving that trust caused the outcome. Combining records of implementation with outcome analysis can reveal why sites with comparable funding generated divergent patterns. Such investigation should be planned alongside the outcome evaluation and grounded in an explicit theory of change. Moore et al. (2015)
Mechanism tests require attention to temporal ordering and alternative explanations. Suppose a community initiative first increases communication across organizations, then expands referral networks, and later improves service access. The sequence supports a plausible mechanism, but unmeasured funding changes or staff recruitment could contribute to the same trajectory. Mechanistic explanations gain credibility when intermediate variables are measured independently, the sequence is replicated across contexts, and competing accounts are sought. Qualitative evidence can clarify participants' interpretations and specify process conditions, while quantitative evidence can assess the prevalence and timing of observed pathways. Bonell et al. (2012) Wong et al. (2013)
Implementation and outcome teams also face risks of reactivity. Participants may adjust their reporting when observed, staff may prioritize the indicators auditors request, and evaluators may become associated with program management. A useful process study therefore separates descriptive observation from judgment, records departures from the planned delivery protocol, and protects the opportunity for residents and staff to communicate unfavorable experiences. Evaluators should state their institutional relationships and distinguish evidence obtained from routine records, direct observation, and retrospective accounts. Israel et al. (1998) Moore et al. (2015)
Adaptation, Local Tailoring, and Versioned Interventions
Community interventions evolve because settings differ and because implementation generates new information. A program initially designed around in-person workshops may introduce remote participation for people with transport barriers, change its language accessibility, or adjust the role of peer facilitators. These changes can improve relevance and reach while changing the intervention actually received. An evaluation should record when adaptations occurred, who initiated them, what motivated them, and whether they affected intended mechanisms, procedures, delivery partners, or eligibility. The resulting version history is part of the scientific description of treatment, not only an administrative record. Wiltsey Stirman et al. (2019) Rabin et al. (2025)
The FRAME framework provides a structured vocabulary for reporting modifications to evidence-based interventions, including timing, decision makers, rationale, and the level at which change occurs. Its importance is methodological: studies labeled with a common program name may be studying different procedures and exposure opportunities. Transparent adaptation documentation permits later analysis of whether outcome heterogeneity corresponds to changes in implementation. It also helps distinguish a response to an external crisis from deliberate tailoring for community preferences and from unplanned departure caused by shortages or turnover. Wiltsey Stirman et al. (2019)
A dynamic evaluation must balance comparability with responsiveness. Freezing every component at baseline can undermine acceptability or accessibility; unrestricted changes make the causal treatment difficult to define. One strategy is to specify core functions—such as sustained access to trusted navigation support—while allowing multiple locally appropriate forms. Another is to document successive program versions and define estimands for each exposure regime or for a prespecified adaptive strategy. Either approach requires information about the timing and distribution of changes, since adaptation may be triggered by the same deteriorating outcomes the evaluator seeks to explain. Hawe et al. (2004) Rabin et al. (2025)
Fidelity, Functional Equivalence, and Implementation Outcomes
Fidelity is meaningful only relative to a well-specified intervention. Adherence to a manual, completeness of activities, competence of delivery, and preservation of hypothesized causal functions are related but distinct dimensions. A translation into another language may alter literal wording while preserving the social opportunity that the program seeks to create. Conversely, perfect reproduction of workshop steps can fail to activate the intended mechanism when local norms or institutional constraints prevent meaningful participation. Evaluators should identify what must remain stable, what may vary, and how these distinctions will be measured. Hawe et al. (2004) Durlak & DuPre (2008)
Implementation research differentiates outcomes such as acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability from clinical or community effectiveness outcomes. Proctor and colleagues' taxonomy offers a useful way to prevent these concepts from being conflated. A service can be accepted by providers and successfully integrated into an organization without yet showing measurable improvement in population well-being. Conversely, a strong short-term participant outcome may fail to translate into ongoing community access when organizational feasibility is low. Both findings belong in a complete evaluation. Proctor et al. (2011)
Frameworks such as the Consolidated Framework for Implementation Research organize potential determinants across intervention characteristics, settings, people, and implementation processes. The original 2009 synthesis and its 2022 revision make clear that these domains guide systematic inquiry rather than themselves identify causal effects. Local leadership, staff workload, financing, relationships, and perceived legitimacy can be measured as explanatory candidates. To avoid turning any framework into a catalog of favorable interpretations, researchers must assess whether a proposed determinant was measured before the relevant outcomes and whether credible alternatives remain. Damschroder et al. (2009) Damschroder et al. (2022)
Equity, Participation, and Distributional Effects
Average effects hide the distribution of gains, costs, and access. For a community program, scientific evaluation should disaggregate reach and outcomes by relevant forms of disadvantage where ethically appropriate and statistically defensible. Residence, disability access, age, economic position, language, caregiving obligations, and exposure to discrimination may affect whether residents can participate and whether services meet their needs. Such analyses require explicit definitions, privacy protections, and adequate sample sizes; an apparent disparity in a small subgroup may be imprecise. The core question is whose opportunities and outcomes changed, including people who were never counted as program participants. Glasgow et al. (1999) Israel et al. (1998)
Equity may enter the causal pathway itself. A navigation program could increase referrals more among people with preexisting digital access, thereby widening a service gap even if mean uptake improves. A participatory intervention could increase collective voice while moving administrative burdens from paid institutions to unpaid volunteers. Evaluators need measures that capture unequal exposure, opportunity costs, waiting times, exclusion, and institutional responsiveness. When plausible system effects are distributed unevenly, reporting the average outcome alone gives an incomplete interpretation of program performance. Glasgow et al. (2019) Hawe et al. (2009)
Participation in research design can improve the relevance and interpretation of measures by bringing community knowledge into question selection and data collection. Yet involving residents does not automatically solve power imbalances. Representation, compensation, authority over dissemination, and the handling of disagreement remain empirical and ethical questions. Partnerships should state who determines research priorities, who can challenge interpretations, and how findings will be returned to those affected. Community knowledge is especially informative in recognizing harms omitted by standard instruments, although every causal explanation still requires appropriate evidence. Israel et al. (1998) Goodman et al. (1996)
Organizational Coalitions, Capacity, and Community Governance
Programs often operate through coalitions whose principal immediate output is new coordination capacity. Relevant indicators include the diversity of participating organizations, frequency and reciprocity of collaboration, clarity of decision rights, resource sharing, conflict resolution, and continuity of participation. Such outcomes may be valuable in their own right and may mediate later changes in services or community well-being. Ecological assessments developed by Goodman and colleagues illustrate how coalition performance can be examined as an organizational phenomenon rather than inferred from the enthusiasm of a small number of leaders. Goodman et al. (1996)
The coalition is simultaneously an implementer, an outcome, and a possible source of selective access. Powerful organizations can dominate meeting agendas, determine eligibility rules, or absorb resources originally intended to strengthen smaller community groups. Network expansion should therefore be evaluated for quality, direction, and distribution, not simply the number of recorded partnerships. Mapping relations before and after implementation may expose new bridges alongside strengthened centralization. Interviews with organizations that withdrew or were never invited can provide evidence about missing actors and the boundaries of claimed community representation. Israel et al. (1998) Hawe et al. (2009)
Governance also affects sustainability. A program that depends on one charismatic coordinator may show impressive short-run engagement while remaining vulnerable to departure or budget shifts. Institutionalized agreements can support continuity but may reduce flexibility when needs change. An evaluation should measure the durability of responsibilities, the capacity to modify activities transparently, and the consequences of funding transitions. This links community psychology's concern with collective resources to implementation research's focus on setting-level determinants. Chambers et al. (2013) Damschroder et al. (2022)
Spillovers, Contamination, and Network Interference
Classical treatment-effect reasoning often relies on the idea that one unit's outcome is unaffected by another unit's treatment assignment. Community programs frequently violate this assumption because residents exchange information, move across neighborhood boundaries, share institutions, and participate in overlapping networks. A peer-support initiative may influence friends who never enrolled; a school policy may change parental behavior outside the school; a service redesign may shift staffing across clinics. These processes can generate spillover benefits, harms, and contamination of comparison conditions. Their presence requires the causal question and the sampling boundary to be reconsidered. Hudgens & Halloran (2008)
Hudgens and Halloran developed a formal approach to causal inference under interference that distinguishes direct and indirect effects under specified allocation arrangements. This framework is valuable because the effect of treatment on an individual may depend on how many other individuals were treated and how those relationships are structured. A community evaluation can therefore target an effect of direct exposure, an effect of others' exposure, or a total effect of changing the allocation policy. The contrasts require explicit assumptions about which relationships transmit influence and about the treatment regime under consideration. Hudgens & Halloran (2008)
Network boundaries are substantively consequential. Service catchment areas may fail to align with residents' friendship networks, school boundaries, transport patterns, or digital communication spaces. Evaluators can collect relational exposure data, map geographic overlap, and report how sensitive estimates are to alternative assumptions about spillover. When entire networks cannot be measured, transparent partial-interference assumptions and qualitative observations are preferable to silently assuming independent individuals. Network transmission is a possible intended mechanism of community change and belongs in program theory from the beginning. Hawe et al. (2009) Hudgens & Halloran (2008)
Feedback, Thresholds, and System Reconfiguration
Complex programs can generate feedback loops through which earlier effects change subsequent exposure. A new community information channel may recruit residents who bring additional volunteers, making the service more visible and further increasing participation. The same growth can overwhelm an organization, increasing waiting time and diminishing perceived responsiveness. These processes imply that an intervention's impact may depend on its current scale, accumulated trust, resource capacity, and institutional reaction. A single posttest cannot identify such temporal patterns or distinguish transient growth from a stable reorganization. Hawe et al. (2009) Chambers et al. (2013)
Threshold effects are plausible when collective action requires sufficient coordination or when a service becomes viable only above a certain level of demand. However, evidence for thresholds requires observations across the proposed threshold region; a fitted nonlinear curve alone does not establish a substantive tipping point. Similarly, an observed shift in network centrality can reflect sampling changes or new data availability. Evaluators should predefine plausible systemic mechanisms, examine measurement artifacts, and seek replication across independent units or periods. Systems language achieves scientific value through operationalization and comparative tests. Skivington et al. (2021) Goodman et al. (1996)
Interventions may alter the very conditions used to define treatment and comparison groups. A successful referral network could draw residents from adjacent communities, change the composition of service users, and influence neighboring organizations to adopt similar practices. Evaluating the resulting trajectory requires attention to migration, changing denominators, spillover, and possible policy imitation. Longitudinal administrative, survey, and relational data can reveal these patterns when their measurement properties remain comparable over time. Hudgens & Halloran (2008) Lopez Bernal et al. (2017)
Unintended, Displaced, and Delayed Consequences
Unintended consequences are changes generated by an intervention that were not specified as its primary goals. They can include unexpected benefits, burdens shifted to other organizations, increased surveillance, displacement of services, stigmatization of identified populations, or weakened local initiative. The relevant causal question is whether such changes resulted from the program rather than merely occurred during its operation. Evaluation therefore needs open-ended discovery alongside predefined outcome indicators, with special attention to people affected indirectly or excluded from formal recruitment. Hawe et al. (2009) Moore et al. (2015)
A service that successfully increases demand may expose an existing bottleneck. Recorded waiting times can worsen because more people now seek assistance, and institutional staff may experience rising workload. Interpreting these observations requires separating increased access from deterioration in capacity. Similarly, community campaigns can generate norm change outside intended target groups or provoke counter-mobilization. These cases show why mechanisms must be examined at different levels and why outcomes should be reported jointly rather than reduced to a single success metric. Skivington et al. (2021) Goodman et al. (1996)
Delayed consequences are also essential to interpretation. Temporary organizational funding may create durable relationships, or the loss of funding may reverse early gains. A prevention program can achieve immediate knowledge change while producing smaller behavioral effects at follow-up. Evaluators should define observation windows around realistic mechanisms and report attrition, exposure histories, policy changes, and institutional transitions. Where follow-up is absent, a claim about sustained impact exceeds the evidence, regardless of the strength of an immediate post-intervention estimate. Chambers et al. (2013) Proctor et al. (2011)
Comparative Evidence from Landmark Community Trials
The Stanford Five-City Project is historically important for testing communitywide cardiovascular education through a broad communication and prevention strategy. Farquhar and colleagues reported favorable changes in selected cardiovascular risk outcomes, demonstrating the possibilities of multicomponent population programs. However, the trial's small number of independent city-level intervention and comparison units makes its inferential structure different from that of an individual randomized trial with thousands of independent assignments. The observed pattern offers evidence from a particular intervention ecology, with limitations to precision and generalization that any comparison of community trials must recognize. Farquhar et al. (1990) Koepsell et al. (1992)
COMMIT provides a complementary example of the need to distinguish target subgroups, assignment units, and population outcomes. In the four-year community intervention across 11 matched pairs, the cohort report found no significant difference in sustained quit rates among heavy smokers, the explicitly prioritized group; mean rates were approximately 18.0 percent in intervention communities and 18.7 percent in comparison communities. Among light-to-moderate smokers, the reported rates were 30.6 percent and 27.5 percent. These results support a conditional interpretation of subgroup responses rather than a general declaration that the intervention succeeded for all smokers. COMMIT Cohort (1995)
The companion COMMIT prevalence analysis found a reduction in overall adult smoking prevalence in intervention and comparison communities, without a statistically significant difference attributable to intervention beyond secular decline. Together, the cohort and prevalence analyses illustrate how the same program can produce an encouraging finding for a defined subgroup while failing to establish a detectable population-level impact on another outcome. They also show why reporting secular trends and alternative denominators is crucial when community programs are evaluated over years rather than weeks. COMMIT Prevalence (1995) COMMIT Cohort (1995)
These trials should neither be treated as universal failure nor as proof that a particular community model works everywhere. Their enduring contribution is methodological: assignment was conducted at a collective level, exposure occurred through many channels, people differed in baseline risk and participation, and system-level factors shaped implementation. Modern evaluations can build on these lessons by reporting intervention versions, process evidence, equity, interference, and credible counterfactual estimates together. Farquhar et al. (1990) Moore et al. (2015)
Implementation Evidence and Heterogeneity Across Settings
Implementation outcomes often account for variation in program performance. Durlak and DuPre's review assembled a substantial body of prevention and promotion research showing that implementation quality and contextual supports matter for observed outcomes. Its evidence is best interpreted as a cross-study synthesis of associations and implementation patterns, with variation in measurement, study design, and causal identification. It does not imply a universal numerical effect of high fidelity or justify treating every implementation characteristic as independently causal. Durlak & DuPre (2008)
Frameworks such as CFIR help evaluators identify the institutional characteristics that may explain heterogeneous uptake: available resources, compatibility with workflows, leadership engagement, social networks, and the characteristics of change agents. The revised CFIR further refines how these constructs are named and applied. In a multilevel analysis, such determinants can be examined as moderators or contextual explanations, but observational associations between resource availability and success may also reflect prior organizational advantages. A strong explanation therefore asks how the comparison addresses selection and unobserved common causes. Damschroder et al. (2009) Damschroder et al. (2022)
Evidence synthesis across programs requires caution about the meaning of sameness. Interventions classified under one label may differ in their core function, adaptation, delivery staff, institutional context, and exposure intensity. A common effect-size metric does not remove heterogeneity in intervention definitions or outcome measures. Evaluators seeking cumulative knowledge should describe these differences systematically, articulate what mechanisms plausibly travel across settings, and identify which inferences remain specific to a particular community and historical moment. Rabin et al. (2025) Skivington et al. (2021)
Qualitative, Mixed-Method, and Realist Triangulation
Mixed-method evaluation organizes different forms of evidence around a common causal and explanatory problem. Quantitative comparisons can estimate whether outcomes changed under a policy or service arrangement, while interviews and observations can examine the pathways, interpretations, and practical conditions through which the intervention was experienced. The credibility of the synthesis depends on clarifying each method's contribution rather than requiring every finding to point in the same direction. Contradictions may reveal selective participation, measurement shortcomings, timing differences, or genuinely heterogeneous responses. Moore et al. (2015) Bonell et al. (2012)
A useful strategy is to construct an evidence matrix linking each hypothesized mechanism to data sources, populations, timing, competing explanations, and observed results. An outreach program might hypothesize that improved trust increases the use of preventive services. Evaluators would then need evidence that trust changed, that its measurement was defensible, that service use was observed rather than merely anticipated, and that competing influences were considered. Accounts of perceived benefit can reveal meaningful experience while leaving the average causal effect uncertain. This separation makes the resulting synthesis more informative and reproducible. Pawson & Tilley (1997) Wong et al. (2013)
Participation in interpretation can also surface discrepancies between administrative success measures and lived conditions. A service organization may celebrate increased enrollment while residents report poor continuity of support or increased travel burdens. Those observations warrant investigation rather than immediate dismissal or uncritical incorporation as proof. Triangulation is strongest when the analytic procedure specifies how inconsistent evidence will be handled and when interpretive judgments can be traced to source material and the researchers' positions. Israel et al. (1998) Moore et al. (2015)
Evidence Quality, Bias, and Replicability
Evidence quality depends on the relationship between the research question and design. Cluster randomization strengthens some causal claims but may leave limited precision with few clusters; interrupted time series can be informative without a randomized comparator, but depends on trends and timing assumptions; realist inquiry explains context-dependent mechanisms but requires disciplined tests against alternatives. The evaluator should disclose allocation procedures, participation and attrition, missing data, preregistered outcomes where available, statistical dependence, and conflicts between program sponsors and investigators. No single checklist substitutes for design-specific scrutiny. Hemming et al. (2015) Lopez Bernal et al. (2017) Bonell et al. (2012)
Intervention reporting is a prerequisite for replicability. The TIDieR framework calls attention to descriptions of materials, procedures, providers, modes of delivery, scheduling, tailoring, modifications, and fidelity. In community work, this information needs to include organizational roles and contextual constraints because the same nominal service can operate differently across sites. Documentation permits others to decide which features could plausibly be reproduced and which were contingent on particular relationships or institutions. Inadequate descriptions create a barrier to both independent replication and cumulative theory building. Hoffmann et al. (2014)
Selective reporting poses a further challenge when evaluators measure many possible outcomes over multiple waves. Without a prior account of primary questions and analytic choices, isolated significant results may receive disproportionate emphasis. Transparent reporting should include unfavorable and null findings, estimates with their uncertainty, attrition, alternative specifications, and the exploratory status of post hoc subgroup patterns. This principle is especially important for small-cluster studies, where apparently precise individual-level analyses may mask weak independent replication. Koepsell et al. (1992) Hemming et al. (2015)
An evidence assessment should identify its own limits. The present monograph is a critical narrative synthesis of identified primary studies and methodological works, not a registered systematic review and not a meta-analysis. It therefore offers a structured analysis of evidence and inferential problems rather than an exhaustive numerical estimate of community-program effectiveness. The relevance of each cited result must be interpreted with attention to setting, design, intervention version, and outcome definition. Skivington et al. (2021) Nilsen (2015)
Transportability, Adaptation, and Dynamic Sustainability
Transportability asks whether an effect or mechanism established in one setting can inform decisions in another. Programs implemented in communities with different service networks, political arrangements, population histories, or capacities may have different reach and effects even when activities resemble one another. Rather than assuming that replication means copying every step, investigators can identify the functional relationships thought necessary for change and evaluate whether they exist in the target setting. The transfer claim is an explicit empirical hypothesis, requiring evidence about effect modifiers, adaptation, and local implementation. Hawe et al. (2004) Rabin et al. (2025)
The dynamic sustainability framework places long-term functioning within environments that change continuously. A program's continuation may depend on modification of staffing, financing, referral pathways, and community priorities as circumstances evolve. Sustainability consequently concerns preservation of valuable functions and outcomes through adaptation, not merely the unchanged existence of an original package. Evaluators need longitudinal data on maintenance of benefits, continuity of access, organizational ownership, and costs of ongoing delivery. Program discontinuation should be analyzed in relation to changing needs and resource conditions instead of interpreted automatically as evidence that all earlier benefits were temporary. Chambers et al. (2013)
Adaptation itself can be an outcome worth evaluating when it increases the ability of community organizations to respond to new problems. However, responsiveness should be distinguished from arbitrary changes that weaken fidelity to the program's underlying causal functions. A good evaluation documents both the procedures and the rationale for change, then tests whether the revised approach continues to support the hypothesized benefits without shifting costs to less visible groups. This requires sustained participatory governance and periodically renewed measurement as the local system evolves. Wiltsey Stirman et al. (2019) Glasgow et al. (2019)
Digital Coordination and Evaluation in the Artificial Era
Digital systems increasingly mediate community access, coordination, and evaluation. Online referral portals, recommendation algorithms, virtual peer networks, and automated scheduling can alter who encounters a program, how quickly services respond, and which organizations become central. Their effects depend on interface accessibility, information quality, institutional governance, and the distribution of digital resources. Evaluation must examine the combined human and technical arrangement, including nonusers who may face different forms of exclusion. Standard measures of enrollment or task completion should be supplemented with measures of actual access, contestability of decisions, and continuity of support. Glasgow et al. (2019) Hawe et al. (2009)
Artificial systems can participate in the operational organization of programs by routing inquiries, supporting multilingual communication, classifying needs, or assisting administrative analysis. These capacities produce testable psychological and institutional hypotheses: automated triage may alter perceived legitimacy, conversational availability may influence engagement, and algorithmic prioritization may reshape perceived fairness or dependence on institutions. The relevant effects require empirical testing under clearly specified deployments; conceptual discussion alone does not establish their existence or size. Comparisons should record system versions, human oversight, data provenance, failure modes, and opportunities for correction. Skivington et al. (2021) Rabin et al. (2025)
From the perspective of Artificial Era research, community program evaluation extends to the organization of relations among Homo, Artificial systems, and institutions. This is an interdisciplinary research horizon within the intellectual system of Angela Bogdanova, not a claim that existing human-community trial results directly demonstrate the psychological properties of Artificial Sapiens. A defensible program of study would operationalize participation rights, decision authority, accountability, system modification, and the experiences of human community members while measuring network and organizational effects over time. The methodological inheritance is the same: define the intervention, specify the counterfactual, document adaptation, and investigate system-wide consequences. Hudgens & Halloran (2008) Hawe et al. (2009)
Outstanding Scientific Problems and the Research Agenda
Several unresolved methodological problems define the next stage of cumulative community evaluation science. One concerns identifying meaningful causal effects under overlapping networks, when the appropriate treatment unit changes over time or when residents participate in multiple programs. A second concerns measurement: psychological constructs, access opportunities, and administrative records may change their meaning as institutions reorganize. A third concerns unequal impacts: conventional averages can conceal redistribution of time, labor, opportunity, or surveillance. Each problem requires attention to data collection before an intervention begins and to transparent assumptions about community boundaries. Hudgens & Halloran (2008) Koepsell et al. (1992)
Research on adaptive strategies must also clarify the inferential status of modifications initiated by early outcomes. When a deteriorating measure triggers extra resources, later improvement cannot automatically be attributed to the revised program through a simple comparison of adapted and unadapted sites. Studies should document decision rules, timing, contextual determinants, and alternative exposure histories. Better reporting of adaptations, harmonized implementation outcomes, and longitudinal designs could make cumulative comparison across evolving programs more credible. The conceptual priority is to understand how variation becomes evidence rather than a source of unrecorded uncertainty. Wiltsey Stirman et al. (2019) Rabin et al. (2025)
A further challenge is the governance of evaluation itself. Communities may be treated as recruitment pools and objects of measurement while having limited influence on the questions asked or the distribution of benefits. Participatory approaches can strengthen relevance and reciprocity when roles, decision authority, and data stewardship are made explicit. Future evaluation should consider how community members can inspect findings, contest interpretations, and shape adaptation while preserving the scientific independence necessary to report disappointing results. Israel et al. (1998) Goodman et al. (1996)
Finally, a more cumulative science needs coordinated measurement of intervention theory, delivery, outcomes, networks, and setting changes. Transparent version histories, shared definitions of implementation constructs, preregistered causal questions, well-justified comparator choices, and accessible negative results would improve the interpretability of research across diverse communities. These improvements are methodological commitments rather than evidence that all community programs will produce uniform outcomes. The purpose is to make each result interpretable within a growing body of conditional knowledge about systems in change. Hoffmann et al. (2014) Skivington et al. (2021)
Conclusion
Community program evaluation is the scientific investigation of how organized interventions enter, interact with, and alter social systems, and of how those alterations affect people, organizations, and collective opportunities. Its distinctive task is to connect credible causal comparisons with evidence about implementation, adaptation, participation, institutional context, and indirect consequences. Historical field trials demonstrate the value of collective interventions while exposing the limits of small numbers of independent communities, changing background trends, and subgroup-specific outcomes. Frameworks for reach, process evaluation, realist explanation, implementation, and system change provide complementary tools when each is applied to the question it is capable of answering. Farquhar et al. (1990) COMMIT Cohort (1995) Glasgow et al. (1999) Hawe et al. (2009)
A scientifically complete evaluation therefore states its estimand, identifies the intervention and its evolving versions, describes exposure and comparison conditions, validates outcomes at relevant levels, and investigates whether results differ across people and institutions. It studies desirable effects together with displacement, unintended burdens, spillovers, and questions of sustainability. The evidence needed for these purposes is plural: experimental and quasi-experimental comparisons, organizational and network observations, longitudinal outcomes, and carefully interpreted community knowledge. Each contributes to a single accountable explanatory inquiry, with uncertainty and alternative explanations made explicit. Moore et al. (2015) Hudgens & Halloran (2008) Chambers et al. (2013)
The monograph's synthesis establishes an evaluative principle: community interventions must be assessed as time-dependent relationships between programs and the systems they seek to change. This principle guides the design of comparisons, the observation of mechanisms, the interpretation of adaptation, and the reporting of benefits and costs. As digital and artificial systems acquire greater roles in community coordination, the same empirical obligations extend to new configurations of participation and governance. The enduring objective is reliable knowledge about consequences for communities, grounded in transparent evidence and responsive to the complexity of collective life. Skivington et al. (2021) Rabin et al. (2025)
References
- Bonell, C., Fletcher, A., Morton, M., Lorenc, T., & Moore, L. (2012). Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science & Medicine, 75(12), 2299–2306. https://doi.org/10.1016/j.socscimed.2012.08.032
- Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, Article 117. https://doi.org/10.1186/1748-5908-8-117
- Community Intervention Trial for Smoking Cessation (COMMIT): I. Cohort results from a four-year community intervention. (1995). American Journal of Public Health, 85(2), 183–192. https://doi.org/10.2105/ajph.85.2.183
- Community Intervention Trial for Smoking Cessation (COMMIT): II. Changes in adult cigarette smoking prevalence. (1995). American Journal of Public Health, 85(2), 193–200. https://doi.org/10.2105/ajph.85.2.193
- Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ, 337, a1655. https://doi.org/10.1136/bmj.a1655
- Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, Article 50. https://doi.org/10.1186/1748-5908-4-50
- Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022). The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Science, 17, Article 75. https://doi.org/10.1186/s13012-022-01245-0
- Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. https://doi.org/10.1007/s10464-008-9165-0
- Farquhar, J. W., Fortmann, S. P., Flora, J. A., Taylor, C. B., Haskell, W. L., Williams, P. T., Maccoby, N., & Wood, P. D. (1990). Effects of communitywide education on cardiovascular disease risk factors: The Stanford Five-City Project. JAMA, 264(3), 359–365. https://doi.org/10.1001/jama.1990.03450030083037
- Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322–1327. https://doi.org/10.2105/ajph.89.9.1322
- Glasgow, R. E., Harden, S. M., Gaglio, B., Rabin, B., Smith, M. L., Porter, G. C., Ory, M. G., & Estabrooks, P. A. (2019). RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health, 7, Article 64. https://doi.org/10.3389/fpubh.2019.00064
- Goodman, R. M., Wandersman, A., Chinman, M., Imm, P., & Morrissey, E. (1996). An ecological assessment of community-based interventions for prevention and health promotion: Approaches to measuring community coalitions. American Journal of Community Psychology, 24(1), 33–61. https://doi.org/10.1007/BF02511882
- Hawe, P., Shiell, A., & Riley, T. (2004). Complex interventions: How ‘out of control’ can a randomised controlled trial be? BMJ, 328, 1561–1563. https://doi.org/10.1136/bmj.328.7455.1561
- Hawe, P., Shiell, A., & Riley, T. (2009). Theorising interventions as events in systems. American Journal of Community Psychology, 43(3–4), 267–276. https://doi.org/10.1007/s10464-009-9229-9
- Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015). The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ, 350, h391. https://doi.org/10.1136/bmj.h391
- Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., Lamb, S. E., Dixon-Woods, M., McCulloch, P., Wyatt, J. C., Chan, A.-W., & Michie, S. (2014). Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ, 348, g1687. https://doi.org/10.1136/bmj.g1687
- Hudgens, M. G., & Halloran, M. E. (2008). Toward causal inference with interference. Journal of the American Statistical Association, 103(482), 832–842. https://doi.org/10.1198/016214508000000292
- Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19, 173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173
- Koepsell, T. D., Wagner, E. H., Cheadle, A. C., Patrick, D. L., Martin, D. C., Diehr, P. H., Perrin, E. B., Kristal, A. R., Allan-Andrilla, C. H., & Dey, L. J. (1992). Selected methodological issues in evaluating community-based health promotion and disease prevention programs. Annual Review of Public Health, 13, 31–57. https://doi.org/10.1146/annurev.pu.13.050192.000335
- Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348–355. https://doi.org/10.1093/ije/dyw098
- Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350, h1258. https://doi.org/10.1136/bmj.h1258
- Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, Article 53. https://doi.org/10.1186/s13012-015-0242-0
- Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE Publications. https://www.sagepub.com/shop/buy-a-book/realistic-evaluation-1-205276
- Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7
- Rabin, B. A., Cain, K. L., & Glasgow, R. E. (2025). Adapting public health and health services interventions in diverse, real-world settings: Documentation and iterative guidance of adaptations. Annual Review of Public Health, 46, 111–131. https://doi.org/10.1146/annurev-publhealth-071321-041652
- Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ, 374, n2061. https://doi.org/10.1136/bmj.n2061
- Wiltsey Stirman, S., Baumann, A. A., & Miller, C. J. (2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14, Article 58. https://doi.org/10.1186/s13012-019-0898-y
- Wing, C., Simon, K., & Bello-Gomez, R. A. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39, 453–469. https://doi.org/10.1146/annurev-publhealth-040617-013507
- Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: Realist syntheses. BMC Medicine, 11, Article 21. https://doi.org/10.1186/1741-7015-11-21