Promises and Pitfalls of the “Research-to-Policy-Pipeline” Process - CONCEPT Professional Training

Rebecca Goodvin, Washington State Institute for Public Policy
Stephanie C. Lee, Washington State Institute for Public Policy

This article is concerned with evidence-based policymaking and the potential for research in psychology and other social science disciplines to contribute to this process. Policymaking is complex and incorporates many factors and voices. Rigorous empirical evidence is one potential factor in this process, but the degree to which it is built into, sought after, or accepted in policymaking varies considerably. From the vantage point of psychologists within a nonpartisan legislative policy research organization in the United States, we offer observations on building credibility, communicating effectively, and avoiding common pitfalls that researchers in psychology and other social science disciplines may encounter as they engage in this process. We also draw from our experiences in this setting to outline key features of research design and process that can enhance the policy relevance of research evidence. Finally, we highlight the field of early childhood education as one example of a long-term exchange between research and policy, briefly reviewing early work in this area before moving into recommendations for future directions to address policy-relevant questions.

evidence-based policymaking, research, early childhood education

“This article is concerned with the promises and pitfalls of evidence-based policymaking and the role of psychological research in this arena. The term ‘evidence-based policymaking’ generally means that, when possible, decision makers consider rigorous scientific evidence about programs, policies, and interventions when creating or revising policy. Many factors go into policymaking, including the voices of constituents, clients, advocates, stakeholders, and lobbyists. These voices often communicate their personal beliefs, conventional wisdom, public opinion, or individual anecdotes to legislators, who, in turn, may have their own relevant personal interests to consider. Policymakers also consider practical concerns, such as the broad economic context and budget negotiations. Another factor is the weight of rigorous scientific evidence that indicates that a certain policy or approach is likely to have the intended desired impact. Scientific evidence is not always available to inform policy decisions; when it is, it does not always receive careful consideration in the policymaking process. As such, we must acknowledge at the outset that we write this article from the privileged position of psychologists working in a policy research organization within a state in which evidence is often considered by policymakers, and sometimes results in changes to law and practice. The process of using evidence—as in all policymaking—is incremental and requires careful and concerted conversation between researchers and decision makers.
Nonetheless, over several decades, Washington State has forged a unique political culture, with regard for using evidence as part of the decision-making process, and has created a ‘research-to-policy pipeline’ to support that process. We acknowledge that this culture does not exist universally and that there are challenges to making it work. Despite this, we contend that although policy-minded psychologists may desire a more direct path from evidence to policy, and improvements can always be made, the process is working in many places and at multiple levels” (p. 490).

“The field of psychology, and the research that psychologists produce, is a critical component to evidence-based policy. At a broad level, “conceptual” research can bring attention to a new issue or move a background issue to a position of salience in the minds of policymakers. For example, growing awareness of research and scientific consensus on the importance of early brain development for later learning and behavior has moved to the forefront of discussion surrounding policies and programs affecting infants and young children” (p. 490-491).

“In short, evidence can inform the problem definition, the solutions to be considered, and the specific details of design, implementation, and management of programs. Difficult decisions about how best to allocate scarce resources might be informed by analyses of what policy or programming approaches are likely to yield the greatest return on investment. In contentious policy areas, rigorous empirical evidence can contribute to a shared set of facts, although legitimate differences in values and goals will remain” (p. 491).

“What is the role of psychologists in this process? Psychologists have a long track record of generating policy-relevant research that has successfully informed policymaking on a wide range of issues. A few examples, among many, include effects of state welfare reform efforts on children, approaches to sex education that are most effective in reducing teen-pregnancy rates, taking development into account in juvenile justice system reform, and the importance of attending to and improving child care quality. Psychologists are trained to conduct rigorous research with nuanced understanding of research design, and to operationalize and assess complex human behavioral constructs in a way that directly speaks to, or can translate into, outcomes of interest to policymakers. In short, researchers in psychology are well-equipped to understand effects of policy, programs, and practice on human behavior at the level of individuals, families, and communities” (p. 491).

“Researchers interested in advancing evidence-based policymaking would do well to become familiar with the landscape in which this work currently takes place, and with steps in the research-to-policy pipeline. The path from research to policy begins with setting a research agenda, and moves into generation of research evidence. From there, research findings are disseminated and then translated into a message that can directly inform policy. Findings may lead to policy action, which occasionally can include resetting the research agenda, and starting again at the beginning of the pipeline” (p. 491).

“Between academia and government, a variety of intermediary organizations operate at both the state and federal levels, and provide infrastructure supporting research contributing to all stages of the research-to-policy pipeline. These intermediaries include research and evaluation firms, philanthropic organizations, professional organizations, and government trade agencies, as well as government agencies themselves” (p. 492).

“Nonetheless, concerns about the challenges inherent in this complex process are also valid. As noted previously, policymakers consider many voices, perspectives, and motivations outside of empirical evidence… Rather than focusing on barriers, which have been discussed extensively in other sources, in the following sections, we draw from our experience working within a nonpartisan legislative research organization to describe what we have found to support a successful evidence based policy process. We focus on three main ideas for facilitating the research-to-policy pipeline: building credibility, communicating effectively, and being aware of and preventing ‘pitfalls’” (p.493).

Policy makers’ request answers to a wide range of questions, thus it is vital to use a research approach that best answers the respected question(s). Washington State Institute for Public Policy is known for its use of benefit-cost analysis, however several other approaches are appropriate for consideration such as meta-analysis, primary research/program evaluation, and summary/synthesis of descriptive information or polices. Each approach has it’s own strengths and drawbacks to consider when evaluating research strategies to inform the policy process.

“Conducting original empirical research aimed at a policy audience requires first ensuring that questions are designed to speak directly to social problems of interest to policymakers, and address interventions, programs, or policies that the state can realistically fund and implement. Assuming this foundation, we must carefully consider methodological issues including research design, sample selection, and measures. In this section, we offer observations around these methodological concerns, informed by our own experience. The issues are applicable to conducting primary research and evaluating research for synthesis, and inform the delivery of a message that can be made clear to policymakers” (p. 496).

“Nonetheless, limitations in existing work, along with new questions that have become paramount with program expansion, present opportunities for psychologists interested in conducting work that will be relevant and useful to current and upcoming policy problems” (p. 497).

“In particular, addressing program scale, long-term follow-up, and more systematic attention to theories of change and implementation are relevant issues across research and policy areas” (p. 500).

As always, please join the discussion below if you have thoughts or comments to add!