Skip to Content

Awareness of Education Research Methods

A Guide to Resources: Understanding Validity in Education Research

Why is “validity” important in education research?

 

Remember the parable about the drunk man searching for his wallet under a streetlight? When someone comes to help, they ask “Are you sure you dropped it here?” The drunk says, “I probably dropped it in the street, but the light is bad there, so it’s easier to look over here.”

 

In science, when researchers look for answers where the data are better—rather than where the truth is most likely to lie—it has been called the “streetlight effect.” In education, it often seems like we are preoccupied with the convenient and well-lit. So, while it seems like we are drowning in education data, are they the data that we need to make sound decisions? (cited from: Shanker Blog — The Data-Driven Education Movement, Oct. 22, 2012.)


For educators to make sound decisions about education practices and policies, we must seek to better understand the foundations of teaching and learning which guide our profession. Yet, only through the exploration of ideas—using valid and reliable research—can these fundamentals be uncovered, and educators working across the board must share this responsibility.

The resources presented here will help educators become more familiar with the methods used to conduct reliable research in education and to better recognize standards needed to ensure the validity of the research findings. They are intended to help educators distinguish between independent, peer-reviewed, unbiased research, and the advocacy research that misinforms and is prevalent in educational research (NEA-NBI #86, 2012). Moreover, these resources will help educators to better assess whether or not claims made about education policies are validated with proper evidence.

This collection of resources includes a variety of scholarly works which span the past decade and includes the contributions of well-noted researchers, scholars and institutions that monitor research in education. As other valuable resources are identified for this collection, they will also be added to this repository.


How these resources are organized

These resources for education research are presented as a self-guided overview focusing on the soundness of education statistics and the applicability of research findings. They are organized around the most fundamental questions asked about education research: (1) How do I understand what the research in education says? (2) How do I judge if the research in education is valid and trustworthy? and (3) How do I know if education policy claims are based on valid research evidence? Each question is thoroughly addressed in non-academic language and interactive formats.

Understanding what the research in education says

In order to judge the validity of research findings in education, you will need a basic understanding of the methods used to gather and analyze information. These resources will guide you through simplified discussions on research designs, types of data and methods of collection, and the most commonly used statistical procedures in education research. These resources are provided in a variety of formats to enable the exploration of topics to varying degrees of depth.

Judging if the research in education is valid and trustworthy

Several resources are provided to help you assess the quality, coherence, applicability and educational significance of the research being considered for education policies. In addition, reputable sources provide thoughtful critiques of the most popular and recent studies commissioned by think-tanks and special-interest groups covering a variety of education issues.

Knowing if education policy claims are based on valid research evidence

Multiple formats are provided to help you critically examine the claims of education researchers and policy makers and determine the validity of their assertions. Several highly regarded sources include research briefs, comprehensive reports, books, articles, and websites, which present policy research and empirical research on numerous topics in education.


Understanding what the research in education says

Reports and Articles

  • Readings in Educational Research: A Program for Professional Self-Development (PDF icon PDF, 28.6 MB, 35 pgs.)
    (Sponsor/Authors: NEA — U. Casanova, D. Berliner, et al.)

    Abstract: Readings in Education Research (RER) is a professional self-development program which clarifies the benefits of reading educational research and focuses on how to make such readings easier for teachers. Unit One is presented here and provides an introduction to knowing how to read and interpret educational research. This packet is designed to help teachers learn to bridge the gap between their own practice and the research that contributes to a constantly expanding knowledge base in education.
  • Using Research and Reason in Education: How Teachers Can Use Scientifically-Based Research to Make Curricular and Instructional Decisions (PDF icon PDF, 228 KB, 47 pgs.)
    (Authors: P. Stanovich and K. Stanovitch

    Abstract: As professionals, teachers can become more effective and powerful by developing the skills to recognize scientifically based practice and, when the evidence is not available, use some basic research concepts to draw conclusions on their own. This paper offers a primer for those skills that will allow teachers to become independent evaluators of educational research.
  • Statistical Literacy Guide: How to Spot Spin and Inappropriate Uses of Statistics (PDF icon PDF, 73 KB, 6 pgs.)
    (Author: P. Bolton)

    Abstract: This guide deals mainly with how statistics are used, rather than how they are originally put together. Educators may not have time to investigate methods of data collection, survey methods, etc., but with the right approach they will be able to better understand how data have been used or interpreted. The same approach may also help the reader spot statistics that are misleading in themselves.

Websites

  • McRel Research Primer: How Do I Know What the Research Says?
    (Sponsor: Mid Continent Research for Education and Learning; and Education Commission of the States)

    Abstract: This primer covers the fundamentals of education research by using simplified language and everyday concepts to explain the various types of research designs, different types of data, and guiding principles that underlie the scientific quality of research. In addition, a tutorial is provided on how to read research reports and identify the strengths and weaknesses of different sources of data.
  • McRel Research Primer: Understanding Statistics
    (Sponsors: Mid Continent Research for Education and Learning; and Education Commission of
    the States)

    Abstract: The purpose of this tutorial is to explain basic statistical concepts commonly used in education research to help readers understand the results reported in quantitative education research. Readers will understand how statistics are merely words which describe the methods and rules for organizing and interpreting quantitative observations.
  • STATS: Stats Simplified
    (Sponsor: George Mason University)

    Abstract: The Statistical Assessment Service — STATS—is a non-profit, non-partisan organization with the goal of correcting scientific misinformation in the media and in public policy resulting from bad science, politics, or a simple lack of information or knowledge. In their work presented here, the meaning and use of statistics is laid our in non-academic language and in concepts that are easy to understand. Terms such as “margin of error” and “statistical significance” that are commonly used in media reports are explained anecdotally for greater clarity.

Judging if research in education is valid and trustworthy

Reports and Articles

Websites

  • McRel Research Primer: How Do I Know If the Research is Trustworthy?
    (Sponsors: Mid Continent Research for Education and Learning; and Education Commission of
    the States)

    Abstract: This primer discusses what makes research conclusions valid and knowing what information to gather about the research study to make a judgment: What is the research question? Does the research design match the research question? How was the study conducted? Are there rival explanations for the results?
  • Think Twice Think Tank Review Project
    (Sponsor: National Education Policy Center - NEPC)

    Abstract: Think Twice is one of the nation's first efforts to serve as a watchdog to review the research reports of “think-tank” organizations which focus on public education issues and policies. Reviewers for the project examine the reports for the validity of assumptions, methodology, results, and strength of links between results and policy recommendations. The reviews, written in non-academic language, are intended to help policy makers, reporters, and others assess the merits of the reports reviewed.
  • Best Evidence Encyclopedia
    (Sponsor: Johns Hopkins University - School of Education)

    Abstract: Reviewing program ratings can be a starting point for identifying an effective program to adopt in schools. The Best Evidence Encyclopedia (BEE) presents reliable, unbiased reviews of research-proven educational programs to help policy makers, principals, teachers, and researchers do their jobs. Educators can use the BEE to support their work on school improvement.
  • What Works Clearinghouse
    (Sponsor: U.S. Department of Education - Institute of Education Sciences (IES))

    Abstract: The clearinghouse reviews research on the different programs, products, practices, and policies in education and, by focusing on the results from high-quality research, analysts try to answer the question “What works in education?” The goal is to provide educators with the information they need to make evidence-based decisions.
  • NEA OPSCAN Surveys (log-in required; NEA Staff, Leadership and State Affiliates only)
    (Sponsor: National Education Association)

    Abstract: Are you searching for surveys to administer in your school or district which have strong research validity? The National Education Association’s OPSCAN Surveys Program assists state and local affiliates in carrying out scientifically based survey research. In collaboration with educators and other experts, the NEA has designed a portfolio of surveys with strong validity covering a broad range of education topics. The program assists affiliates with the analysis of the survey results and reporting. Results from these surveys help guide policy decisions for students and educators.

Books

  • When Can You Trust The Experts? How to tell good science from bad in education
    (Author: D. Willingham)

    Abstract: The purpose of this book is to offer a short cut, a work-around, so that one can size up the likelihood that claims like “all the research supports it” are actually true. The first half of the book focuses on the cues which tell us “this is probably true,” and how they can be misleading. By understanding these cues, it less likely that we’ll believe something that is false. The second half of the book describes the short cut, which consists of four steps: (1) strip it; (2) trace it; (3) analyze it; and (4) should I do it?
  • Scientific Research in Education
    (Sponsor: National Research Council -NRC)

    Abstract: This book describes the similarities and differences between scientific inquiry in education and scientific inquiry in other fields and disciplines and provides a number of examples to illustrate these ideas. Its main argument is that all scientific endeavors share a common set of principles, and that each field including education research develops a specialization that accounts for the particulars of what is being studied. The book also provides suggestions for how the federal government can best support high-quality scientific research in education.

Knowing if education policy claims are based on valid research evidence

Reports and Articles

  • Research-based Options for Education Policymaking
    (Sponsor: National Education Policy Center - NEPC)

    Abstract: This10-part brief takes up important policy issues and identifies policies supported by research. Each section focuses on a different issue, and its recommendations for policymakers are based on the latest scholarship. The topics include: teacher evaluation, dropout strategies, effective school expenditures, and public funding for school choice.

Websites

  • McRel Research Primer: How Do I Know If the Research Warrants Policy Changes?
    (Sponsors: Mid Continent Research for Education and Learning; and Education Commission of the States)

    Abstract: This primer helps policymakers decide whether and how the research should be used to influence education policy after making a judgment about whether the research results and conclusions can be trusted. The discussions cover state or local factors, including the cost of implementation, that influence policy decisions. In addition, it discusses how the quality and educational significance of the research should be considered.
  • Education Policy Alliance
    (Sponsor: A consortium of over 40 U.S. universities)

    Abstract: The Education Policy Alliance is a nationwide network of university-based research centers and organizations who collaborate and help guide the current focus of research in education. Members of this consortium provide high-quality research on the nation's most important education policy and practice issues. This website provides direct access to the individual research centers at each participating university or organization.
  • National Education Policy Center
    (Sponsor: University of Colorado — Boulder)

    Abstract: The National Education Policy Center (NEPC) sponsors research, produces policy briefs, and publishes expert third-party reviews of “think tank” reports. The Center’s publications are written in accessible language and are intended for a broad audience that includes academic experts, policy makers, the media, and the general public. Research briefs present original, blind peer reviewed research on significant policy questions.
  • Great Lakes Center
    (Sponsor: National Education Association and the State Education Affiliate Associations in
    Illinois, Indiana, Michigan, Minnesota, Ohio and Wisconsin)

    Abstract: The Great Lakes Center for Education Research and Practice supports and disseminates high quality research and reviews of research for the purpose of informing education policy and to develop research-based resources for use by those who advocate for education reform. The Center funds research briefs and reports on important education issues written by recognized academic experts.
  • The Consortium for Policy Research in Education — CPRE
    (Sponsors: U-PA, Harvard, Columbia, Stanford, U-MI, Northwestern, U-Wis-Madison)

    Abstract: The Consortium for Policy Research in Education (CPRE) is a team of analysts from seven major research universities who employ diverse and innovative methods in studying education policy, practice, and reform. CPRE has conducted extensive research on several topics, examining ways to bridge the gap and design policies that develop and sustain high-quality instruction.
  • The Annenberg Institute for School Reform
    (Sponsor: Brown University)

    Abstract: The Annenberg Institute for School Reform is a national policy-research organization that promotes quality education for all children, especially in urban communities. The Institute primarily focuses on school transformation, college and career readiness, and extended learning time. This work includes original research, analyses of education policies, and the development of tools for best practices.
  • Institute for Education Sciences — National Center for Education Statistics
    (Sponsor: U.S. Department of Education)

    Abstract: The National Center for Education Statistics (NCES) is the primary federal entity for collecting and analyzing data related to education in the U.S. and other nations. NCES addresses high priority education data needs; provides consistent and reliable indicators of education status and trends; and reports useful and high quality data to education policymakers and data users.
  • Education Research Information Center (ERIC) Database
    (Sponsor: Institute of Education Sciences - US Department of Education)

    Abstract: The Education Resources Information Center - ERIC- is an online digital library of education research and information which provides ready access to education literature. ERIC is intended to improve practice in learning, teaching, educational decision-making, and research. The library provides a comprehensive, easy-to-use, Internet-based bibliographic and full-text database of education research and information.
  • NEA Research Briefs (log-in required; NEA Staff, Leadership and State Affiliates only)
    (Sponsor: National Education Association)

    Abstract: The National Education Association (NEA) periodically prepares special briefs covering important research topics and major education issues. A particular emphasis is placed on identifying research which strengthens education policies which NEA supports. The categories of briefs include: Best Practices; Education Myth-Busters; Emerging Issues; and Strength of the Evidence.
  • Education Commission of the States — Research Studies Database
    (Sponsor: US state membership fees, contracts, and corporate grants)

    Abstract: The Education Commission of the States provides data, research, analysis and leadership to help states develop effective policy and practice for public education. The Research Studies Database provides trustworthy research that avoids potential bias and selective use of data, and presents straight forward discussions on research findings and policy implications.

Books

  • The Manufactured Crisis
    (Authors: D. Berliner and B. Biddle)

    Abstract: For over a decade this book has provided a strong foundation to debunk the myths that test scores in America’s schools are falling, that illiteracy is rising, and that better funding has no benefit. It shares the good news about public education. Disputing conventional wisdom, this book ignited debate in Newsweek, The New York Times, and the entire teaching profession.
  • Putting Research To Work in Your School
    (Authors: D. Berliner and U. Casanova)

    Abstract: This book presents research that can be immediately applied by the teacher in the classroom. The themes presented are: (1) teaching; (2) instructional strategies; (3) learning; (4) motivation; (5) school and society; and (6) testing.
  • Collateral Damage: How High Stakes Testing Corrupts American Schools
    (Authors: S. Nichols and D. Berliner)

    Abstract: This book is grounded in the application of Campbell’s Law, which states that the greater the social consequences associated with a quantitative indicator (such as test scores), the more likely it is that the indicator itself will become corrupted—and the more likely it is that the use of the indicator will corrupt the social processes it was intended to monitor. The book illustrates both aspects of this “corruption,” showing how the pressures of high-stakes testing erode the validity of test scores and distort the integrity of the education system.

 


OTHER NEA RESOURCES

  • anc_dyn_linksNEA: A Voice for GLBT Educators
  • anc_dyn_linksTeacher Evaluation: A Resource Guide for NEA Leaders and Staff (2011)
  • anc_dyn_linksNEA Teacher Evaluation and Accountability Toolkit (2011)
  • anc_dyn_linksNEA Teacher Recruitment Toolkit (2011)
  • anc_dyn_links2011 C.A.R.E. Guide: Strategies for Closing the Achievement Gaps
  • Family-School-Community Partnerships 2.0