How Hattie’s Research Helps (and Doesn’t Help) Improve Student Achievement

Hattie Discusses What to Consider, Not How to Implement It . . . More Criticisms, Critiques, and Contexts

Dear Colleagues,


   By the time you read this Blog, I will have just landed in Singapore where I am one of six presenters at the World EduLead 2019 Conference [CLICK HERE] sponsored by the International Association for Scholastic Excellence (INTASE).

   During my week here, I will be presenting two full-day Master Classes, two Keynotes, and a Symposium with Michael Fullan (School Leadership), Carol Ann Tomlinson (Differentiated Instruction), and three other international education greats.

   Altogether, I will be presenting the following:

  • Seven Evidence-Based Strategies to Systemic Success in Schools
  • The Seven C’s of Success: Strengthening Staff Relationships to Ensure Student Success
  • School Reform: Strategic Planning, Shared Leadership, and Student Success
  • Helping Hattie Work: Translating Meta-Analysis into Meaningful Student Learning Outcomes

   While re-researching John Hattie’s work for last full-day presentation, I uncovered new “criticisms, critiques, and contexts” that motivated me to update at least two past Hattie Blog messages with this new one.

   My previous Hattie-focused Blogs include (along with a third “application” Blog):

July 21, 2018. Hattie Haters and Lovers: Both Still Miss the Effective Implementation that Practitioners Need. Critical Questions to Ask your “Hattie Consultant” Before You Sign the Contract

[CLICK HERE to Read]

_ _ _ _ _

September 25, 2017.  Hattie’s Meta-Analysis Madness: The Method is Missing!!! Why Hattie’s Research is a Starting-Point, but NOT the End-Game for Effective Schools

[CLICK HERE to Read]

_ _ _ _ _

August 26, 2017.  The Top Ten Ways that Educators Make Bad, Large-Scale Programmatic Decisions: The Hazards of ESEA/ESSA’s Freedom and Flexibility at the State and Local Levels


_ _ _ _ _

   As is evident from these titles, there are very specific concerns with Hattie’s research and how it is being used.

   In this Blog, we will describe these concerns in detail, and then discuss examples of how Hattie’s work can be effectively and defensibly implemented—from a science-to-practice perspective—for students, by staff, and in schools.

   To accomplish this, we will (a) briefly overview the concerns; (b) present a primer on meta-analysis; (c) quote from the concerns of three notable researchers; (d) discuss how to go from “effect to effective practice;” and (e) describe the questions to ask the “outside” Hattie consultant—before you hire him or her.

_ _ _ _ _ _ _ _ _ _

A Brief Overview of Concerns with Hattie’s Research 

   Over the past decade especially, John Hattie has become internationally-known for his meta-meta-analytic research into the variables that most-predict students’ academic achievement.  Indeed, some view his different Visible Learning books (which have now generated a “Hattie-explosion” of presentations, workshops, institutes, and “certified” Hattie consultants) as the books of an educational “Bible” that shows educators “the way” to succeed with students.

   As such, Hattie has assumed a “rock star” status. . . which creates an illusion that his work is “untouchable,” that it cannot be critiqued, and that it certainly can’t be wrong.

   As of this writing, Hattie’s research is based on the synthesis of over 1,500 meta-analyses comprising more than 90,000 studies involving more than 300 million students around the world.  In more statistical terms, Hattie takes others’ published meta-analyses—investigating, for example, a specific educational approach (e.g., cooperative learning) or intervention (e.g., Reading Recovery), and he pools them together—statistically conducting a meta-meta-analysis.

   In doing this, he averages the effect sizes from many other meta-analyses that themselves have pooled research that investigated—once again—the effect of one psychoeducational variable, strategy, intervention, or approach on student achievement.

_ _ _ _ _

   While the magnitude and sheer effort of what Hattie has done is impressive. . . there are a number of major methodological problems with his statistical approaches and interpretations; and a number of additional major science-to-practice implementation problems

   To foreshadow the more comprehensive discussion later in this Blog, below is an example of one of his primary methodological problems, and one of his primary implementation problems.

   Educators need to fully understand these problems in order to be able to benefit—especially on behalf of their students—from this research.

_ _ _ _ _

An Example of a Methodological Problem in Hattie’s Research

  One major methodological problem is that Hattie’s statistical analyses may be flawed

   More specifically, a number of notable statisticians (see the section on this below) have questioned whether the effect sizes from different independent meta-analyses can be averaged and pooled into a single meta-meta-analytical effect size—which is exactly what Hattie is doing.

   As such, they don’t believe that the statistical approach used by Hattie in his research is defensible. . . which means that some of his research results may be incorrect.

   Metaphorically, what Hattie is doing is akin to averaging the average temperatures for 100 years of each day in March. . . and then saying that the 100-year average temperature for March in, say, Washington, D.C. is 48 degrees (it actually is—I looked this up).

   While you can statistically calculate this, the conclusion—regarding the 48 degree average temperature—may not be functionally accurate or, more importantly, meaningful (if you are planning a trip to DC). 

   First of all, in a typical year, Washington, D.C.’s March temperature may range from 37 degrees on one day to 59 degrees on another day—a variance of 22 degrees.  So, even in looking at one year’s worth of March temperatures, you need to statistically address the temperature range during any specific month. . . and then you need to look at this variability over 100 years. 

   Given all of this, the 48 degree 100-year average clearly does not accurately tell the entire story.

   The “single” temperature is compounded by the fact that there may be different “micro-climates” in Washington, D.C.  Thus, the daily temperature on any one March 15th, for example, may be 45 degrees in the Northwest part of the city, but 52 degrees in the Southeast part.

   Finally, from year to year. . . over 100 years. . . there may be some seasons that are colder or warmer than others.  Not to get political, but if we were to factor in the impact of Global Warming, it may be that the most-recent 10-year March temperature is significantly warmer than the average temperatures for the 90 years before. . . and, therefore, more accurate and meaningful for our current needs.

_ _ _ _ _

   There is, at least, one additional embedded issue.  Measuring temperature is scientifically far more reliable and valid than the diverse measures used in different studies (or at different times in a school) to measure student achievement.  A temperature is measured by a thermometer, and most thermometers will give basically the same reading because they are scientifically calibrated instruments.

   With the meta-analyses used by Hattie, different researchers operationalize “student achievement” (as an independent outcome measure) in different ways.  Even if a bunch of them operationalize student achievement the same way, they still may use different measurement tools or metrics. . . that provide significantly different results. 

   Thus, the measurement of achievement is going to have far more variability from Hattie study to study than a thermometer in Washington, D.C. in March.

_ _ _ _ _

An Example of an Implementation Problem in Hattie’s Research

  The one major implementation problem that we will discuss right now is that, in a specific effect size area, educators need to know the implementation methods that were used in all of the studies included in the original meta-analytic studies that Hattie pooled into his meta-meta-analyses. 

   The point here is that, unless a program or intervention has been standardized in a specific effect area, and the same program or same intervention implementation steps were used in every study included in a meta-analysis or Hattie’s meta-meta-analyses in that area, it is possible that one implementation approach contributed more to the positive effect size on student achievement than another approach.

   For example, given Hattie’s current data, “cognitive task analysis” has a 1.29 effect size relative to positively impacting student achievement.  It is unlikely, however, that every study in every meta-analysis pooled by Hattie used the same step-by-step implementation process representing “cognitive task analysis.”

   Thus, Hattie’s research tells us what to consider (i.e., cognitive task analysis), but not necessarily the specific research-validated steps in how to implement it.

   For an individual school to implement the cognitive task analysis approach or steps that contributed most to the positive effect size that Hattie reports, its leaders need to know—statistically and relative to their implementation steps—what individual studies were integrated into the meta-analyses and Hattie’s meta-meta-analysis.

   But they also need to know which studies were done with the same type of students (e.g., gender, socio-economic status, race, geographical location, type and quality of school, etc.) that they are currently teaching in their school.

   That is, it may be that the students involved in the meta-analytic studies used by Hattie do not match the students in the schools that we are working with.  Thus, while the research used by Hattie may be “good” research (for some students in some schools in some communities), it may not be the “right” research for our students, schools, and community.

   To summarize so far:  If schools are going to use Hattie’s research in the most effective way for their specific students, a Multiple-gating process of decision-making must be used.

   This Multiple-gating Process should include:

  • Step 1.  Identify your school’s history and status, resources and capacity, and current positive and needed outcome relative to student achievement.
  • Step 2.  Determine which Hattie variables will most improve student achievement—with a constant awareness that many of these variables will interact or are interdependent.
  • Step 3.  Evaluate the methodological and statistical quality and integrity of the meta-analytic studies that Hattie included in his meta-meta-analyses.

NOTE:  If Hattie’s meta-meta-analysis has flaws or included flawed meta-analytic studies, identify the best separate meta-analysis studies and continue this multiple-gating process.

  • Step 4.  Evaluate the demographics and other background characteristics of the schools, staff, and students involved in the meta-analytic studies used by Hattie in his meta-meta-analyses to validate that they match the school demographics and background characteristics where you plan to implement the program, strategy, or intervention.
  • Step 5.  Using and analyzing Hattie’ best meta-meta-analytic study (or the best individual meta-analysis studies—as immediately above), identify what program(s) or strategy(ies), and what specific implementation approaches and steps were most responsible for the positive effects on student achievement.
  • Step 6.  Finalize the select of your program or strategy, and its implementation approaches and steps, and develop an Implementation Action Plan that identifies who will be involved in implementation, what training and resources they need, how you will engage the students (staff, and parents), how you will evaluate the short-and long-term student achievement outcomes, and what will be the implementation steps and timelines.
  • Step 7.  Resource, train, engage, implement, evaluate, fine-tune, implement, and evaluate.

_ _ _ _ _

   As we proceed to the next section of this Blog, let me be clear.  This Blog was not written to criticize or denigrate, in any way, Hattie on a personal or professional level.  He is a prolific researcher and writer, and his work is quite impressive.

   However, this Blog will critique the statistical and methodological underpinnings of meta- and meta-meta-analytic research, and discuss its strengths and limitations.  But most essentially, the focus ultimately will be on delineating the research-to-practice implications of Hattie’s work, and how to implementation it with students in the most effective and efficient ways.

_ _ _ _ _

   To this end, and once again, it is important that educators understand:


   While this all provides a “working outline,” let’s look at some more details.

_ _ _ _ _ _ _ _ _

A Primer on Meta-Analysis

What is it?



_ _ _ _ _

How it is done?


  • Identifying the program, strategy, or intervention to be studied
  • Completing a literature search of relevant research studies
  • Reporting and interpreting the meta-analytic results

   As with all research, and as reflected in the steps above, there are a number of subjective decisions that those completing a meta-analytic study must make.  And, these decisions could be sound, or they could be not so sound.  They could be defensible, or they could be arbitrary and capricious.  They could be well-meaning, or they could be biased or self-serving. 

   Thus, there are good and bad meta-analytic studies.  And, educators are depending on the authors of each meta-analytic study (or, perhaps the journal reviewers who are accepting the study for publication) to include only those studies that are sound.

   By extension, educators also are depending on Hattie to include only those well-designed and well-executed meta-analytic studies in his meta-meta-analyses.

   But, unfortunately, this may not be the case.

   In his 2009 Visible Learning book, Hattie states (pg. 11), “There is. . . no reason to throw out studies automatically because of lower quality.”

   This suggests that Hattie may have included some lower quality meta-analytic studies in some (which ones?) of his many meta-meta-analyses.

   Indeed. . . What criteria did he use to when including some lesser-quality meta-analytic studies?  How did he rationalize including even one lower quality study?  But—most importantly—how did these lower quality studies impact the results of the effect sizes and functional implications of the research?

   These are all important questions that speak directly to the educators who are trying to decide which Hattie-endorsed approaches to use in their pursuit of improved student achievement scores.  These questions similarly relate to educators’ decisions on how to effectively implement the approaches that they choose.

_ _ _ _ _

How do you Interpret an Effect Size?

   As noted above, Hattie (and other researchers) use an effect size of 0.40 as the “cut-score” or “hinge point” where a service, support, strategy, program, or intervention has a “meaningful” impact on student achievement.

   Visually, Hattie represents the continuum of effect sizes as a “Barometer” (see below).

Image title

   But this doesn’t tell the entire story.  In fact, some researchers are very uncomfortable with this barometer and how Hattie characterizes some of the effect sizes along the continuum.
_ _ _ _ _

   Matthew A. Kraft, from Brown University, is one such researcher.  In his December, 2018 working paper, Interpreting Effect Sizes of Education Interventions, Brown identified five guidelines when interpreting effect sizes in education.  These guidelines are summarized below.

[CLICK HERE for this paper]

Kraft’s Effect Size Interpretation Guideline #1.  The results from correlational studies, when presented as effect sizes, are not causal effects.  Moreover, effect sizes from descriptive and correlational studies are often larger than causal studies.

   Implication.  When Hattie reports that a specific program or intervention has a strong effect size relative to student achievement, he is most often meaning (a) that there is a contributory (not causal) relationship between the program and student achievement; and thus (b) that students experiencing that program have a higher probability of increasing their achievement when compared with students who have not experienced that program. 

   Once again, he is not suggesting that the program caused the achievement.  In fact, the program might be enhancing another factor that actually is more directly causing the increase in student achievement.

   Example.  Hattie’s most up-to-date November, 2018 research concludes that Phonics Instruction has a strong 0.70 effect size on student achievement.

  Hypothetically (yet, most probably), let’s say that most of the research contributing to this effect size involved correlational studies that compared Kindergarten through Grade 3 students who learned to read through phonics-based literacy instruction with students who were taught using other literacy instruction approaches.  These studies then, for example, looked at all of these students’ high school Grade Point Averages (GPAs) when they graduated from 12th grade.

   Per Hattie, the meta-meta-analytic result of pooling all of the separate meta-analyses of the hypothetical studies above was a 0.70 effect size.

   But what does this mean?

   A 0.70 effect size technically means that:

  • 76% of the K to Grade 3 phonics-taught students’ high school GPAs will be above the average high school GPA of the K to Grade 3 non-phonics taught comparison group. 
  • In addition, 73% of the two groups’ GPAs will overlap, and there is a 69% chance that a student picked at random from the Phonics Instruction group will have a higher GPA than a randomly-chosen student from the Non-Phonics Instruction group. 

The latter percentage is called the “probability of superiority.” 

  • In the end, for every 100 students receiving Phonics Instruction in K to Grade 3, 24 of these students will have a higher GPA than if they had not received phonics instruction.

[CLICK HERE to see a great interactive effect size calculator where you can input the effect size, and its meaning (as immediately above) is automatically described]

_ _ _ _ _

   As you can see from the description above, an effect size here is correlational and not causal.  If the phonics instruction-to-student achievement relationship were causal, 100% of the students receiving that approach would out-perform (i.e., have higher high school GPAs) 100% of the non-phonics-taught students.

   But, finally, let’s reinforce the earlier point about the potential presence of a “third” factor or variable that may better explain the initial effect size relationship.

   What if—as is possible—students who learn how to read using phonics instruction demonstrate higher levels of reading comprehension from Grade 3 through Grade 12 than students where phonics instruction was not used.

   In this scenario, the (a) phonics instruction impacted (b) students’ comprehension which helped them (c) to understand and learn material more easily, which helped (d) increase their GPA.  Thus, the initial phonics instruction, for many students receiving it, “triggered” a chain of events that more fully explains the “static” 0.70 phonics instruction-to-student achievement effect size.

   Critically, this (as above) is what educators need to understand and think through when they read a Hattie effect size. . . or when it is presented in  a Hattie workshop or cited by a “Hattie Consultant.”

   There is critical context here.  Hattie’s effect size posters or charts are not restaurant menus.  Once again, they largely tell us what to consider, but there is so much between consideration and implementation.

_ _ _ _ _

Kraft’s Effect Size Interpretation Guideline #2.  The magnitude of effect sizes depends on what outcomes are evaluated and when these outcomes are measured.

   Implication.  As we have already discussed, it is important to know how different researchers defined and measured “student achievement” when they conducted their research on a specific strategy, program, or intervention.  In addition, it is important to know when and how often they measured for the “student achievement” in their study.

   Finally, it is important to know how many different student achievement definitions, measures, and data collection timeframes are represented in any single meta-analysis. . . and then how these differences were pooled when Hattie took different meta-analyses and included them in his meta-meta-analysis.

   Some basic meta-analytic results here are:

  • Expect outcomes affected by short-term decisions and effort to be larger than outcomes that are the result of cumulative decisions and sustained effort over time.
  • Expect outcomes that are more closely aligned with the intervention to have larger effect sizes than those that measure broader or more generalized outcomes.
  • Expect outcomes measured immediately after the intervention to have larger effect sizes than outcomes measured later.

   This all means that schools should not implement some strategies or programs when the research underlying their reported effect sizes do not match the school’s desired implementation outcomes or intended timeframes. 

   Example.  If schools are looking for longer-term (e.g., over the course of an entire school year) and more global (e.g., state test score proficiencies) outcomes than reflected in the Hattie research for a specific effect size, they probably should “pass” on the strategies or programs involved, and look at research studies that more closely mirror their goals.

_ _ _ _ _

Kraft’s Effect Size Interpretation Guideline #3.  Effect sizes are impacted by subjective decisions researchers make about the study design and analyses.

   Implication.  We addressed this Guideline earlier when we discussed the effective size results for Cognitive Task Analysis” and concluded that it was unlikely that every study in every meta-analysis pooled by Hattie used the same step-by-step cognitive task analysis implementation process.

   The functional meta-analytic outcomes here include:

  • Expect studies with more targeted samples, more homogeneous, and less representative samples to have smaller variation in their outcomes and, thus, lar
  • Expect studies to have larger effect sizes when control groups do not have access to resources or supports similar to the intervention.
  • Expect studies that report the effect of offering an intervention (e.g., tutoring) to have smaller effect sizes than studies that report the effect of receiving a more-targeted intervention.

   Another Example.  Hattie reports that Behavior Interventions have a .62 effect size for student achievement.  But exactly what specific behavioral interventions (of the twenty or more available) are represented (or most represented) in the research included in the meta-analyses that Hattie pooled into his meta-meta-analysis?

   Moreover, were the students involved in these studies general education students, students with disabilities (and, then, which disabilities), alternative education students, or juvenile justice-placed students? 

   At what age or grade level were the students? 

   Did the studies use a Comparison Group, a Matched Control Group (that received no special staff interactions), or a Matched Attention-Control Group (that received the same amount of attention as the Intervention Group, but “neutral” or non-intervention-based attention)?

   All of this information is critical because it allows educators both to evaluate the integrity of the research that contributes to any Hattie effect size, and to validate that the students involved in the research match the students in their schools who they are targeting for the program or intervention.

_ _ _ _ _

Kraft’s Effect Size Interpretation Guideline #4.  Strong program or intervention effect sizes must be covaried with how much it costs to implement the program or intervention—both relative to the initial start-up and ongoing maintenance.

   Implication.  While most researchers do not report the costs or cost-benefits of their programs or interventions, this is an important consideration for educators in the field.  To estimate the costs, educators can review the individual studies that contributed most to the meta-analytic or meta-meta-analytic results as these studies’ implementation methods and steps can suggest how expensive a program or intervention will be to initiate and then maintain.

   Indeed, while it’s nice to know that specific programs or interventions have a strong impact on student achievement, if they are too expensive to implement or maintain, the strong effect size may be less relevant.

   These implementation characteristics are even more important given our earlier discussion of exactly what an effect size means—relative to the probability of a program or intervention impacting specific percentages of participating (or receiving) versus non-participating (or non-receiving) students.

   Thus, in the end, schools may do well to select programs or interventions that have smaller (but significant) effect sizes, but that can be implemented and maintained in the most cost-effective ways.

   Example.  Rogers and Feller (2018) found that delivering personalized information to parents about their students’ absenteeism reduced total absences by 6%.  Student absences obviously connects with many, many Hattie effects that can only occur if students are attending school.

   The personalized information intervention cost between $1 and $15 per student. 

    The question ultimately is:  Does the cost of this intervention and the predicted number of students positively impacted represent a good “return-on-investment” for the school—in the context of everything else that it is doing and funding?

_ _ _ _ _

Kraft’s Effect Size Interpretation Guideline #5.  The ease or difficulty in scaling-up a program or intervention also matters when evaluating the policy relevance of effect sizes.

   Implication.  Adding to the cost-effectiveness (See Guideline #4) of a program or intervention is its complexity and/or difficulty relative to scaling it up across an entire group of students, school, or district

   This, once again, can only be determined by analyzing the individual studies that contributed most to a program or intervention’s meta-analytic or meta-meta-analytic results. 

   More specifically, programs are unlikely to scale with fidelity and maintain their effectiveness if they are (a) only effective with a narrow population; (b) entail substantial behavioral changes (both of staff and students); (c) require a skill level greater than that possessed by the typical educators in a school; (d) face considerable opposition among the staff, students, and/or parents; (e) are prohibitively costly; or (f) depend on the charisma of a single person or a small group of highly-trained and dedicated individuals.

   Example.  Hattie has calculated a strong effect size (0.88) for the impact    Micro-teaching and Video Review of Lessons on student achievement.  At the same time, the research suggests that these approaches improve the effectiveness of teacher instruction, and this instruction is what has the more direct effect on student achievement.

   But beyond this, statewide teacher coaching programs are expensive and staff-intensive, they are difficulty to sustain (both relative to budget and effort), and they have not resulted in the large student achievement gains that have been documented in more controlled and “small-scale” studies.

   This, then, is a good example of where the research does not translate (as hoped) into practice.  Educators need to understand not just the characteristics of the research that contributed to the large effect size, but also the research showing that, pragmatically, it is not easy to scale-up these approaches to have large, systemic success.

_ _ _ _ _

Summary: What Questions should Educators Ask when Reviewing a Meta-Meta-Analytic Result?


   And so, educational leaders need to independently evaluate the results of any reported meta-analytic research—including the meta-meta-analytic research discussed by Hattie—before accepting and taking action on the results.

   Among the questions that leaders should ask when reviewing (or when told about the results from) Hattie’s or others’ meta-analytic research are the following:


      I have “asked and answered” these questions in my September 25, 2017 Blog.


   Clearly, these questions must be answered for the individual meta-analyses used in any meta-meta-analytic study.

_ _ _ _ _ _ _ _ _ _

Others’ Concerns with Hattie’s Research 

   While we have expressed, directly or indirectly, some of the primary concerns with Hattie’s research above, it important to include two additional voices so that educators have a full understanding of the concerns.

   In a past Blog, we discussed the concerns of Dr. Robert Slavin from John’s Hopkins University.  We will summarize his concerns below, and then add the perspectives of Drs. Pierre-Jerome Bergeron and Lysanne Rivard (from the University of Ottawa and McGill University, respectively) who wrote an article in the McGill Journal of Education titled, “How to Engage in Pseudoscience with Real Data: A Criticism of John Hattie’s Arguments in Visible Learning from the Perspective of a Statistician.”

_ _ _ _ _

Robert Slavin’s June, 2018 Concerns with Hattie’s Work






_ _ _ _ _

   On July 17, 2018, Slavin added to his critique—responding to Peter DeWitt who defended Hattie’s approaches.  In this Education Week Letter to the Editor, he stated:


_ _ _ _ _

Bergeron and Rivard’s 2017 Concerns with Hattie’s Work

  While Slavin provides an important research and practitioner perspective, Drs. Pierre-Jerome Bergeron and Lysanne Rivard provide an important statistical perspective.

[CLICK HERE for their original article]

   Summarizing their McGill Journal of Education article, “How to Engage in Pseudoscience with Real Data: A Criticism of John Hattie’s Arguments in Visible Learning from the Perspective of a Statistician,” they make the following points:

  • Hattie’s meta-meta-analyses ignore the presence of negative probabilities; He confounds correlation and causality.
  • Hattie believes that Effect Sizes from separate meta-analytic studies can be compared because Cohen’s d is a measure without a unit/metric; his averages, therefore, do not make sense.
  • In conducting meta-meta-analyses, Hattie is comparing Before Treatment versus After Treatment results, not (as in the original meta-analyses he uses) Treatment versus Control Group results.
  • Hattie pools studies that have different definitions (and measurements of) student achievement, and treats them as one and the same.
  • Hattie believes that effects below zero are bad. Between 0 and 0.4 we go from “developmental” effects to “teacher” effects. Above 0.4 represents the desired effect zone. There is no justification for this classification.

   For people interested in more detail regarding these points, please go to the original article.

   But before you go, here are some of the comments you will read:

The work of John Hattie on education contains, seemingly, the most comprehensive synthesis of existing research in the field. Many consider his book, Visible Learning, to be a Bible or a Holy Grail.
For those who are unaccustomed to dissecting numbers, such a synthesis does seem to represent a colossal and meticulous task, which in turn gives the impression of scientific validity. For a statistician familiar with the scientific method, from the elaboration of research questions to the interpretation of analyses, appearances, however, are not sufficient. The researcher from New Zealand obviously has laudable intentions, which we describe first and foremost.  Good intentions, nevertheless, do not prevent major errors in Visible Learning.
In reading Visible Learning, anybody who is knowledgeable in statistical analysis is quickly disillusioned. Why? Because data cannot be collected in any which way nor analyzed or interpreted in any which way either. Yet, this summarizes the New Zealander’s actual methodology.
Fundamentally, Hattie’s method is not statistically sophisticated and can be summarized as calculating averages and standard deviations, the latter of which he does not really use. He uses bar graphs (no histograms) and is capable of using a formula that converts a correlation into Cohen’s d (which can be found in Borenstein, Hedges, Higgins, & Rothsten, 2009), without understanding the prerequisites for this type of conversion to become valid.

_ _ _ _ _ _ _ _ _ _

My PerspectiveHow Do You Go from Effect to Effective Practice?

   In the most-current (October, 2018) version of Hattie’s Visible Learning effect sizes, Hattie has organized more than 250 variables into clusters that include: Student, Curricula, Home, School, Classroom, Teacher, and Teaching. 

   In the Figure below, I have listed the top eight effect sizes with their respective “Areas of Research Focus.”

   I have also added a descriptor identifying whether each variable can be changed through an external intervention.  Thus, I am saying that “Students’ Self-Reported Grades,” “Teacher Estimates of Student Achievement,” and a “Teacher’s Credibility with his/her Students” cannot be changed in a sustained way through some type of intervention, and that—even if they could—they would not causally change student achievement.

   Parenthetically, in most cases, these three variables were independent variables in the research investigated by Hattie.

Image title

   At this point, we need to discuss how to go from “effect to effective practice.”  To do this, we need to understand exactly what each of the variables in the Figure actually are.

   And . . . OK . . . I’ll admit it. 


   To begin to figure this out, we would need to take the following research-to-practice steps:

  • Translate these methods into actionable steps, while also identifying the provide the professional development and support needed for sound implementation
  • Implement and evaluate the short- and long-term results

   If we don’t do this, our districts and schools will be unable to select the best approaches to enhance their student achievement and implement these approaches in the most effective and efficient ways?

   This, I believe, is what the researchers are not talking about.

_ _ _ _ _

The Method is Missing


  • Response to Intervention (Effect Size: 1.09)





_ _ _ _ _

   Hattie’s meta-analytic research recently added Interventions for Students with Learning Needs.


   Previously, Hattie identified Comprehensive Interventions for Learning Disabled Students as having one of the five top effect sizes relative to impacting student learning and achievement. 



_ _ _ _ _

   Even though this construct is now gone, if a school or district was still using this book—from a research-to-practice perspective—once again, there is no specificity here.  No one reading Hattie’s books—or looking at the Corwin Press website—would have a clue as to where to begin to operationalize a sound implementation process.

   More specifically:  Hattie described Comprehensive Interventions for Learning Disabled Studentsin the plural.




   But now, we have an additional dilemma.  What research-based criteria went into Hattie’s label change to Interventions for Students with Learning Needs, and what does this “new” variable or construct mean?

_ _ _ _ _

   Summary.  All of this must be incredibly confusing and challenging to educators in the field.  Indeed, if Hattie’s list of effects is constantly changing—both in rank order and in name—how do we keep up with the changes and make the practitioner-oriented programmatic decisions that need to be made?


_ _ _ _ _ _ _ _ _ _

The Questions to Ask the Outside “Hattie Consultants”


   And yet, the vast majority of districts—much less schools—have the personnel with the time and skills to do this.

   To fill this gap:  We now have a “cottage industry” of “official and unofficial” Hattie consultants who are available to assist.


   With no disrespect intended, just because someone has been trained by Hattie, has heard Hattie, or has read Hattie—that does not give them the expertise, across all of the 250+ rank-ordered influences on student learning and achievement, to analyze and implement any of the approaches identified through Hattie’s research.

   And so, districts and schools need to ask a series of specific questions when consultants say that their consultation is guided by Hattie’s research.

   Among the initial set of questions are the following:

  • What training and experience do you have in evaluating psychoeducational research as applied to schools, teaching staff, and students—including students have significant academic and/or social, emotional, or behavioral challenges?







   If I do not have the expertise to help a district or school in a specific area, I typically recommend two to four colleagues who I believe have the skills—and then let the district or school leaders contact and vet them on their own.



_ _ _ _ _ _ _ _ _ _


   Once again, none of the points expressed in this Blog are personally about John Hattie.  Hattie has made many astounding contributions to our understanding of the research in areas that impact student learning and the school and schooling process.



   To do this, I have suggested (from earlier in this Blog) the following Multiple-gating approach:




_ _ _ _ _

   Given the limited resources of most districts and schools, they will likely look to outside consultants for assistance.  But if the educational leaders are going to use other staff or outside consultants to guide the process, they must ask the questions and get the answers to ensure that these professionals have the knowledge, skills, and experience to accomplish the work.

   Still too often, districts and schools are “blinded” by consultants who use their affiliations to “validate” their expertise, rather than their research-to-practice experience and field-based success.

   In the end, schools and districts should not invest time, money, professional development, supervision, or other resources in programs or interventions that have not been fully validated for use with their students and/or staff. 

   Such investments are not fair to anyone—especially when they (a) do not delivering the needed results, (b) leave students further behind, and/or (c) fail and create staff resistance to “the next program”—which might, parenthetically, be the “right” program.

_ _ _ _ _

   I hope that this discussion has been useful to you.

