Teaching Expertise

We are constantly trying to drive up standards of teaching and learning with new approaches, preferably those with a strong evidence base. But is ‘What Works?’ the right question? Should we really be asking ‘How do good teachers get better?’ Elaine Hall reflects on the messages from a meta-analysis of teaching and learning interventions

In teaching and learning, we are increasingly concerned with improving students’ attainment through focused interventions and it is becoming the responsibility of individual teachers, subject leaders or teaching and learning coordinators to select the best intervention for their students from a dazzling array of options. But how and where to start? How can teachers make informed choices? One important tool at our disposal is meta-analysis, though it is only one of a range of tools that teachers need.

Meta-analysis is a method for pooling the quantitative estimates of effects of interventions from multiple studies to give a more reliable and precise estimate of their benefits (or potential harm). Comparing these estimates across different types of interventions can also pinpoint which aspects of interventions offer the most potential in the classroom. Meta-analysis is proving to be a useful approach to addressing the key question of ‘What works?’ by providing an opportunity for ‘ground-clearing’ and priority- setting, since by offering comparative information about how well different interventions work, teachers can make informed decisions. Hattie – author of the largest meta-analysis – made the purpose of this clear:

  • ‘We need to make relative statements about what impacts on student work.
  • We need estimates of magnitude as well as statistical significance – it is not good enough to say that this works because lots of people use it etc, but that this works because of the magnitude of impact.
  • We need to be building a model based on these relative magnitudes of effects.’   (Hattie, 2003)

Intervention
 By intervention we mean any change or programme of change which is implemented in the classroom, it could be formal or informal and might be initiated by the individual teacher, the school or another organisation from outside.

Meta-analysis
A meta-analysis looks across the results of several studies that address a set of related research questions with the aim of bringing about generalisable conclusions.

Magnitude of impact is measured by effect size, a statistical calculation of the amount of change (for a discussion of effect sizes please see the links section at the end of this article). The enormous meta-analysis of 500,000 effect sizes conducted by John Hattie enables us to place interventions relative to others and to ‘normal development’. Of course, some interventions have minimal, or even negative effects but in terms of positive effect sizes, we must judge them in relation to the following key benchmarks:

0.1 – (maturation) the yearly improvement from normal development without any teaching
0.25 – (teaching) the average effect of having a teacher, any teacher, regardless of the quality of instruction
0.42 – (mean) the average effect size for all educational interventions
0.5 – (minimum for intervention) since better than average effects should be a basic requirement for expending resources
0.8 – (significant effects) where the impacts start to become visible to the ‘naked eye’.

The 0.42 ‘average intervention effect size’ is equivalent to raising the average attainment of class by about 16 percentage points. This sounds quite good but it is important to remember that, since this is the mean, there are many more effective interventions out there. For this reason, Hattie considers 0.5 as a minimum for an intervention to be considered ‘educationally significant’. It is worth noting that in medicine, where effect sizes are commonly used, the bar is set much higher and effect sizes approaching 1.0 are required to provide a warrant for action.

So which interventions work better than others? Equally, which interventions have little evidence of positive effects? The effects results below gives details of some of the high and low performing approaches and may cause many readers concern in terms of what is and is not included (the full list is available online, see the Links section). It is important to emphasise – meta-analysis tells us the effect that these interventions have on pupil attainment, not whether something is a good idea or not, not whether these interventions improve teacher or pupil morale and engagement, not whether they make schools operate more efficiently or more pleasantly. However, if the purpose of the interventions in the low-performing column is to raise attainment, then we might want to think again.

The effect of different interventions on student attainment

High performing innovations ‘top ten’ / Effect size

Feedback  / 1.13

Prior Ability / 1.04

Instructional Quality / 1.00

Direct instruction / 0.82

Remediation feedback / 0.65

Student disposition / 0.61

Class environment (culture) / 0.56

Low Performing Innovations 'bottom ten' / Effect Size

Retention / -0.15

Television / -0.12

Physical elements of school / -0.05

Behavioural objectives / 0.12

Programmed instruction / 0.18

It seems pretty clear that areas where schools have been directed to spend a considerable amount of their time and energy – new buildings and ability grouping – are not going to ‘drive standards up’. Indeed, those implementing the next wave of change, under the banner of ‘personalised learning’ should be aware of the low leverage of individualisation. Throwing money at the problem also appears to be less effective than having a teacher in the room (0.12 compared with 0.25). Other popular approaches, testing (0.3) and ICT (0.31) also have a below-average impact.

There are also some high leverage factors over which teachers have relatively little influence: social and demographic factors and prior ability will always determine a significant amount of each student’s success. However, it is clear that there are very important effects from the quality of the teaching and the quality of the reinforcement of learning, through feedback conversations with teachers and peers. There is certainly mounting evidence that adopting approaches which make aspects of thinking explicit or which focus on particular kinds of thinking are successful at raising attainment (particularly metacognitive approaches or cognitively demanding interventions such as problem solving and hypothesis testing or those that enhance surface and deep learning). Is this a sufficiently compelling argument for using only interventions from Hattie’s ‘top ten’?

Experience suggests that the use of meta-analysis can only be one part of the decision-making process. The Research Centre for Learning and Teaching at Newcastle University conducted an analysis of thinking skills interventions and found there is both strong evidence for the value of thinking skills – particularly in relation to cognitive and curricular gains – and areas where evidence is less coherent. For thinking skills interventions, it appears that the effects on learners are age and context specific. The blended results of a meta-analysis are not sufficient for teachers’ needs. Teachers need to act in a particular context and to make a particular choice about which approach is likely to work for them in their school – and meta-analysis data can help them only on the first part of that journey.

Indeed, when Hattie broke down the teaching and learning factors in his meta-analysis by ‘who’ rather than ‘what’ he discovered that the influence of teachers on student outcomes was extremely important.

Moreover, he found that the differences in teacher practice linked in with the evidence from successful interventions. The meta-analysis evidence suggests that successful innovations tend to be structured around feedback loops which encourage teachers to reflect and improve: Hattie’s significant innovations group is packed with things that tell the teacher ‘How am I doing?’ (not just ‘How are the pupils doing?’). It is suggestive that merely ‘doing formative assessment’ to pupils is not sufficient to provide this feedback, since evidence-based teaching and learning is dependent on how teachers perceive and use the information.

‘If students do not know something, or cannot process the information, this should be a cue for teacher action... Merely ascribing to the student the information that they can or cannot do something is not as powerful as ascribing to the teacher what they have or have not taught well.’   (Hattie, 2005, p17)

Tight feedback loops focus the teacher’s attention on what is in their sphere of influence, gives information that can be put to use immediately and increases the positive reinforcement between learners in the classroom.

A study of the difference between experienced and expert teachers (Hattie, 2003 – see results below) found that the distinction was not between the level of content knowledge or procedural expertise but to a series of dimensions and attributes of pedagogical practice. The difference between experienced and expert teachers was not reflected simply in their teaching but in the quality of the students’ learning. Many of the facets of expert teacher practice were strongly related to feedback and the quality of relationships between teachers and students. The remaining factors relate to the individual teacher’s readiness to engage with feedback and relationships: his or her personal and professional ‘comfort zone’.

Dimensions of expert teachers' attributes

Identify essential representations of their subject 

  • Have deeper representations about teaching and learning
  • Adopt a problem-solving stance
  • Anticipate, plan and improvise as required
  • Are better decision makers

Guide learning through classroom interactions 

  • Create an optimal climate for learning
  • Have a complex perception of classrooms
  • Are more context dependent and aware of situations

Monitor learning and provide feedback  

  • More adept at monitoring and provide more relevant feedback
  • Test hypotheses about learning problems or teaching strategies
  • Work more automatically

Attend to affective outcomes 

  • Have high respect for students
  • Are passionate about learning and teaching

Influence student outcomes 

  • Develop students’ self-regulation and esteem
  • Provide appropriate challenge
  • Have positive influences on student achievement
  • Enhance surface and deep learning

There is evidence that not all interventions are equal and that some have greater leverage than others. It is becoming clear that the individuals and groups of teachers operating the interventions are at least as important a part of the equation as the content or approach. Meta-analyses may enable us to make smart choices about ‘what’, it then becomes vital to focus upon ‘who’ and ‘how’.

Elaine Hall is a senior researcher in the Research Centre for Learning and Teaching at Newcastle University

A very clear introduction to effect size by Robert Coe is available from www.cemcentre.org/renderpage.asp?linkID=30325016

A good summary of Hattie’s meta-analysis www.learningandteaching.info/teaching/what_works.htm

The Thinking Skills meta-analysis http://eppi.ioe.ac.uk/EPPIWebContent/reel/review_groups/thinking_skills/t_s_rv2/t_s_rv2.pdf

  • Hattie, J, Biggs, J, and Purdie, N (1996). ‘Effects of Learning Skills Interventions on Student Learning: A Meta-analysis.‘ Review of Educational Research, 66, 99-136
  • Hattie, J (2003) ‘Teachers Make a Difference: What is the Research Evidence?’ Paper presented at the Australian Council for Educational Research Conference ‘Building Teacher Quality: What Does the Research Tell Us?’ 19-21 October 2003, Melbourne. Available for download from www.acer.edu.au/workshops/documents/Teachers_Make_a_Difference_Hattie.pdf
  • Hattie, J (2004) ‘Factors that Influence Children‘s Learning: Results from a Study Involving 500 Meta-analyses.’ Paper presented at the ESRC Seminar ‘Effective Educational Interventions‘, University of Newcastle upon Tyne, 8 July, 2004
  • Hattie, J (2005) ‘What is the Nature of Evidence that Makes a Difference to Learning?’ Paper presented at the Australian Council for Educational Research Conference ‘Using Data to Support Learning’
    7-9 August 2005, Melbourne. Available for download from www.acer.edu.au/workshops/documents/Hattie.pdf
  • Higgins, S, Hall, E, Baumfield, V, and Moseley, D (2005) Thinking Skills Approaches to Effective Teaching and Learning: What is the Evidence for Impact on Learners? London: EPPI-Centre