Management information systems (MIS) and the analysis of data can lead to significant improvements in teaching and learning – as long as curriculum managers analyse and apply the findings effectively when considering pupil achievement, says Anthony Powell
In many schools, the power of management information systems (MIS) is not properly exploited, in large part because the amount of data is so great. The purpose of information systems is to identify patterns and anomalies in pupil performance and enable schools to trace the causal links with the quality of education. This means schools need to identify what they want to monitor and evaluate, what information they need and how they are going to organise and interrogate the evidence base they have created.
It takes time to set up effective systems and this is one of the main reasons why there is a wide variation in practice in using MIS to improve teaching and learning (T&L). There are audit tools to help you with this – see the box below for one example. To achieve maximum benefit from using MIS, schools need to spend time on the early stages of development to lay the foundations essential for evaluating all aspects of the school’s work in future.
Improve: audit tool
Improve: data for school improvement is a free audit instrument available for schools that is organised into four domains, using Bloom’s taxonomy (see Taxonomy of educational objectives, the classification of educational goals, 1956, McKay):
This detailed software package aims to show schools the stages of development towards using management information systems efficiently and effectively. Schools can register to use Improve at: www.dataforimprovement.co.uk
Uses and misuses
It is a capital mistake to theorise before using data. (The Memoirs of Sherlock Holmes, by Sir Arthur Conan Doyle, Penguin Books, 1973)
‘Data’ here means any information, but Holmes pioneered in literature the methodical use of evidence to reconstruct the chain of events. In education, the starting point for measuring impact (before theorising) is to analyse standards and progress.
The use of MIS has developed in response to need. Level of use also depends on expertise within the school and its stance towards sharing information. Common stages, from simplest to most sophisticated, are listed in the box below.
Stages of MIS use
More extensive use
More extensive use
The main data sets used by schools are RAISEonline and Fischer Family Trust. Any analysis of data will show that different pupils and groups of pupils make different rates of progress across the curriculum. To identify patterns, schools should analyse results using the questions in the box below.
Identifying patterns in performance: questions to ask
Past performance – standards
Past performance – progress
It is worth drilling down beyond the headline results to make the next stages more efficient. For example, in one school, history consistently has a RPI significantly above other subjects, due to:
In this case, the school should investigate the reasons for this high attainment by looking at the quality of teaching, assessment and target-setting.
When assessing progress, ask yourself:
Predicted performance Where pupils do well in a subject (in comparison with others), it is often because there is a higher level of challenge. There are a number of ways to evaluate high expectations, but a good starting point is by comparing the targets set in different subjects. For example, if allied subjects such as history and English have different overall targets for the end of Key Stage 3, this needs to be investigated. Ask yourself:
There are dangers in having such a wealth of information available. The most common are set out in the box below.
Dangers of data
Too much time processing data
Focusing on small differences
Assessment is too frequent
Lack of standardisation
Analysis is driven by external factors such as league tables
Confusing data with knowledge
Making invalid comparisons
Ignoring qualitative data
Ignoring the present
Many of these can be avoided if schools think through their procedures and purposes for MIS. The questions in the box below provide a focus for discussion and a way to evaluate the effectiveness of your system.
Evaluating the use of data
Does the school have a clear purpose for using data?
Is data analysis integrated into whole-school self-evaluation and school improvement planning?
Is data reliable and standardised?
Does data give a baseline for measuring progress?
Does the school use data to set targets and track progress?
Does the school analyse the progress of groups of pupils?
Does use of data lead to action?
Does the use of data have an impact on pupils’ achievement?
Is the school’s use of data cost-effective?
Blanket monitoring versus focused evaluation
All schools now regularly collect information about their provision and procedures, for example observation of teaching, and outcomes such as attendance, behaviour records and academic results. Other information is collected and often not stored systematically or collated, for example records of rewards and sanctions. This can be called ‘blanket monitoring’, whereby the school has a lot of ‘knowledge’ about itself but does little analysis, synthesis or evaluation of this information. But monitoring should not be dismissed as having no value, as it often ensures that policies are implemented consistently and this is one of its main purposes.
‘Focused evaluation’ starts with an area for investigation and a hypothesis. For this Case in Point, the area for investigation that we are focusing on is teaching and learning and the hypothesis is that it is variable across the school, which has an impact on pupil performance. Put bluntly, pupils make better progress in some subjects and classes than in others and the school wants to know why.
Schools will be familiar with this process from Ofsted inspections. With the changes in the Ofsted methodology introduced in September 2005 (see: Every Child Matters: framework for the inspection of schools in England from September 2005, Ofsted, 2005), inspectors now carry out a pre-inspection briefing (PIB). From the evidence available at that point, inspectors identify and provisionally agree what are expected to be strengths of the school and also issues for the inspection. These will largely derive from differences in performance.
Aims and success criteria
Monitoring and evaluation of teaching and learning consumes a great deal of staff time. Since this is the most valuable (and the most costly) resource in the school, it is important to ensure the purpose of the exercise is to raise student achievement. Pupil achievement is an effect, while any improvement strategy is a cause that will almost certainly take time to have an impact. It is essential to think through short-, medium- and long-term success criteria so that you can then measure progress at each stage, which in this case would be along the lines of:
- short term: the school will have a clear understanding of the patterns of achievement across the school and how these are related to the quality of teaching and learning
- medium term: the continuing professional development (CPD) programme will be successful in improving the professional skills and expertise of all staff
- long term: there will be a long-term rising trend in achievement across the school.
Schools often complain about Ofsted inspectors having preconceived ideas, usually based on an analysis of data. While school staff have a more rounded understanding based on a much greater evidence base, it is still true that they are encumbered with too much knowledge. The danger can be that some underlying assumptions will not be questioned. For example, the senior leadership team may begin this evaluation with judgements about the quality of teaching and individual teachers. Reality will be far more complex.
Schools should adopt a perspective that is objective, fair and based on robust evidence. Monitoring and evaluation should always be a team effort, seen as professional and supportive. It is useful to adopt the standpoint of someone who does not know the school but needs to understand it quickly.
Organise the evidence base
Evidence will be in many forms, but broadly, it will fall into two categories:
- evidence about outputs (academic and personal development)
- evidence about provision (the quality of teaching and learning, curriculum and care, guidance and support)
There will also be different types of evidence, for example qualitative and quantitative, and evidence will be generated from different sources such as data, interviews, observations and so on.
Whatever the type of evidence, it must always be interpreted, cross-referenced and synthesised.
This process will result in written notes from brief jottings, records, ideas and speculations to organised reports. The danger is that during the process, the evidence base becomes scrappy and disorganised. Ofsted use what is called an evidence form (EF) to record all the information gathered during the inspection. There are several advantages in using a common template for recording findings. For example:
- it can contain background information such as the author, the date and time and the nature of the evidence, and record year groups, classes, numbers, ability and gender – information that is essential for making objective comparisons.
- it allows all the evidence to be recorded in one place for ease of reference.
The Ofsted evidence form can be easily adapted for school use. It can be downloaded from the section on ‘forms and guidance’ (follow the link through ‘schools’, ‘ inspection resources’, and then ‘guidance for inspectors of schools’).
The framework for evaluating all aspects of a school’s work is now provided by the self-evaluation form (SEF). If a school files all its evidence against the SEF sections, then all the information generated by this exercise should be stored in Section 5a: the quality of teaching and learning.
There is a logical link between good teaching and good learning, so the starting point is to evaluate pupil performance. Differences in achievement lead into an investigation of the quality of T&L using agreed criteria. This will involve a variety of techniques. While criteria for evaluating T&L are well known and used extensively by schools. techniques for gathering evidence tend to be limited, with too much emphasis placed on lesson observation. The main criteria are listed in the box below with suggestions for questions and for collecting evidence not normally consulted. All of this evidence would be supplemented by focused lesson observation.
Criteria evaluation T&L
|Evaluation question||Assessment criteria||Evidence type|
|Do teachers have a secure knowledge and understanding of the subjects or areas they teach?||
||Staff files will show qualifications and experience, courses attended and performance management objectives and reviews.|
|Do teachers set high expectations so as to challenge pupils and deepen knowledge and understanding?||
||Evidence includes results (particularly higher grades), planning, teacher assessment and scrutiny of work.|
|Do teachers plan effectively?||
||Consider schemes of work, pupil groupings, classroom organisation and use of support staff.|
|Do teachers employ methods and organisational strategies that match curricular objectives and the needs of all pupils?||
Teaching and learning methods should be evaluated by their appropriateness since there are no ‘best’ strategies. However, across the school a range of strategies should be used to cater for different learning styles:
Teacher activity During the course of the lesson do teachers move through a range of activities such as explaining, narrating, demonstrating, instructing, questioning?
Pupil activity During the course of the lesson do pupils move through a range of activities such as listening, watching, discussing, reading, writing, experimenting and problem-solving?
Working pattern Do pupils work in groups, pairs, as individuals and as a whole class appropriate to task?
Resources Is there a good range of quality resources, for example books, pictures, artefacts, equipment and ICT?
|The range of strategies will be shown in planning, pupils’ work, student tracking and staff development interviews. Drop-in observations (lasting about 10 minutes) can also be used to survey the strategies being used across the school.|
|Do teachers manage pupils well and achieve high standards of discipline?||
||The use of rewards and sanctions, classroom rules and organisation, pupil referrals, parental responses and student interviews all give evidence of the different nature of relationships across the school.|
|Do teachers use time and resources effectively?||
||Evidence includes planning, teacher assessment, pupils’ work, range and quality of resources used.|
|Do teachers assess pupils’ work thoroughly and constructively, and use assessments to inform teaching?||
||Check against the homework timetable and consider whether homework simply replicates classwork.|
|Do teachers use homework effectively to reinforce and/or extend what is learned in school?||
||Look through assessment records, check the accuracy of assessment, planning and how books are marked.|
At the end of this exercise, schools will have detailed knowledge of teaching methods and strategies across the curriculum. More importantly, they should understand the causal links between various methods and pupil achievement. The creative part is to design and implement a CPD programme that gives all teachers the professional expertise to use all the successful strategies.
The related case study explores how beneficial effective data analysis can be, showcasing one school that uses MIS to equip staff to take T&L to the next level.
Anthony Powell, consulting educationist specialising in self-evaluation, school improvement and CPD