Strategic Forum for Research in Education

Is there appropriate provision and incentivisation for the production of high quality and innovative developmental and evaluation research?

Participants at Forum II (held on 17th and 18th June 2009 in Reading) were invited to discuss research that aimed to inform directly “the development and evaluation of applied policy and practice initiatives (typically local and national government, charitable and private sector funding of a wider range of universities, centres and private organisations)”.

Examples/ overview from countries

In England the evaluation of the Education Maintenance Allowance pilot is often referred to in terms of a rigorous evaluation which was relatively well planned and had considerable impact (for all the reports see: http://www.ifs.org.uk/projects/98).

The Institute for Effective Education (IEE, see: www.york.ac.uk/iee) has recently been established at the University of York which develops and evaluates effective education programmes with a focus on randomised-control trials. IEE also manages the Best Evidence Encyclopaedia UK (BEE UK) which provides information regarding the weight of evidence supporting the effectiveness of different education programmes (www.bestevidence.org.uk).  The IEE (in collaboration with other colleagues) is also undertaking the national evaluation of the Every Child Counts initiative, which is taking a randomised trial approach in order to establish the effectiveness of the programme (for more information see: http://www.york.ac.uk/iee/research/t_every_child_counts.htm).

The Welsh Assembly Government has used reviews of research and practice to inform policy (Davies, 2009). Notable examples which have had considerable impact are the Daugherty Review of statutory assessment (Daugherty et al., 2004), the two Rees Reviews of student fees (Rees, 2001; 2005) and the Furlong Review of initial teacher training provision (Furlong et al., 2006).

As a result of the nature of the organisation and history of academic education research in Wales, there is a lot of developmental and evaluative work which takes place between researchers and practitioners at a local level due to the experience and skills of many researchers in teaching-focused institutions (see Davies, 2009 for examples).  

In Northern Ireland, the recently established Centre for Effective Education (CEE) (which grew out of the NFER@Queen’s centre) has been an important development in the area of evaluation and developmental research. The CEE is an ‘applied and interdisciplinary research centre committed to improving outcomes for children and young people by ensuring that the design, delivery and evaluation of educational programmes and interventions are informed by the best available evidence’ (http://www.qub.ac.uk/schools/SchoolofEducation/CentreforEffectiveEducation/). (For more on CEE and its projects see the Northern Ireland input report for Forum II – Leitch with McCullough, 2009).

The Schools of Ambition initiative is a Scottish Government programme of school improvement through practitioner research which was launched in 2005 (see: http://www.ltscotland.org.uk/schoolsofambition/). The schools implemented interventions from their transformative plans and school-based collaborative enquiry was used to evaluate and develop their effectiveness and potential. The central activity in Schools of Ambition is described in the practitioner-research section below. There was also an important formative evaluative element so that lessons learned from the 52 Schools of Ambition could be used to develop the interventions as part of the action research cycle and to inform similar activity across Scottish schools (Hulme and Menter, 2008). Teacher evaluators are supported by university researchers (from Universities of Glasgow, Aberdeen and Strathclyde). The team works together to design research evaluation strategies to evaluation the interventions carried out as part of the school’s transformative plan. University researchers also provide feedback on the process and activities undertaken for stakeholders.

The Future Learning and Teaching (FLaT) Programme (see: http://www.flatprojects.org.uk/) allowed practitioners in schools and local authorities to gain funding to run innovative pilot projects which challenged current thinking in teaching and learning. The programme  included projects on a wide range of topics, from assessment for learning to school design. All of these projects (above a certain funding level) were evaluated by researchers with a view to provide feedback to the Executive Board and to provide formative feedback to schools and local authorities in terms of impacts on teaching and learning.      

For further examples of strengths and areas for development in evaluative/developmental research for each UK country see the country mappings prepared for Forum II.

The current state of evaluative and developmental research

Developmental and evaluative research describe a large range of activities in the UK, in terms of scale, methodology, purpose and who undertakes them. They range from government-commissioned nation-wide pilots through to small-scale developmental projects involving a small number of practitioners.  A significant proportion of this work takes place outside HEIs.

Although recognising them as area of intense activity in recent years, participants at Forum II also raised a number of problematic issues about evaluative and developmental research. Issues of discussion included the constraints surrounding thes types of work, their quality, scale and stimeliness, and the ways in which they linked into/ were drawn upon in policy and practice.

Quality in developmental and evaluative research

Several groups at SFRE II discussed whether some of the work in this area fell short of accepted definitions of what should count as “research”, or, if it did count of research, fell short of some of its standards of quality. It was argued that, despite existent weaknesses, developmental and evaluative work can include research of a high standard and can make important contributions to education practice, as well as contributing to increased understanding of education. In considering aspects of quality in applied research, Oancea and Furlong (2005) note that a multidimensional approach should be taken and that applying undiscerningly the same criteria used for disciplinary research is problematic (also see Forum I report). 

The differing reasons which commissioners have for commissioning an evaluation are associated with different levels of resources allocated and with different levels of freedom over the approach taken, avenues explored and editorial rights – all of which can have implications for the quality of evaluations. Group discussions at the Forum were clear that in order to achieve high quality evaluations it is important to have the evaluation built in and planned from the outset (ideally before the intervention begins) and to be clear about the precise questions to be investigated.

Current challenges in promoting and incentivising quality in developmental and evaluative research

Challenges identified at SFRE II included:

• Scaling up and building on existing knowledge (including knowledge of evaluation techniques; enabling access to findings; and suporting, as appropriate, larger-scale, joined-up, and longer-term projects).
• Commissioning the appropriate evaluation (including honing the right questions; using judiciously the range of research skills and methods available; supporting capacity building; and fostering innovation through commissioning practice).
• Effectively negotiating contractual complexities and relationships (including stating clearly expectations; developing good communication and partnership between researchers and commissioners; and ensuring appropriate integration of evaluation and intervention schedules).

Further reading

Oancea, A. and Furlong, J. (2007) Expressions of excellence and the assessment of applied and practice-based research. In: Research Papers in Education, 22(2), pp. 119 – 137.

Further questions/ outstanding issues

• How explicit were the boundaries of the project at the beginning? Was there scope for the researcher to assess/critique the underlying aims of the initiative? (Perhaps this is easier with charities than government departments).
• If HEIs want to compete with other research organisations in this area of research, they may need to explore developing centres which focus on evaluative projects and models. Perhaps there is more potential in collaborations between HEIs and research organisations in this area than currently being explored.
• As funding for research, and social science research in particular, is likely to be increasingly squeezed, will there be pressure to undertaken proportionally more instrumental developmental and evaluative research at the expense of other types of research? Should there be fewer but better evaluations commissioned?
• Ought evaluations hold more value in terms of the REF? If so, how?