What is a systematic review?
A systematic review is a summary of available research on a particular topic; and similar to a literature review. It identifies, assesses and synthesises the findings from multiple studies to answer specific questions. The key difference from a literature review is that it approaches evidence systematically, thus avoiding potential bias in findings.
What did we do in our systematic review?
Last year, the EIB Group’s Evaluation Division conducted a systematic review of residential energy efficiency. This was the first time we used a systematic review. To ensure that we would get the most from this pilot, we worked with the International Initiative for Impact Evaluations, known as 3ie, a global leader in producing and synthesising rigorous evidence. Along the way we learned some lessons useful for other evaluators we’d like to share:
- Advantages of using a systematic review
- Why starting with an evidence map can be helpful
- Trade-offs to bear in mind when undertaking a systematic review
What are the advantages of a systematic review?
Often there is a large amount of existing evidence available. What’s more, findings from the evidence can at times appear to be contradictory, either due to the quality of the research, or because results are influenced by the circumstances under which the research has taken place. By collating, appraising and then summarising the evidence, systematic reviews facilitate the use of the evidence.
This makes them especially useful for decision makers without much time on their hands, who want to understand what works, and what doesn’t work under which circumstances. But systematic reviews are equally useful for evaluators and researchers who can see where a lot of solid evidence already exists and where gaps remain.
In a systematic review:
- The scope and methods are defined upfront in a study protocol, which sets out clear inclusion criteria (transparent)
- The search is comprehensive and includes published and unpublished studies, in any language (unbiased)
- A critical appraisal of the quality of studies is carried out to assess the reliability of studies that are included (rigorous)
- Data extraction and organisation is systematic and reproducible (transparency)
What is an evidence (gap) map?
An evidence gap map is a tool designed to provide an overview of the existing evidence on a topic.
Why starting with an evidence map can be useful
Instead of agreeing a topic and research question for our systematic review, then collecting the available evidence and synthesising the findings, we started the review by mapping the evidence. The advantages of this approach are that it puts together an inventory of all the available evidence, which can help identify future research priorities and void duplication of effort.