To update or not to update systematic literature searches? Deciding what constitutes stable evidence

25/11/21

The Belgian Red Cross has been developing evidence-based first aid guidelines and training materials for use in Flanders (Belgium) and Sub-Saharan Africa for over 10 years. To provide a basis for these evidence-based materials, CEBaP collects the best available evidence through systematic literature searches in electronic databases of medical literature and synthesizes the evidence in structured evidence summaries (ES). Because science evolves with the accumulation of the research, the guidelines are updated every five years, to maintain the validity of recommendations. However, because of the volume of research questions, the process of updating and revisiting the systematic literature searches is time consuming. 

Update classification system

We here first aimed to develop a classification model to decide whether an ES needs to be updated primarily based on estimating the likelihood that the current conclusions will maintain their relevance in the foreseeable future. Situations in which an evidence summary may be declared to be stable include the following:

  1. the conclusion is so certain that the addition of new information will not change it, and there are no foreseeable adverse effects of the intervention (conclusive evidence);
  2. the intervention can be considered common sense and the available evidence supports the recommendation.

First, what defines ‘conclusive evidence’? We here assess whether future studies are likely to change the findings of the evidence summaries. According to the GRADE Working Group, certainty of evidence reflects the extent of our confidence that the estimates of the effect are correct. The GRADE approach results in the assessment of the certainty of a body of evidence as high, moderate, low, or very low. In some cases, the decision can be very straightforward, for example when the existing findings of the summary are based on high certainty evidence, no further update might be needed. Although, it is equally possible that an evidence summary can (temporarily) be declared as stable if the certainty of the evidence supporting it is only, at best, low or very low. The primary reason is that further research is very unlikely to change our confidence in the effect estimate given the inherent and anticipated limitations in the research domain.

Secondly, when interventions or risk factors can be considered common sense and the conclusions of an evidence summary confirm the effectiveness or harm, the evidence can be categorized as being stable. At the same time, it should be reasonable that future studies would not change the recommendation. This can be anticipated by controlling for inconsistent results and by paying attention to potential adverse events or harm associated with the intervention or risk factor.

A pilot study

The Belgian Red Cross has updated its first aid guidelines for Flanders (Belgium) and Sub-Saharan Africa for the third time in 2021. During this process, ES for 454 research questions were newly developed or updated and, after completion, the update status of each individual ES was assessed by two reviewers independently. Based on the in-house developed classification system, 95 ES (21% of total) were considered to be stable. 27 ES (28%) were stable due to conclusive evidence and 68 (72%) were considered common sense supported by the best available evidence. In only 7% of the ES, a conflict between reviewers was noted. A final decision was then made by a third reviewer. The update status should be revaluated every five years. Based on the total time investment for updating the guideline ES (412 working days), one can estimate that implementing the update classification system could save about 87 working days. These results provide proof-of-concept that the update classification system will help us to increase time-efficiency of future updates of evidence-based guidelines. 

Curious about the details of our update classification model and further results? Our findings will be published in 2022, stay tuned!

×

      The CEBaP website use cookies to help make your visit as smooth as possible. We install necessary and technical cookies to ensure the websites run properly. We also use analytical and advertising cookies to measure and improve the general use of the websites. Indicate below whether you agree to this cookie policy in order to continue.

          These necessary cookies ensure all parts of the website work properly. We can install these cookies without permission. By continuing to use our websites, you agree to the use of the necessary cookies.

          We use cookies for functional purposes. This means that these cookies are necessary for the proper functioning of our websites. These cookies identify and store your preferences (including language) and record whether you have accepted cookies. 

          We use cookies for analytical purposes. These cookies gather information on the technical details of the exchange or the use of the website (pages visited, average length of visit etc.) in order to improve the site’s operation. We use Google Analytics for this (in accordance with the Google Analytics Terms of Use: https://www.google.com/analytics/terms/us.html).

          We use cookies for advertising. These cookies provide us with statistics (which cannot be traced to individual users) to improve our online communications and recruitment for our activities through various channels and to tailor our service to your needs. By accepting these cookies, you also accept the use of the Facebook pixel.