Can I Include Validation Studies in a Systematic Review

Systematic reviews

"I am preparing a systematic review" – this is something we oftentimes hear at KIB. Simply what exactly is a systematic review? How does information technology differ from other types of reviews? And how exercise you systematically search the literature?

This page is aimed primarily at researchers and doctoral students. If you are a Bachelor'due south or Master'due south student, you lot will find our guide Structured literature reviews – A guide for students useful. As a pupil you should rather practice a structured literature review, including just certain aspects of the methodology described on this page.

Introduction

According to the Cochrane Handbook, a "systematic review attempts to collate all empirical testify that fits pre-specified eligibility criteria in order to answer a specific research question". A systematic review likewise adheres to a specific methodology and oft includes meta-analyses, wherein the collected data is combined using statistical methods.

Since they summarise the results from all original studies within a given field, systematic reviews are commonly regarded equally high quality bear witness. A pyramid is often used to visualise this bureaucracy of bear witness, with systematic reviews placed at the apex of the pyramid.

Pyramid with 6 levels. Top: Meta-analyses and Systematic reviews (filtered information). Below Randomised controlled trials, Cohort studies and Case studies (unfiltered). Bottom: Background information or Expert opinion.

In this pyramid, only report design variations take been included. Other, more complex pyramids add other types of layers to the systematic reviews, for example synopses of syntheses (see Alper & Haynes, 2016).

Since the total output of medical scientific literature produced every year is increasing at an exponential charge per unit, systematic reviews that collate the bachelor show have become increasingly of import. Consequently, more and more systematic reviews are published every twelvemonth, a trend evident in scientific databases.

This is one reason why articles summarising the literature are of such importance today. Accordingly, the number of published systematic reviews has too increased (run across Bastian, Glasziou & Chalmers, 2010). Today, there are even reviews of systematic reviews (see for case Aromataris et al, 2015).

As a doctoral student at KI, yous are allowed to include a systematic review as part of your thesis. The Committee of Doctoral Education besides emphasises the importance of a structured review of the literature in the thesis "kappa". A literature review of your research field is at present mandatory in the one-half-time summary.

Present, there are also journals – for example, The Lancet and Alzheimer'south & Dementia – that crave primary studies provide a structured, comprehensive and documented literature search ("Research in context").

For more about systematic reviews in full general, see the Farther reading section at the lesser of this page.

What is the difference between a systematic review and a traditional review?

In that location are 4 essential criteria for a systematic review:

It should be exhaustive: all relevant literature in a research field should exist included.
A rigorous methodology must be followed throughout – from defining the research question, writing a protocol and searching the literature, to gathering, screening and analysing. The unabridged process should likewise be thoroughly documented.
At least two people should be involved, peculiarly for screening manufactures and extracting data.
Plenty of time resource are needed, but also in terms of availing yourself of others' expertise – for case in database searching – and tools and software.

There are a variety of review types, each using a more than or less different methodology; the terminology can be confusing. In an article from 2009, Grant & Booth described fourteen review types, for case scoping reviews, and their associated methodologies.

For a condensed overview, run across the comparison below (from Jesson, Matheson & Lacey, 2011, p. 105).

	Traditional (scoping) review	Systematic review
Aim	To gain a broad understanding, and description of the field	Tightly specified aim and objectives with a specific review question
Scope	Big picture	Narrow focus
Planning the review	No defined path, allows for inventiveness and exploration	Transparent procedure and documented audit trail
Identifying studies	Searching is probing, moving from one report to another, following upwards leads	Rigorous and comprehensive search for ALL studies
Selection of studies	Purposive selection made by the reviewer	Predetermined criteria for including and excluding studies
Quality cess	Based on the reviewer'south opinion	Checklists to assess the methodological quality of studies
Analysis and synthesis	Discursive	In tabular format and short summary answers
Methodological study	Non necessarily given	Must be presented for transparency

A cardinal office of a systematic review is an exhaustive literature search that finds all relevant studies on a topic. Hence, it is important that the search strategy is rigorously developed with a high sensitivity to observe all these potential relevant articles. This means that you will finish upward with a big amount of references, often more than thousands of references. A high percent will probable be irrelevant.

Information technology is also important to conduct the search in several databases. Cochrane recommends using at least 3. In biomedicine and wellness, standard databases include PubMed/MEDLINE, Embase and Cochrane Library. Depending on the topic, a multi-disciplinary database like Web of Science or subject specific databases, such equally CINAHL, PsycInfo or ERIC, should also exist considered.

In addition, greyness literature, for example dissertations and clinical trials, should exist considered in your search. These publication types demand other databases. The literature search is usually besides supplemented with a screening of reference lists, commodity suggestions from colleagues and sometimes with a citation analysis – i.east. an analysis of which articles have cited an older, notwithstanding relevant study.

Systematic search techniques

In this section, we cover the specific techniques used during a systematic search. For a full general introduction, have a look at our searching for information page. In that location you will discover basic data about search techniques and search logic, such equally controlled vocabulary, gratis-text, boolean operators, truncation and parenthesis.

Do you need search assistance? Read more about our support to students and researchers nether Contact us.

A fundamental office of a systematic review is an exhaustive literature search that finds all relevant studies on a topic. Hence, it is of import that the search strategy is rigorously adult with a high sensitivity to find all these potential relevant articles. This means that y'all volition end up with a large amount of references, often more than thousands of references. A loftier per centum will likely be irrelevant.

Information technology is as well important to comport the search in several databases. Cochrane recommends using at least three. In biomedicine and wellness, standard databases include PubMed/MEDLINE, Embase and Cochrane Library. Depending on the topic, a multi-disciplinary database like Web of Science or subject specific databases, such equally CINAHL, PsycInfo or ERIC, should also be considered.

In improver, gray literature, for instance dissertations and clinical trials, should be considered in your search. These publication types demand other databases. The literature search is commonly too supplemented with a screening of reference lists, article suggestions from colleagues and sometimes with a citation analysis – i.e. an analysis of which articles have cited an older, still relevant study.

Sensitivity, specificity and precision

To construct an exhaustive search strategy, and at the same time avoid too many references to screen, is a challenging task. In library and information science, the concepts of sensitivity, precision and specificity are used. According to the Cochrane Handbook, sensitivity "is defined as the number of relevant reports identified divided by the total number of relevant reports in existence." A high sensitivity search strategy should thus think all relevant studies on a topic. Precision and specificity state the part of not-relevant literature in the search.

Sensitivity and precision/specificity are almost ever irreconcilable: a highly sensitive search is too often less precise. This is illustrated in the effigy below, where the less theoretical concepts wide (sensitive) versus narrow (specific) search are used.

illustration_bredlitteratur_smallitteratur_eng

Modified figure from the SBU handbook, p. 34

For a systematic review, the search strategy should be highly sensitive. A large portion of the search result will thus not be relevant. In systematic literature searching, a precision of ii-three percent is common, i.e. two-three references out of a hundred will be relevant.

Do non reinvent the wheel

Earlier yous create a search strategy, it is a skillful thought to await at what has already been washed. Start by collecting published literature reviews on the topic. Fifty-fifty if they are not totally updated or exactly lucifer your topic, information technology is oft worthwhile to have a look at the method department of the reviews; sometimes the search strategies are bachelor as supplemental textile.

In improver, at that place are validated search filters that might exist useful. Search filters, or hedges, are sets of search terms chosen to restrict a search to a selection of references, for instance based on report blazon (in Cochrane reviews usually randomised controlled trials) or method (for instance qualitative methods). A search filter is often developed in different variants based on different levels of sensitivity, specificity and precision.

Some search filters are integrated into databases like PubMed/MEDLINE, PsycInfo and CINAHL. In PubMed, Clinical Queries allows you to use search filters to restrict the search to clinical studies, genetic studies or systematic reviews.

Web sites for validated search filters:

ISSG Search Filters Resources
McMaster University
PubMed Search Strategies Blog

Create search blocks

A well-defined and articulate research question is an essential starting signal for a systematic search. To create a logical search strategy, always start by identifying the fundamental elements of the research question – i.e., establish what the master concepts of the topic are. With these concepts, you can then create the search blocks that class the basis for the search strategies used in the different databases.

We will employ this research question equally an example:

Does routine apply of inhaled oxygen in acute myocardial infarction improve patient-centered outcomes, in particular hurting and death?

Some of the potential concepts that could form the search blocks are marked in bold. The PICO structure is a common fashion of formulating clinical research questions: Population, Intervention, Control and Outcome. In general, you lot should not include all parts of the PICO question in the search. Focus is generally on the population and the intervention, in our example P = patients with myocardial infarction and I = inhaled oxygen.

The search blocks are combined with the boolean operator AND in the search strategy. A full general principle is that a search for a systematic review should consist of few blocks. More search blocks means a less sensitive search and a higher run a risk of missing relevant manufactures. Ii upward to maximum four search blocks is a dominion of thumb.

The boolean operator Non should be avoided in systematic search strategies. Non decreases the sensitivity of the search and the risk of missing relevant manufactures increases. Please notation that this applies to the concluding search strategy. Not can be a useful tool when constructing and comparison searches – see the section Analyse your search beneath.

We include ii concepts from our search topic: inhaled oxygen and myocardial infarction. These two concepts will constitute our 2 search blocks. The concept acute is thus excluded considering a eye attack is always an acute medical condition and therefore already implicitly included. We also exclude pain and decease from the outcome part.

Detect search terms

In every search block, you should include all relevant search terms and variants, combined with the boolean operator OR. In contrast to the number of search blocks, you should also try to include as many relevant terms equally possible, since this makes your search more than sensitive.

Use both bailiwick headings and free-text terms. In this way you will utilize the potential of the controlled vocabulary and retrieve articles where the authors used diverse terminologies. You will also think articles that lack subject field headings, or are indexed with other discipline headings than those y'all have used in your search.

In many databases, the default search setting is all fields or a combination of fields, which means searching in both free-text and discipline headings. Still, in a systematic literature search we emphasize the importance of specifying the search fields manually. This gives you more than control and facilitates a more logical and cohesive search strategy.

The process of finding relevant search terms is an important office of the systematic search. Normally you start off with a scoping search, with the subject field headings and synonyms you already know of. A scoping search will give y'all a sense of how much has been written on your topic; past screening titles, abstracts and subject headings more than relevant search terms tin be constitute.

You should besides identify a set of key articles, i.e. pregnant studies in your field. These fundamental articles should stand for exactly to your research question and be retrieved by your search strategy. In sum, the key articles could be used for both constructing and for validating your search strategy. If you don't recall these studies in your search, modify your search query.

An additional suggestion is to check the bibliographic data bachelor for the key articles: championship, abstruse, author keywords, subject headings etc. These might provide additional search terms.

There are tools specific tools for finding search terms which analyses a search query or a set of references. The tools below are adapted for PubMed, but the free-text terms are generic and can thus be used in all databases:

Yale MeSH Analyzer
MEDSUM
PubReminer

Subject area headings

In many databases, manufactures are for meliorate retrieval indexed with subject field headings or controlled terms from a thesaurus. In PubMed, the controlled vocabulary is MeSH: Medical Subject Headings.

Y'all need to observe subject area headings to build a good search strategy. Search for your key concepts in the controlled vocabulary and bank check what subject field headings your key manufactures are indexed with to observe relevant terms. Explore the hierarchies and related terms. Delight note that there might be several field of study headings for closely related concepts, and that these headings might be role of different hierarchies.

Check if some of the terms are new as subject headings. If so, you might also consider including the previous subject field heading used for indexing the concept.

Avoid limits for subject headings, such as subheadings or main concept, for example MeSH Major Topic in PubMed.

The default setting in most databases is the exploding of field of study headings, i.e. narrower terms of the subject area headings are included. In general, this functionality is helpful, but you might in some cases consider if a subject field heading should exist searched without existence exploded, i.eastward.non including hyponym.

Complimentary-text terms

Include all the relevant synonyms and spelling variations in championship and abstract from the fundamental manufactures and other relevant studies retrieved past the test search. The controlled vocabularies are useful tools to discover free-text terms. You should, of form, search for the field of study heading terms equally free-text likewise. In add-on, you tin find more costless-text terms among the synonyms of a subject heading. These synonyms are named differently in different databases. In PubMed, they are called Entry terms.

Narrower terms should likewise be considered every bit free-text. These terms are included automatically in the controlled vocabulary search, thanks to the explode functionality, only for the free-text search they take to be explicitly stated to be included. One example of this is different forms of myocardial infarction,STEMI andnon-STEMI,. Those terms are automatically included in the exploded MeSH search for Mycardial Infarction, simply take to be added in the free-text part.

Employ truncation to discover dissimilar variants of a word;therap* retrieves for example:

therapy
therapies
therapeutic

Quotation marks are useful for keeping words in phrases together. On the other hand, you need to exist conscientious with this in systematic searching: quotation marks make the search more than precise and hence can relevant literature exist missed. One alternative is to apply proximity operators, which lets yous specify the number of words that can appear between two search terms. Proximity operators thus retrieves different variations of phrases, for instance in the word order, as a dissimilarity to phrases with quotation marks.

Let us applyoxygen treatment every bit an example. If searching for this concept as a phrase, we'll miss to catch potential relevant references including:

"oxygen(HBO) treatment"
"treatmentwith oxygen"
"oxygenin the treatment"

However, these variations are retrieved if nosotros employ a proximity operator (oxygen NEAR/3 treatment in Web of science).

Spelling differences in British and American English

Some medical terms, a few of them quite mutual, are spelled differently in British and American English language. Consider if your search strategy includes a word with a different spelling. If and so, include both of them. Some databases can compensate for this automatically, so called lemmatization, but yous should not rely completely on this functionality.

Six examples (British vis-à-vis American English):

Tummyour / Tumor
Gynaecology / Gynecology
Coeliac / Celiac
Ageing / Aging
Behaviour / Beliefs
Labour / Labor

Read more: Spelling differences in medical terms in British and American English language.

Analyse your search

When a tentative search strategy has been constructed, you should analyse if your key articles have been retrieved by the search.

Conduct your search (A).
Do a new search (B) for your key articles merely, for case in PubMed by using PMID.
Which key manufactures are non retrieved by your searchSearch B NOT A. If all key articles are retrieved, the search result will be nil.

If key manufactures are not retrieved, you should analyse why. Try to identify which search block that causes the exclusion of the article. What search terms tin can be added to include this key article? Maybe you lot should consider removing a complete search block? However, sometimes it's non possible to recollect all primal articles and, at the aforementioned fourth dimension, take a reasonable precision.

You might also useNOT to find additional subject headings or free-text terms. Conduct a search (A) including subject headings only and one search (B) with costless-text just.And then search A NOT B. In this case, you'll call back all studies indexed past the relevant subject field headings just without the free-text terms you have included. Bank check for additional free-text terms in the title and abstruse. After that, do the reverse (B NOT A) to find more relevant subject area headings.

The next step is to screen the search results. Do the references represent to your topic? Because the search strategy should be highly sensitive, the precision will surely be quite low (hence many irrelevant hits). Even so, y'all should of grade likewise find some eligible references.

Maybe some of the search terms will generate a lot of simulated hits. If and then, you lot might consider to remove them. Y'all can use the Non operator again in order to analyse the search.

Conduct a search including the term (A).
Comport a search without the term (B).
SearchA NOT B.

Analyse the event. If you do not find any relevant hits, consider excluding the term from the search strategy.

Databases – examples and tips

Always use identical search blocks and complimentary-text terms in all databases. However, the search strategy has to be translated to the controlled vocabulary in each database and adapted to the field tags etc used in the different interfaces. In this part, we demonstrate our search example in four different databases: PubMed, MEDLINE (Ovid), Embase and Web of Science.

In improver to the search tips beneath, the database sail made by the search group at KIB is useful for creating and translating search strategies.

PubMed

In biomedicine, you lot'll usually begin the literature search in PubMed and thus include the subset MEDLINE – the larger office of indexed articles in PubMed. However, when conducting more than advanced search strategies, MEDLINE via the Ovid interface is commonly used (see below).

You are probably quite familiar with PubMed, and it is an easy database to start with. There are, nevertheless, a few things to consider:

You can utilise truncation, just only the terminal discussion i phrases.
Not all phrases are searchable, only the phrases included in the phrase index. Await out for fault messages or check if your phrase is searchable past browsing the phrase index: select Title/Abstract in the Advanced Search Builder, add your phrase and choose Show alphabetize list. If your phrase is not included in the phrase alphabetize it volition be separated and searched as independant terms combined with the boolean operator AND.
Do not trust the automatic term mapping, specify the fields to be searched instead to gain full command. Moreover, if yous are using truncation it will plough off the automatic term mapping. If you want to run into how the automatic term mapping works, search for a term without specifying which field to search in so click on Search Details.
If you are using field tags (as in the example below) it automatically translates into a phrase search, like to using quotation marks.
You cannot use proximity operators, in contrast to almost all other advanced databases.

Example PubMed

MEDLINE Ovid

In MEDLINE Ovid, the indexed office of PubMed is included, but also newly added references ("Epub Ahead of Print" and "In-Process") and other non-indexed material. Ovid is a database platform used for many different databases. Through the KI library subscriptions, yous likewise have access to the database PsycInfo via Ovid.

There are several advantages of searching MEDLINE via Ovid instead of PubMed. In Ovid it is possible to use proximity operators, truncate (without any limitations) and search for all phrases (as opposed to PubMed, which is express by the phrase alphabetize). Information technology is also easier to structure comprehensive and avant-garde search strategies. Information specialists tend to prefer MEDLINE Ovid and the platform is, in most cases, the standard interface for Cochrane Reviews.

Equally in the example beneath, we deconstruct our search strategy into lines and then combine them, get-go each line within a search block with OR, so the search blocks with AND.

Example ovid medline

Embase

There are several unlike options for searching in Embase (via embase.com). In the example beneath, the search strategy is divided in lines as in Ovid: one for each Emtree term, ane line for all free-text. These are then combined into one search block. Emtree is the controlled vocabulary in Embase, similiar but not identical to MeSH in PubMed. You can access the Emtree terms in the top carte under "Browse".

Embase example

Web of Science

Web of Scientific discipline (Core Collection) is a multi-disciplinary database that includes inquiry areas other than biomedicine and wellness. There is no controlled vocabulary in Web of Science and merely free-text searches are possible.

A few things to consider in Web of Science:

Employ quotation marks around phrases to proceed the words together for greater precision.
Truncation and proximity operators can be used and are more of import considering information technology'due south a free-text database.
Consider dividing the search strategy as in the example below, using one search field per search block.
The defaultBasic Search is a good start, merely besides familiarize yourself withSearch History for a meliorate overview and editing possibilities.

Example Web of Science

Structure and documentation

PRISMA

To be systematic implies a focus on structure, organization and documentation. Equally in all research, the review process should be transparently documented in all parts, reported clearly in the final publication, and reproducible.

Equally a support in the review procedure there are the PRISMA Guidelines: "an bear witness-based minimum set of items for reporting in systematic reviews and meta-analyses." PRISMA – an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses – consists, among other things, of a check listing and a menstruum diagram. Many journals, for example Lancet, BMJ and Plos One, endorse the guidelines stated by PRISMA and require a PRISMA flow diagram when publishing systematic reviews.

According to PRISMA, the search strategy should exist reported in total for at least one database; nonetheless, nosotros recommend attaching all search strategies as an appendix to the published article. The majority of journals allow you to upload supplemental fabric. If the consummate search strategies are attached, you lot merely need to briefly describe the search process in the methods section and thereafter refer to the supplemental material section.

In PRISMA, 2 items concern the literature search:

Item 7: Information sources – Describe all information sources (e.g., databases with dates of coverage, contact with study authors to place additional studies) in the search and date last searched.
Particular viii: Search – Present total electronic search strategy or at least one database, including any limits used, such that it could be repeated.

Protocol

As with a clinical trial, a protocol should be established for a systematic review; preferably, information technology should also be registered. This is actually one of the items for reporting in the PRISMA Guidelines. Some journals –for instance BMJ, The Lancet and British Journal of Dermatology – requires a registered protocol for systematic reviews. There are several reasons for this. For example, it increases the overall transparency of the review, making information technology more scientifically sound. Additionally, the registration of the protocol also adds visibility to your review. This can help to avoid duplication of the work.

There is also a guideline for developing protocols for systematic reviews: PRISMA-P and a specific database aimed for the registration of protocols for systematic reviews: PROSPERO – International Prospective Register of Systematic Reviews. It's also possible to publish protocols in the journal Systematic Reviews.

Search in PROSPERO using the phrase "karolinska" and see ongoing review projects by authors from KI/KS.

Reference management

Systematic literature searches e'er produce a large corporeality of references. Also, when searching in several unlike databases many duplicate references will announced – i.e., identical references that are included in more than one database.

Reference management software, such as EndNote, is recommended for collecting, storing and organizing your references. Using a reference manager, you tin organize your references in different groups, for instance, included and excluded studies. Yous can also remove duplicates by sorting the references by title or past using the duplicate functionality.

All the same, EndNote and other types of reference managers have limitations when information technology comes to supporting the systematic review process. Other software has therefore been developed and is dedicated to supporting the systematic review procedure, for instance in the screening of abstracts and references co-ordinate to inclusion and exclusion criteria. For the time being, KI offers no institutional licens on this kind of software.

Rayyan (costless of charge)
EPPI-Reviewer iv
Covidence
DistillerSR

An thorough summary of available tools can exist found at Systematic Review Toolbox.

Press – checklist for search strategies

As the literature search is a central part of a systematic review, it's important that the search strategy is of high quality. To ensure this, and to avoid errors, there are several options. One option is to involve an experienced librarian/information specialist in the review project. In either case, it's skilful if a second person has a thorough look at the search strategy. There is as well a tool for validating search strategies chosen Peer Review of Eastwardlectronic Search Strategies (PRESS).

PRESS was originally published in 2008–2010, only was revised in 2015. The Press checklist was originally made for expert searchers, such as librarians, but can likewise exist used by students and researchers when creating more extensive search strategies. The testify and conclusions in PRESS are based on a comprehensive research project.

Contact us

If you are a bachelor's or primary'south pupil, or a doctoral student writing a literature review for the half-time summary, delight book a consultation for a certain fourth dimension.

If you're a researcher or a doctoral student preparing a systematic review, please contact the search group.

Last updated: 2022-02-17

kemptonbunecand.blogspot.com

Source: https://kib.ki.se/en/search-evaluate/systematic-reviews