Increase Font Size Decrease Font Size View as PDF Print

Nutrition Evidence Library Systematic Review Methodology


Extracting Data and Assessing Risk of Bias

Information from each study included in a systematic review (SR) is extracted and summarized to describe the body of evidence available to answer the SR question. Key data relevant to the SR question are extracted by NEL Fellows/Abstractors based on a data extraction template developed by NEL and the TEC, and an Evidence Grid is developed that includes all of the data extracted for each study included in the review. In addition, the risk of bias (i.e., internal validity) for each study is assessed using the NEL Bias Assessment Tool (BAT). These materials are then used to develop a description of the evidence that is used by the TEC in their review and synthesis of the body of evidence.
 
A Closer Look: Data Extraction
The data extraction from each article included in a systematic review should provide an overview of the methodology and key findings of an individual study as it relates to the SR question being addressed. Standard types of data extracted include:
  • Sample size
  • Location
  • Subject characteristics: age, gender, race/ethnicity, socioeconomic status, health status
  • A description of the study
  • Study duration
  • Dietary assessment method
  • Description of the independent variables
  • Description of the outcome(s) measured and methods of outcome assessment
  • Statistical adjustments/models
  • Results
  • Risk of Bias rating and limitations
  • Funding source
A Closer Look: Assessing Risk of Bias using the NEL Bias Assessment Tool
The NEL Bias Assessment Tool (NEL BAT) is used to assess the risk of bias (i.e., internal validity) of each individual study included in a SR to determine whether any systematic error exists to either over- or under-estimate the study results. The types of bias that are addressed in the NEL BAT include:
 
Selection Bias Systematic differences between baseline characteristics of the groups that are compared; error in choosing the individuals or groups taking part in a study
Performance Bias Systematic differences between groups in the intervention/exposure received, or in experience with factors other than the interventions/exposures of interest
Detection Bias Systematic differences between groups in how outcomes are determined; outcomes are more likely to be observed or reported in certain subjects
Attrition Bias Systematic differences between groups in withdrawals from a study, particularly if those who drop out of the study are systematically different from those who remain in the study
Adapted from: Cochrane Bias Methods Group: http://bmg.cochrane.org/assessing-risk-bias-included-studies
 
The NEL BAT is tailored by study design, with different sets of questions applying to randomized controlled trials (14 questions), non-randomized controlled trials (14 questions), and observational studies (12 questions). Fellows/Abstractors complete the NEL BAT after data extraction for the article is complete, answering the questions based on the SR question being addressed. There are four response options:
  • Yes: Information provided in the article is adequate to answer “yes”.
  • No: Information provided in the article clearly indicates an answer of “no”.
  • Cannot Determine: No information or insufficient information is provided in the article, so an answer of “yes” or “no” is not possible.
  • N/A: The question is not applicable to the article.
 The NEL Bias Assessment Tool (NEL BAT)
Risk of Bias Questions Study Designs Type of Bias
Were the inclusion/exclusion criteria similar across study groups? Controlled trials
Observational studies
Selection Bias
Was the strategy for recruiting or allocating participants similar across study groups? Controlled trials
Observational studies
Selection Bias
Was the allocation sequence randomly generated? RCTs Selection Bias
Was the group allocation concealed (so that assignments could not be predicted)? RCTs Selection Bias
Performance Bias
Was distribution of health status, demographics, and other critical confounding factors similar across study groups at baseline? If not, does the analysis control for baseline differences between groups? RCTs
Controlled trials
Observational studies
Selection Bias
Did the investigators account for important variations in the execution of the study from the proposed protocol or research plan? RCTs
Controlled trials
Observational studies
Performance Bias
Was adherence to the study protocols similar across study groups? RCTs
Controlled trials
Observational studies
Performance Bias
Did the investigators account for the impact of unintended/unplanned concurrent interventions or exposures that were differentially experienced by study groups and might bias results? RCTs
Controlled trials
Observational studies
Performance Bias
Were participants blinded to their intervention or exposure status? RCTs
Controlled trials
Performance Bias
Were investigators blinded to the intervention or exposure status of participants? RCTs
Controlled trials
Performance Bias
Were outcome assessors blinded to the intervention or exposure status of participants? RCTs
Controlled trials
Observational studies
Detection Bias
Were valid and reliable measures used consistently across all study groups to assess inclusion/exclusion criteria, interventions/exposures, outcomes, participant health benefits and harms, and confounding? RCTs
Controlled trials
Observational studies
 
Detection Bias
Was the length of follow-up similar across study groups? RCTs
Controlled trials
Observational studies
Attrition Bias
In cases of high or differential loss to follow-up, was the impact assessed (e.g., through sensitivity analysis or other adjustment method)? RCTs
Controlled trials
Observational studies
Attrition Bias
Were other sources of bias taken into account in the design and/or analysis of the study (e.g., through matching, stratification, interaction terms, multivariate analysis, or other statistical adjustment such as instrumental variables)? RCTs
Controlled trials
Observational studies
Attrition, Detection, Performance, and Selection Bias
Were the statistical methods used to assess the primary outcomes adequate? RCTs
Controlled trials
Observational studies
Detection Bias

The completed NEL BAT is used to rate the overall risk of bias for the article by tallying the responses to each question. Each “Yes” response receives 0 points, each “Cannot Determine” response receives 1 point, each “No” response receives 2 points, and each “N/A” response receives 0 points. Since 14 questions are answered for randomized controlled trials and non-randomized controlled trials, they will be assigned a risk of bias rating out of a maximum of 28 points; while observational studies will be out of 24 points. The lower the number of points received, the lower the risk of bias.
 

More information on the Nutrition Evidence Library Systematic Review Methodology



Last Updated: 05/05/2014