Nutrition Education Systematic Review Project: Methodology
(From Chapter 3. Methods, of the Nutrition Education Systematic Review Project full report.)
The Nutrition Evidence Library (NEL) was launched in July 2008 by the US Department of Agriculture’s Center for Nutrition Policy and Promotion. The NEL uses a rigorous, transparent, and reproducible methodology to conduct systematic review on food- and nutrition-related topics to support Federal nutrition policies and programs. The following section describes the systematic review methodology used to conduct a series of systematic reviews on nutrition education.
Nutrition Education Systematic Review Project: Roles and Responsibilities
When conducting systematic reviews, NEL staff are assigned to a NEL Project Systematic Review Management Team, and work with a Technical Expert Collaborate (TEC), a Stakeholder Group, Abstractors, and Peer Reviewers. The roles and responsibilities of each of these groups and individuals are outlined below.
NEL Systematic Review Management Team
This NEL systematic review project was planned, organized, and guided by a NEL Systematic Review Management Team composed of Federal nutritionists trained in systematic review methodology. This team was led by the Director of the Evidence Analysis Library Division (EALD) and included a Project Manager, Lead Analysts, a Research Librarian, and additional staff from the EALD. The Project Manager was responsible for leading, planning, organizing, and facilitating the work necessary for timely execution of the systematic review within an allotted budget. The Lead Analysts reviewed individual research questions, and the NEL Research Librarian developed the systematic review search strategy for the scientific articles. Specific responsibilities of the NEL Systematic Review Management Team included the following:
- Facilitated the initial planning and led development of the systematic review project protocol
- Directed the execution and quality control of the NEL systematic review project based on input from the TEC and Stakeholders in accordance with the principles and procedures outlined in the NEL systematic review methodology manual
- Developed and disseminated products of the review, including website content, a systematic review report, CNPP NEL Insights, and peer reviewed publications.
The NEL Systematic Review Management Team met regularly throughout the systematic review project from conceptualization of the project through completion of the final products. They coordinated regular meetings with the TEC and Stakeholder Groups through conference call and webinar. They also served as the conduit between the TEC and the Stakeholder Group, Abstractors, and Peer Reviewers.
Technical Expert Collaborative
The Technical Expert Collaborative (TEC) consisted of eight leading nutrition education experts, including researchers, nutrition educators, methodologists, communicators, and end users of the reviews. A list of TEC members and their affiliations is found in the Acknowledgements section (on page 4). TEC members assisted the NEL Systematic Review Management Team by reviewing and providing expert feedback to refine systematic review materials. Their expertise was needed to address specific issues related to the topic of nutrition education and to guide synthesis of the body of evidence to answer the systematic review questions posed. TEC members guided the systematic review process in the following ways:
- Reviewed and refined materials drafted by the NEL Systematic Review Management Team, including:
- Analytic framework and definitions of key terms to be used in the systematic reviews
- Research questions
- Including identifying outcomes, comparators, key confounders, or effect modifiers
- Inclusion/exclusion criteria used to initially select articles used in the review, search strategy, and literature selection
- Including guidance on potential search terms, databases, and literature search strategies
- Lists of included and excluded articles
- Data extraction plan
- Including guidance on format and structure of evidence paragraphs and tables
- Summaries of the body of evidence, including evidence worksheets with quality ratings
- Conclusion statements and grades assigned based upon the body of evidence
- Additionally, the TEC:
- Provided input on research recommendations
- Served as co-authors of any manuscript(s) that are submitted for peer-review publication
- Provided suggestions for communicating the findings of the review project.
NEL Project Stakeholder Group
Members of the Stakeholder Group included Federal employees who represent end-users of the review and possessed varying perspectives and expertise related to nutrition education. Specifically, the Stakeholder Group:
- Assisted in refining and prioritizing systematic review questions to ensure the questions were valuable for informing policy and programs
- Provided input on research recommendations and implications
- Identified strategies to communicate results.
NEL Abstractors are National Service Volunteers from across the United States with advanced degrees in nutrition or a related field. They received training to review individual research articles included in the systematic reviews and rate the methodological rigor (quality) of each study. They extracted evidence from the research articles and posted this information to data fields in evidence worksheet templates in the NEL online portal. The methodological rigor of each individual study was assessed using the Research Design and Implementation Checklist. Worksheets prepared by the Abstractors were reviewed by the NEL Systematic Review Management Team and provided a templated presentation of each article to assist the TEC in their review of the evidence.
Peer reviewers were individuals from USDA who reviewed and provided comment on the systematic review products. The peer reviewers provided written input after this draft report was produced. A variety of perspectives were sought to ensure that this report provides a transparent and comprehensive description of the review. The NEL Systematic Review Management Team, particularly the Project Leader, coordinated the peer-review and developed responses to comments.
Nutrition Education Systematic Reviews: Methodology
The NEL uses a rigorous, transparent, and reproducible methodology that was informed by the Agency for Healthcare Research Quality (AHRQ), the Academy of Nutrition and Dietetics (AND) (formerly the American Dietetic Association), and the US Cochrane Collaboration process. The NEL utilizes a six-step systematic review process to conduct systematic reviews. The steps include:
- Develop systematic review questions
- Create and implement literature search and sort plans
- Develop evidence portfolios
- Synthesize the bodies of evidence
- Develop conclusion statements and grade the evidence
- Describe research recommendations.
In addition, for this project, two systematic evidence scans were conducted. A systematic evidence scan includes the following steps:
- Develop a systematic review question
- Create and implement a literature search and sort plan
- Develop an evidence portfolio
- Prepare a preliminary description of the evidence and rationale for not completing a full systematic review
- Describe future systematic review recommendations.
Each step of these NEL processes and how it was applied to the Nutrition Education Systematic Evidence-Based Review Project is described below.
The first step of the evidence analysis process was the development of systematic review questions. In order to identify which questions would be addressed, NEL staff gathered input from both the TEC and the Project Stakeholder Group to identify topic areas of interest. The Stakeholder group and TEC members were asked to identify topic areas and/or questions that they felt needed to be reviewed in order to enhance current nutrition education practice.
Once key topic areas were identified, the PICO (Population, Intervention, Comparator, and Outcomes) method was used in order to focus each of the systematic review questions. In addition, an analytical framework was created to provide a visual map of key variables, such as the population, interventions, comparators, outcomes, and potential confounders, to be addressed within each review (Appendix A). The TEC also agreed to define “nutrition education” as “any set of learning experiences designed to facilitate the voluntary adoption of eating and other nutrition-related behaviors conducive to good health.” For each systematic review question developed, the following PICO components were used:
- Population: Children and adolescents in preschool through high school
- Intervention: A nutrition education intervention
- Comparator: No nutrition education intervention (control) or a different nutrition education intervention (comparison)
- Outcomes: Dietary intake-related outcomes.
After prioritization, the TEC agreed upon the following seven systematic review questions:
- What is the effect of nutrition education delivered via digital media and/or technology on children’s dietary intake-related behaviors? (“Digital media and technology”)
- What is the effect of nutrition education with parental involvement compared to no parental involvement on children’s dietary intake-related behaviors? (“Parental involvement”)
- Which type of educator who delivers nutrition education is most effective in changing children’s dietary intake-related behaviors? (“Type of educator”)
- What are the effects of combining changes to the food environment and nutrition education compared to either of these strategies alone on children’s and adolescent’s dietary intake-related behaviors? (“Food environment”)
- What are the effects of multi-component compared to single-component nutrition education interventions on children’s and adolescents’ dietary intake-related behaviors? (“Multi- vs. single-component interventions”)
- What factors mediate the effects of nutrition education interventions on children and adolescent’s dietary-intake related behaviors? (“Mediators”)
- What factors moderate the effects of nutrition education interventions on children and adolescent’s dietary-intake related behaviors? (“Moderators”).
Create and Implement Literature Search and Sort Plans
After the systematic review questions were developed, the NEL Project Management Team developed the literature search and sort plan used to identify scientific articles analyzed to answer each systematic review question. The search and sort plan includes the development of inclusion and exclusion criteria, identification of databases and search terms used to identify relevant articles, implementation of the search strategy, and selection of studies to include in each systematic review. The TEC reviewed and provided feedback on inclusion and exclusion criteria, the literature search strategy, and the list of articles included and excluded for each review.
Inclusion and Exclusion Criteria
Inclusion criteria for the nutrition education systematic reviews include studies conducted using:
- Human subjects
- Subject populations from countries with high or very high human development, according to the Human Development Index
- Children and adolescents aged zero to 18 years
- Subjects who were healthy or at elevated chronic disease risk
- randomized controlled trial, non-randomized controlled trial, quasi-experimental study designs
- School, home, and community settings.
In addition, articles were included if they were published in English in a peer-reviewed journal between January 1995 and December 2010 (or were available via on-line prepublication prior to December 31, 2010). If an author is included on more than one review article or primary research article that is similar in content, the paper with the most pertinent data/endpoints was included. If data/endpoints from both papers are appropriate, it was made clear that results are from the same intervention.
Exclusion criteria for the nutrition education systematic reviews include studies conducted using:
- Animals and in vitro models
- Subject populations from countries with medium or low human development, according to the Human Development Index
- Adults over the age of 18 years
- Subjects who were hospitalized, diagnosed with disease, and/or receiving medical treatment
- Studies with no control or comparison group
- Systematic review, meta-analysis, narrative review, prospective cohort, cross-sectional study, or case-control designs
Articles were excluded if they were not published in English, or were published before January 1995 or after December 2010. Articles, abstracts, and presentations not published in peer-reviewed journals (e.g., Web sites, magazine articles, Federal reports) were also excluded. Finally, if an author was included on more than one review article or primary research article that is similar in content, the paper with the most pertinent data/endpoints was included, and others were excluded.
Databases and Search Terms
Several databases were searched using a wide variety of search terms and key words, including subject headings such as MeSH and thesauri terms. The following is a list of databases and corresponding search terms that were used:
- PubMed: “child nutrition sciences/education”[mh] OR “nutritional sciences/ education”[mh] OR (nutrition* AND (“Health Knowledge, Attitudes, Practice”[mh] OR “health education/methods”[mh] OR “Health Behavior”[mh] OR “Health Promotion/methods”[mh])) Limits: Humans, English, All Child: 0-18 years, Publication Date from 1995; Teacher* AND (nutritionist* OR dietitian*) AND education*; ("nutrition education" OR "nutrition instruction") AND ( ((parent* OR teacher*) AND (dietician* OR nutritionist*)) OR fruit* OR vegetable* OR garden*); school* AND environment* AND food[mh] AND nutrition* AND education* ; ("School Health Services"[majr] OR "Schools"[majr]) AND ( “Health Knowledge, Attitudes, Practice”[majr] OR “child nutrition sciences/education”[mh] OR “Health Promotion/methods”[majr] OR “nutritional sciences/education”[mh] OR fruit[mh] OR vegetables[mh] OR food[majr])
- Embase: 'nutrition education'/exp AND ([article]/lim OR [article in press]/lim) AND ([newborn]/lim OR [infant]/lim OR [preschool]/lim OR [school]/lim OR [child]/lim OR [adolescent]/lim) AND [humans]/lim AND [english]/lim AND [embase]/lim AND [1995-2010]/py; nutrition OR diet OR food AND education AND mediat* AND ([preschool]/lim OR [school]/lim OR [child]/lim OR [adolescent]/lim) AND [embase]/lim NOT [medline]/lim
- EBSCOhost (CINAHL; ERIC; PsycINFO; PsycARTICLES; SocINDEX with Full Text; Education Research Complete): SU nutrition* n2 education* and behavior*; SU nutrition* n2 education* and (meat or (whole w1 grain*) or fruit* or vegetable* or eggs or dairy or milk or yogurt or snack* or fish* or seafood* or nuts or environment*); ("nutrition education" OR "nutrition instruction") AND ( ((parent* OR teacher*) AND (dietitian* OR nutritionist*)) OR fruit* OR vegetable* OR garden*)
Additional databases and search terms were used to search for studies related to the systematic review question on the use of digital media or technology to deliver nutrition education. The following is a list of the additional databases and corresponding search terms that were used for this question:
- PubMed: (Video Game* OR (“virtual reality” AND gaming) OR Computer game* OR Youtube OR Facebook OR Twitter OR Aps OR Wii OR DDR OR Advergam* OR Social media* OR Television[majr] OR Mobile phone* OR cell phone* OR Web-based learning* OR internet[majr] OR computers[majr] OR “Computer-Assisted Instruction”[mesh]) AND ((nutrition AND education) OR “child nutrition sciences/education”[mh] OR “nutritional sciences/ education”[mh] OR obesity OR overweight OR fruit* OR vegetables OR “Health Knowledge, Attitudes, Practice”[mh] OR “health education/methods”[mh] OR “Behavior Control/methods”[mh] OR “Health Behavior”[majr] OR “Health Promotion/methods”[mh]) Limits: Humans, English, All Child: 0-18
- EBSCOhost (CINAHL; ERIC; PsycINFO; PsycARTICLES; SocINDEX with Full Text; Education Research Complete): (((Video OR “virtual reality” OR computer*) AND gaming OR games) OR Advergam* OR (Mobile w1 phone*) OR (cell w1 phone*) OR "Web-based learning" OR internet ) and ( obesity OR overweight OR fruit* OR vegetables ) ( ((Video OR “virtual reality” OR computer*) AND gaming OR games) OR Advergam* OR (Mobile w1 phone*) OR (cell w1 phone*) OR "Web-based learning" OR internet ) and ( (nutrition* w8 education*) OR (nutrition* w8 behavior*) ) ( "social media" OR twitter OR facebook OR "virtual reality" OR youtube ) and ( obesity OR overweight OR fruit* OR vegetables ) (((Video or (virtual adj reality) or computer* or online) and (gaming or games)) or Advergam* or (Mobile near1 phone*) or (cell near1 phone*) or internet ) and ( (nutrition* near8 education*) or (nutrition* near8 behavior*))
- Education Fulltext (Wilson): (("Games" OR "Video game machines" OR "Virtual reality games" OR "Electronic games") Or ("Wireless communication systems" OR "Cellular telephones"))
Subject(s) AND (nutrition OR diet OR food OR "Behavior" OR "Family" OR "Parents") Subject(s) AND Limited to: PEER_REVIEWED ("Computers/Nutritional use" OR "Nutrition/Computer software") Subject(s) (“digital media” OR internet) Subject(s) AND (nutrition education)
- Global Health: (((Video OR virtual adj reality OR computer* OR online) AND (gaming OR games)) OR Advergam* OR (Mobile near1 phone*) OR (cell near1 phone*) OR internet ) and ( (nutrition* near8 education*) OR (nutrition* near8 behavior*) ) ( social adj media OR twitter OR facebook OR virtual adj reality OR youtube)) AND ( (nutrition* near8 education*) or (nutrition* near8 behavior*) )
Search Strategy and Study Selection
The NEL librarian conducted all database searches and performed initial title sorts to exclude articles that clearly did not address the question. NEL staff sorted abstracts and full text articles based upon approved criteria and developed a list of included and excluded articles (citing rationale for exclusion). Additionally, reference lists from review and primary articles were hand searched. TEC members reviewed and provided feedback on the sort lists. If TEC members identified relevant articles that were not on the sort list, or if results were too expansive or too limited, the subcommittee refined the search strategy and the search was rerun. NEL staff continued to monitor the literature for new articles through December 2010.
Relevant information from all included articles in each systematic review was assembled into evidence portfolios. For each study in a systematic review, the evidence portfolio includes an evidence worksheet with a study quality rating that reflects the assessment of methodological rigor of the study, as well as an evidence paragraph and overview table entry that summarizes the study methodology and results, as it relates to the systematic review question being addressed. The components of the evidence portfolio are described in more detail below.
Each article included in a review was assigned to a NEL Abstractor to analyze and extract key data into an evidence worksheet template. The quality, or methodological rigor, of each article was assessed using the Research Design and Implementation (RDI) Checklist, developed and validated by the Academy of Nutrition and Dietetics. The RDI Checklist is based on criteria outlined in the Agency for Healthcare Research and Quality (AHRQ) report on Systems to Rate the Strength of Scientific Evidence (West et al, 2002). The RDI checklist for primary research articles includes four relevancy questions and ten scientific validity questions (ADA, 2009) (Appendix B). Based on responses to the checklist, each article was assigned a quality rating, positive, neutral, or negative, that reflects the methodological rigor with which the research was designed and executed. NEL staff reviewed the accuracy and quality of each evidence worksheet and RDI checklist, and the TEC was provided the worksheets as a component of the evidence portfolio.
The NEL staff worked with the TEC to define the content of evidence paragraphs and overview tables for each systematic review question. Using the evidence worksheets, RDI checklists, and full text articles, NEL staff drafted evidence paragraphs and overview tables to summarize the evidence in a uniform way for study-to-study evaluation and comparison. The evidence paragraphs briefly summarized each included study and reported relevant data including: Authors, year, rating; population, location, sample size and subject age; description of the intervention and comparison/control condition; methods used to assess dietary intake-related outcomes; results related to dietary intake-related outcomes. The overview tables provided information parallel to the evidence paragraphs, as well as additional information about study subjects (gender, race/ethnicity, and SES) and key limitations of each included study.
Synthesize the Body of Evidence
TEC members and the NEL Systematic Review Management Team reviewed the portfolio of evidence and defined an approach to analyzing the body of evidence, based on:
- Intervention characteristics (e.g., methods used to deliver nutrition education, dose and/or duration)
- Comparators (e.g., whether the nutrition education intervention was compared to a control group that received no nutrition education, or a group that received a different kind of intervention)
- Subject characteristics (e.g., age, gender, race/ethnicity)
- Study location (e.g., whether the study was conducted in the US or not)
- Study outcomes (e.g., fruit and vegetable intake, sugar-sweetened beverage intake, fat intake).
The NEL Systematic Review Management Team worked to develop evidence summary overviews that enumerated the number, type, and quality of included studies, key definitions, statements about methodology and strengths and limitations of the body of evidence, key findings or trends, potential rationale for variations observed, and a discussion of findings. TEC members provided input throughout the development of the evidence summary overviews.
TEC members and the NEL Systematic Review Management Team reviewed the body of evidence and a conclusion statement was developed to answer each research question. Conclusion statements focused on areas of general agreement among the studies and when evidence addressed only one gender, age group, ethnicity, or level of health risk, this was reflected in the conclusion statement.
The NEL Systematic Review Management Team then facilitated an evaluation by the TEC members of the strength of the body of evidence supporting each conclusion using a pre-established set of criteria. These criteria were adapted and validated by the American Dietetic Association (now the Academy of Nutrition and Dietetics) based upon the original work by Greer and colleagues (Greer et al, 2000). Grading criteria included: Quality, quantity, consistency, generalizability, and magnitude of effect. The following grades were used to describe the strength of the evidence supporting their conclusion statements: I – Strong, II – Moderate, III – Limited, IV – Expert Opinion, and V – Grade Not Assignable. Appendix C provides more detail on the grading criteria.
Define Research Recommendations
Once TEC members and the NEL Systematic Review Management Team developed a graded conclusion statement for a research question, they identified research recommendations related to the question or topic area. The Stakeholder group also reviewed and provided feedback on the research recommendations.
 Contento IR, Senior author. The effectiveness of nutrition education and implications for nutrition education policy, programs and research. A Review of research. J Nutr Educ. 1995;27:279-418.
 West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, Lux L. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47. Prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011. AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality; April 2002.
 American Dietetic Association, Research and Strategic Business Development. Evidence Analysis Manual Adapted for the USDA Nutrition Evidence Library. Chicago, IL: American Dietetic Association; 2009.
 Greer N, Mosser G, Logan G, Halaas GW. A practical approach to evidence grading. The Joint Commission Journal on Quality Improvement. 2000;26:700-712.