I. Development of Evidence-based Clinical Practice Guidelines (CPGs)
II. Joint Development of CPGs with External Organizations (CMSS P 9)
IV. Systematic Evidence Review of CPG Clinical Topic (IOM Standard 4)
VI. Conflict of Interest (COI) Policy and Process (IOM Standard 2, CMSS-P 3, CMSS-C 7.5-7.8)
IX. Grading Evidence and Strength of Recommendation (CMSS-P 6; IOM Standards 5 and 6)
The American Academy of Family Physicians (AAFP) develops evidence-based clinical practice guidelines (CPGs), which serve as a framework for clinical decisions and supporting best practices. Clinical practice guidelines are statements that include recommendations intended to optimize patient care. They are informed by a systematic review of evidence, and an assessment of the benefits and harms of alternative care options. CPGs should follow a sound, transparent methodology to translate best evidence into clinical practice for improved patient outcomes. Additionally, evidence-based CPGs are a key aspect of patient-centered care.
This manual summarizes the processes used by the AAFP to produce high-quality, evidence-based guidelines. The AAFP’s development process adheres to the following standards and principles:
Clinical practice guidelines should be developed using rigorous evidence-based methodology with the strength of evidence for each guideline explicitly stated.
a. Definition: Clinical practice guidelines are statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options. Rather than dictating a one-size-fits-all approach to patient care, clinical practice guidelines offer an evaluation of the quality of the relevant scientific literature, and an assessment of the likely benefits and harms of a particular treatment. This information enables health care clinicians to select the best care for a unique patient based on his or her preferences.
b. AAFP’s Commission on Health of the Public and Science (CHPS) and Board of Directors provides oversight for the development and approval of its clinical practice guidelines.
c. Principles for Development (IOM 1.1, CMSS-P 11, CMSS-C): The IOM identified eight standards for developing trustworthy guidelines. The standards reflect best practices across the entire guideline development process, including attention to:
The Council of Medical Specialty Societies (CMSS) provides directions and standards for the development of clinical practice guidelines through the CMSS Principles for Guideline Development (CMSS-P) and the Code for Interactions with Companies (CMSS-C). Where possible, the standards outlined by the IOM (now the National Academies of Medicine) and CMSS are referenced in the corresponding sections below.
The AAFP advocates the development of explicit patient-centered clinical practice guidelines which focus on what should be done for patients rather than who should do it. When clinical practice guidelines address the issue of who should provide care, then recommendations for management, consultation, or referral should emphasize appropriate specific competencies rather than a clinician's specialty designation. The AAFP may participate with other medical organizations in the development of clinical practice guidelines (also known as practice parameters or clinical policies) when appropriate criteria are met.
When AAFP enters into a joint development of a CPG with external organizations, a Memorandum of Understanding (MOU) should be developed to guide the process.
a. A clinical topic for a new or updated CPG is first vetted by the Subcommittee on Clinical Recommendations and Policies (SCRP) of CHPS using the following criteria:
b. AAFP Board approval is obtained for topic nomination and collaborators.
c. Prior to topic nomination, potential co-nominators/collaborators are contacted to involve them in the process.
In most cases, the AAFP utilizes the Agency for Healthcare Research and Quality (AHRQ) for development of an independent systematic review of the evidence based on the key questions identified for the CPG.
a. Develop topic nomination proposal to AHRQ (CMSS P 5)
i. Include key clinical questions and parameters with patient-oriented outcomes prioritized
ii. Members and content experts assist in drafting and providing feedback on the key questions
iii. Include collaborators for co-nomination (if applicable)
b. AHRQ Evidence-based Practice Center (EPC)
i. Establish staff contact with AHRQ program officer and EPC staff for evidence report
ii. Include one or more family physicians to serve on the technical expert panel (TEP) of the EPC. Often the TEP members will also serve on the guideline development panel.
c. AHRQ EPC Evidence Report on Clinical Topic
i. Staff and SCPG members provide review and feedback on the draft evidence report as requested by AHRQ throughout the process
ii. The draft report can be used to begin development of the draft CPG
iii. When the final EPC evidence report is published and available, it is used to finalize the CPG.
Staff works with the chairs of SCRP and CHPS to form the Guideline Development Group using the process outlined below:
a. Identify AAFP GDG Chair through the CHPS and obtain approval from the Board of Directors
b. Identify family physicians panel members including at least one member of SCPG and one member of the Science Advisory Panel in addition to other CHPS and AAFP members.
c. Identify GDG members from collaborators including a patient representative or patient advocacy group(s) when available/appropriate
d. Obtain a Disclosure of Interest Form from all panel members (IOM Standards 2)
A conflict of interest (COI) is an important potential source of bias in the development of CPGs. A COI has been defined as a set of conditions in which professional judgment concerning a primary interest (guideline recommendations), is unduly influenced by a secondary interest (financial or intellectual interests) (Norris et al 2012 and Thompson DF 1993).
To limit both actual and perceived bias in guideline development, the AAFP has set forth the following policy for COI:
a. Prior to selection of the Guideline Development Group (GDG), individuals being considered for membership should declare all interests and activities potentially resulting in COI with development group activity, by written disclosure to those convening the GDG.
b. Disclosures should include activities that may be considered financial or intellectual COI as defined below:
i. Financial COI = material interest that could influence, or be perceived as influencing, an individual’s point of view
ii. Intellectual COI = “…activities that create the potential for an attachment to a specific point of view that could unduly affect an individual’s judgement about a specific recommendation” (Guyatt et al 2010).
c. Review and Management of COIs
d. Divestment
Members of the GDG should divest themselves of financial investments they or their family members have, and not participate in marketing activities or advisory boards, of entities whose interests could be affected by CPG recommendations.
e. Exclusions
In some circumstances, a GDG may not be able to perform its work without members who have COIs, such as relevant clinical specialists who receive a substantial portion of their incomes from services pertinent to the CPG.
a. Clinical practice guideline time-line and expectations
AAFP staff members, in collaboration with the GDG chair, will develop a time-line for the guideline being developed. This time-line will be distributed to GDG members during the first meeting of the GDG. Though this time-line is developed with the goal of adherence, it is recognized that circumstances during the development process may affect the time-line. Thus, this is a living document throughout the guideline process and should be updated as appropriate.
Expectations of GDG members and staff will be reviewed by the GDG chair during the first meeting of the GDG.
b. CPG outline
The GDG will work with the GDG chair and AAFP staff members to develop an outline of the proposed guideline. The outline will include the key questions from the evidence report, the potential draft recommendations, key points for supporting text, and identification of potential information for shared decision-making tables and implementation algorithms.
c. Conference calls
Conference calls will be convened at the start of the guideline development process and throughout as needed. AAFP staff members will work with GDG members to ensure availability for calls. When a member cannot be present on a call, that member will be provided opportunities to provide any written comments prior to the call and feedback to the meeting summary after the call.
d. Electronic communication
Electronic communication will be used throughout the guideline process. Reasonable response times are expected for electronic communications and deadlines for requested action items will be clearly stated in the communications from AAFP staff members.
d. CPG publication(s) and dissemination (CMSS P 9)
When AAFP collaborates with others on a joint guideline, it will be decided where publication is expected at the start of the collaboration. All parties will agree to the publication plan. For guidelines developed solely by the AAFP, the GDG and staff will identify an appropriate publication venue which may include a peer-reviewed journal and/or the AAFP website. Dissemination activities should also be identified early on to facilitate work load and collaboration. These activities can include one or more of the following:
a. Section IV of this manual described the AAFP process for nominating topics to AHRQ for a systematic review of the evidence. Once the systematic review has been completed, a draft evidence report is published by AHRQ. The GDG reviews the draft evidence report to determine if applicable for development of a guideline. Systematic literature review performed by the AAFP.
If more than 12 months has passed between the publication of an AHRQ evidence review and development of the guideline, an update of the systematic review will be conducted. The GDG and AAFP staff members will work with the AAFP librarian to perform the updated review. The librarian will use the same search criteria that were used in the AHRQ systematic review. Inclusion and exclusion criteria will be set a priori to determine which studies will be reviewed for quality. AAFP staff members review the updated search results and obtain articles relevant to the systematic review.
b. As outlined in section IV, AHRQ has a process for performing systematic reviews that is consistent with the 2011 Standards for Systematic Reviews from the IOM. The AAFP also uses this as a guide to ensure the systematic literature reviews we are performing or that we are using for guideline development are compliant with the best standards available. These standards include: establishing a team with appropriate experience and expertise to do the review, including those with content expertise; providing methodological expertise and other expertise as appropriate; ensuring any conflict of interest is managed with regard to the team; ensuring that there is user and stakeholder input as the review is designed and conducted; managing conflict of interest with regard to any individuals providing input into the review; and formulating the topic for review.
The standards also discuss “finding and assessing individual studies.” This includes steps such as:
i. Conducting a comprehensive search for the evidence. This step will likely include:
a. Working with a librarian, and
b. Searching appropriate databases, citation indexes and other sources for relevant information.
ii. Taking action to address potential bias in reporting of research results.
iii. Screening and selecting relevant studies. Here it is very important to include and exclude studies based on a priori specified criteria developed in the protocol. It is recommended that two or more people screen studies and that these reviewers are tested for accuracy and consistency in their reviews.
iv. Documenting the search strategy, including dates of searches and how each item identified in the search was addressed. If excluded, include the reason for exclusion.
v. If data is extracted for a meta-analysis, data collection should be managed appropriately. The IOM standards recommend that systematic review developers:
a. Use two or more researchers to extract relevant data from a report;
b. Link publications from the same studies to avoid duplication of data; and
c. Use data extraction forms that are pilot tested.
vi. Finally, at least two reviewers should critically appraise each study using the specified protocol and forms derived for the review.
Compiling evidence and assessing it for quality are important steps in a systematic review. The quality of the evidence should be linked to the strength of the recommendations in that guideline. Consistent with the IOM standards for systematic reviews, the AAFP uses a specified framework for assessing the quality of studies and providing strength for each recommendation.
a. GRADE methodology
The AAFP uses a modified version of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method to systematically examine research to rate the quality of the evidence, and designate the strength of a recommendation based upon that evidence. The GRADE system provides a transparent process and framework for developing evidence-based recommendations using the following system to rate the quality of evidence:
i. High Quality (Level A): Further research is very unlikely to change our confidence in the estimate of effect.
ii. Moderate Quality (Level B): Further research is likely to have an important impact on our confidence in the estimate of effect, and may change the estimate.
iii. Low Quality (Level C): Further research is very likely to have an important impact on our confidence in the estimate of effect, and is likely to change the estimate.
iv. Very Low Quality (Level D): Any estimate of effect is very uncertain.
b. Strength of Recommendation
GRADE uses the term “strength of recommendation” to rate the extent of confidence that the desirable effects of an intervention outweigh the undesirable effects. Recommendations can be either for or against an intervention or testing modality. The AAFP prefers the strength of the recommendation be consistent with the quality of the evidence such that strong recommendations are based on moderate to high quality evidence and weak recommendations are based on low to moderate quality evidence. Very low-quality evidence should be considered insufficient for a recommendation except when the benefits greatly outweigh the harms.
The strength of evidence should also reflect the degree to which there is evidence of improved patient oriented outcomes such as morbidity, mortality, quality of life, or symptoms (as opposed to only disease oriented outcomes such as blood pressure or hemoglobin A1C). Strong recommendations should be based on high quality evidence of improved patient oriented outcomes. Weak recommendations should be supported by some evidence of improvement in patient oriented outcomes; although, the evidence may be inconsistent, of lower quality, or rely on an indirect chain linking surrogate outcomes to patient oriented outcomes.
i. Strong recommendation: Based on consistent evidence of a net benefit in terms of patient oriented health outcomes, most informed patients would choose the option recommended, and clinicians can structure their interactions with patients accordingly.
ii. Weak recommendation: Evidence of a net benefit in terms of patient oriented outcomes is inconsistent or is based on lower quality evidence, or patient choices will vary based upon their values and preferences, and clinicians must help to ensure that patient care stays true to these values and preferences.
iii. Good practice points: These are recommendations that can be made when it is deemed they will be helpful to the clinician, such as recommendations to perform something that is standard of care, but where there is no direct evidence to support the recommendation and that is unlikely to ever be formally studied. These should be used sparingly in guidelines.
c. Upgrading and downgrading evidence: The GRADE system allows the evidence to be upgraded or downgraded based upon specific criteria.
i. Downgrading evidence: Evidence may be downgraded due to the following reasons:
1. Risk of bias refers to factors that make it less likely that the answer found in the study may not represent the true answer in the population. Faulty randomization, such as lack of concealment at allocation to the study group; lack of blinding to the study group when assessing outcomes; large losses to follow-up; the failure to analyze everyone in the group to which they were randomized; stopping the study early when the benefit seems too great to ignore; or failure to report all outcomes.
2. Inconsistency of findings across a number of studies must be explained. Were the interventions really the same? Were the samples very different? Inconsistencies that cannot be explained make it very difficult to assess the true effect of the treatment.
3. Directness refers to the extent to which two interventions are being compared to each other in similar populations. Indirect comparisons are more difficult to interpret. Two types of indirectness exist.
a. The first includes indirect comparisons. For instance, if two drugs are being examined for an outcome, but there are no studies that directly compare the drugs, which is an indirect comparison.
b. The second includes differences in population, intervention, comparator, and/or outcome.
4. Imprecision refers to a study that may show statistically significant effects, but the sample size is small and the measure of benefit is imprecise, meanin that it has a wide confidence interval.
5. Publication bias may also exist. Investigators are more likely to submit studies for publication when the results are positive and journals may be more likely to accept them for publication. An effort should be made in a systematic review to uncover studies that have not been published. This is a particularly important issue when the studies are funded by industry
ii. Evidence may be upgraded based upon the following factors:
1. Large effect size: A large effect is much less likely to be spurious than a small effect. Small effect sizes can much more easily result from chance.
2. Dose response: This exists when there is evidence that differences in dosage result in different effects/outcomes. This is one aspect of a finding that suggests an association based on cause and effect.
3. All plausible confounding: In observational trials, it is particularly difficult to measure and control for all plausible confounding. When all unmeasured plausible confounders and biases in an observational study would result in an underestimate of an apparent treatment effect, then it is more likely that a finding is real rather than the result of unmeasured confounding. For instance, if only sicker patients receive an experimental intervention or exposure, yet they still fare better, it is likely that the actual intervention or exposure effect is even larger than the data suggest.
a. Scope of the guideline (CMSS-P 5.1-5.3)
The AAFP includes the intent, rationale, and scope in all guidelines. This includes, but may not be limited to, the appropriate users of the guideline, situations in which the guideline should be used, and appropriate patient populations for the guideline.
b. Methodology
Information regarding the methodology used in the evidence report and in any updated literature searches will be provided in the guideline. This information should include search terms, search dates, outcomes assessed, and key questions that were addressed. A summary of included and excluded articles should be made available for all literature searches performed by the GDG. This information can be included in the guideline as a PRISMA diagram. Records of the study quality assessment should be maintained.
c. Moving from evidence to recommendations (IOM standards 5 and 6; CMSS-P section 6 and 7)
i. As noted in section IX, the AAFP uses a modified GRADE methodology for rating the quality of the evidence, and guiding the
strength of recommendations. It is worth noting that moving from examining the evidence to making a recommendation is where
much of the disagreement happens in guideline development. Different groups that develop guidelines may disagree on how
much weight they give to lower-level evidence; may not fully take into account benefits and harms, costs or burdens; and may
give differing emphasis on patient or provider preferences and values. However, all of these factors should be considered when
making recommendations. AAFP’s use of the GRADE system helps to systematically examine many of the factors mentioned
to determine the quality of the evidence and strength of recommendations.
ii. The AAFP strives to only make strong recommendations based on high-level evidence. However, there are few instances where
strong recommendations can be made based on moderate or low-level evidence. In these instances, there must be certainty
that benefits outweigh harms.
iii. Recommendations made include an explanation of the reason for the recommendation; description of benefits and harms; a
summary of the relevant available evidence; any explanation of values and preferences that went into the recommendation; a
rating of the level of evidence and strength of recommendation; and differences in opinions of GDG panel members, if they exist,
for that recommendation.
iv. Recommendations made are specific and actionable and worded such that it is clear to the reader that they are (1) strong recommendations, (2) weak recommendations, or (3) good practice points. Suggested language for each type of recommendation is shown below. A table should be included in the guideline to outline the methodology and can highlight the differences between the wording used for the strength of the recommendation.
1. Strong recommendation: Use directive language in the recommendation such as “Family physicians should discuss…” or “Do not order a chest radiograph for children with suspected pneumonia unless...”. The statement should begin with “The AAFP strongly recommends…”.
2. Weak recommendation: Language should reflect the lower quality of evidence and the lower level of certainty regarding the recommendation. Suggested wording includes “Consider offering counseling regarding…” or “Patients may wish to consider…”. The statement should begin with “The AAFP recommends”.
3. Good practice points: Language should reflect the low quality or absence of evidence. “Although not studied in clinical trials, it is standard of care to perform an ECG in patients presenting with chest pain.”c.
d. Panel assignments
With direction from the GDG chair, members of the GDG will be given writing assignments to complete during guideline development. When possible, GDG members will be asked for preferences regarding sections of the guideline they would like to write.
e. Making the CPG implementable
i. For implementation, the recommendations should be specific and provide clear direction. The number of recommendations
should be kept to a minimum.
ii. Access to the guideline should be provided through publication in a journal, the AAFP website, and the guideline clearinghouse.
(CMSS 10.1-10.2)
iii. When available or appropriate, actions should be taken to incorporate the recommendations at point of care through electronic
health records (EHR) reminders or toolkit/checklist for physicians. (CMSS 10.3)
iv. Additional implementation methods include mass media campaigns (news article, leadership blog, other avenues as suggested
by the AAFP content strategy team—see dissemination section), and interactive educational meetings with quality
improvement resources as appropriate (expanded learning session at Family Medicine Experience [formerly Assembly],
workshops)
e. Compilation of draft(s)
All drafts of the guideline should be sent (or made available) to the GDG chair and staff members at the AAFP. Most often, staff members at the AAFP will compile all sections of the draft guideline and the chair will review the draft(s) before it is sent to other members of the panel.
a. Internal Peer Review
The first round of peer review of the CPG is conducted by members of SCPG and the Science Advisory Panel. All reviewers are given four weeks to complete and return their review form to the staff members at the AAFP. (see Appendix A for an example of the review form). Upon receipt of the reviews, all comments will be recorded. Comments will be addressed when the chair determines that there is a need. A written record will be kept of the rationale for responding or not responding to all comments received.
b. External reviewers (including collaborating organizations)
Relevant stakeholders are included in the external review, including collaborating organizations, and organizations that may be affected by the guideline. AAFP members who are identified as experts in the field may also be asked to participate in the review. All reviewer comments are collected and recorded. A record of how the comment was addressed is kept. Reviewers’ names are kept confidential unless a reviewer wants to be recognized for his or her review.
The draft guideline will not routinely be made available for a period of public comment, but will be reviewed by key stakeholders including patient advocacy groups if a patient voice was unavailable for inclusion on the guideline panel.
a. Following the peer review process, the revised CPG is reviewed by members of CHPS. Upon approval, a recommendation is made to the full commission, which upon approval makes a recommendation to the AAFP Board of Directors for approval.
b. Board of Directors
The AAFP Board of Directors reviews the guideline. Any questions from the Board are addressed by the GDG, and staff at the AAFP.
c. Endorsement by collaborators
Collaborators on the guideline are given the chance to endorse the guideline after approval by the Board of Directors and before it is published. The collaborators will be sent the embargoed guideline, and given a month to decide upon endorsement.
a. Peer-reviewed journal
Upon completion, the AAFP submits its guidelines for publication in a peer-reviewed journal, as appropriate. The guideline manuscript undergoes independent editorial review, and a decision is made about publication. Due to the nature of journals, all supporting materials (such as tables with quality ratings of studies) may not be able to be published. All supporting materials that are relevant to the guideline that are not published will be made available on the AAFP website.
b. Copyright issues
Copyright issues are negotiated with the publication journal with appropriate licensing agreements made to the AAFP.
c. AAFP website (CMSS-P 9.1)
After publication, the guideline is placed on the AAFP website for easy accessibility. Supporting documents that were not published with the original guideline will be available on the AAFP website as well.
a. Dissemination/marketing plan
The AAFP guideline staff members work with other divisions including Communications and Marketing to disseminate the guideline upon publication. Staff members also submit the published guideline to the National Guidelines Clearinghouse for dissemination. Any derivatives made relating to the guideline will also be publicized via a marketing plan.
a. Determination by CHPS
All guidelines developed by the AAFP are scheduled for a review five years after completion. However, literature pertaining to a guideline is monitored regularly, and if it is deemed necessary, a review can be initiated sooner. Whichever the case, when a guideline review is initiated, a preliminary search of the literature is completed and brought to the commission to determine if a new systematic review is necessary. If so, the topic will be nominated to AHRQ for a full systematic evidence update. If not, a decision whether to reaffirm the guideline for additional time not to exceed five years, or sunset the guideline. The commission’s recommendation is then approved by the Board of Directors.
a. External guidelines may be reviewed for endorsement by the AAFP following a request from another organization, a request from a member, or if identified as having a high applicability to family medicine.
b. The guideline will then be reviewed using set criteria (see Appendix A for review form) by members of the commission, or AAFPmembers at large with appropriate expertise and/or review experience. In certain cases, staff may review guidelines from selected organizations.
c. The member or staff reviewers will then submit comments and a recommendation to staff (see Appendix B for AAFP’s Endorsement Policy).
i. Endorse—the AAFP fully endorses the guideline
ii. Affirmation of Value to Family Physicians—the guideline does not meet the requirements for full endorsement, or the AAFP is not able to endorse all the recommendations, but feel the guideline is of some benefit for family physicians
iii. Not Endorse—the AAFP does not endorse the guideline and the reasons are stated. The AAFP may also choose to remain silent.
d. Reviewer comments and recommendations will be collated and reviewed by staff and the chairs of CHPS and SCRP. If substantial differences occur, the reviewers will discuss and determine if a consensus can be reached.
e. A recommendation will then be given to the SCRP for approval, and if approved, will then be taken to the commission for approval.
f. A recommendation describing the commission’s action will be submitted to the Board or board chair for approval.
g. Once approved, the organization will be notified by staff and a summary of the key recommendations, and a link to the full guideline will be posted on the AAFP website. Only guidelines with endorsement or affirmation of value will be placed on the website.
h. External guidelines designated as endorsed, or affirmation of value will be reviewed every five years following their date of publication. Guidelines may be reviewed earlier if new evidence warrants an update.
REFERENCES
Council of Medical Specialty Societies: Code for Interaction with Companies
Council of Medical Specialty Societies: CMSS Principles for the Development of Specialty Society Clinical Guidelines
Institute of Medicine. Clinical Practice Guidelines We Can Trust. Washington, DC. National Academies Press. 2011. https://www.nap.edu/catalog/13058/clinical-practice-guidelines-we-can-trust. Accessed November 15, 2017.
IOM (Institute of Medicine). Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC. National Academies Press. 2011.
http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx. Accessed November 15, 2017.
Agency of Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2014. AHRQ Publication No. 10(14)-EHC063-EF. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/conflict-management_research.pdf. Accessed Nov. 1, 2016.
Norris SL, Holmer HK, Burda BU, Ogden LA, Fu R. Conflict of interest policies for organizations producing a large number of clinical practice guidelines. 2012; PLoS One. 7(5):e375413.
Thompson DF. Understanding financial conflicts of interest. 1993; N Engl J Med. 329(8):573-6.
For information about the AAFP Guideline Endorsement Form, please contact Melanie Bird, PhD, MSAM, Senior Manager, Clinical and Health Policies at (800) 906-6000 extension 3165, or mbird@aafp.org.
Board Approved: December 2017
B. AAFP Endorsement of Clinical Practice Guidelines
E. Grading of Recommendations Assessment, Development and Evaluation (short GRADE) Working Group