Published on in Vol 4 , No 1 (2019) :Jan-Dec

Preprints (earlier versions) of this paper are available at, first published .
An Analytics Framework for Physician Adherence to Clinical Practice Guidelines: Knowledge-Based Approach

An Analytics Framework for Physician Adherence to Clinical Practice Guidelines: Knowledge-Based Approach

An Analytics Framework for Physician Adherence to Clinical Practice Guidelines: Knowledge-Based Approach

Authors of this article:

Jaehoon Lee 1, 2 Author Orcid Image ;   Nathan C Hulse 1, 2 Author Orcid Image

Original Paper

1Intermountain Healthcare, Salt Lake City, UT, United States

2Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States

Corresponding Author:

Nathan C Hulse, PhD

Intermountain Healthcare

36 State Street

Salt Lake City, UT, 84111

United States

Phone: 1 3852976482


Background: One of the problems in evaluating clinical practice guidelines (CPGs) is the occurrence of knowledge gaps. These gaps may occur when evaluation logics and definitions in analytics pipelines are translated differently.

Objective: The objective of this paper is to develop a systematic method that will fill in the cognitive and computational gaps of CPG knowledge components in analytics pipelines.

Methods: We used locally developed CPGs that resulted in care process models (CPMs). We derived adherence definitions from the CPMs, transformed them into computationally executable queries, and deployed them into an enterprise knowledge base that specializes in managing clinical knowledge content. We developed a visual analytics framework, whose data pipelines are connected to queries in the knowledge base, to automate the extraction of data from clinical databases and calculation of evaluation metrics.

Results: In this pilot study, we implemented 21 CPMs within the proposed framework, which is connected to an enterprise data warehouse (EDW) as a data source. We built a Web–based dashboard for monitoring and evaluating adherence to the CPMs. The dashboard ran for 18 months during which CPM adherence definitions were updated a number of times.

Conclusions: The proposed framework was demonstrated to accommodate complicated knowledge management for CPM adherence evaluation in analytics pipelines using a knowledge base. At the same time, knowledge consistency and computational efficiency were maintained.

JMIR Biomed Eng 2019;4(1):e11659



Clinical practice guidelines (CPGs) are systematically developed statements that assist clinicians in making decisions about appropriate patient care for specific clinical circumstances [1]. Since the nature of CPGs includes complicated clinical knowledge, it is known to be challenging not only to formulate clinicians' logics into the form of guidelines, but also to translate and implement these guidelines properly into clinical tasks and processes. Therefore, a number of studies have tried to develop systematic ways to implement CPGs [2,3], including computer-aided clinical decision support (CDS)-based approaches that enable personalized and timely implementation of CPGs [4-6] and knowledge-based approaches to systematically transform complicated knowledge of CPGs into clinical decision and practices [7-9].

Since clinical knowledge within CPGs originated from evidence of best practices, realization of CPGs in practice has had a positive impact on clinical workflow and patient outcome [10-14]. Therefore, it is important to measure and evaluate physicians’ adherence to CPGs in order to understand how providers are following guidelines in the postimplementation phase [15,16]. This evaluation may be an interdisciplinary project involving domain experts, knowledge engineers, and data analysts, among others. Domain experts should derive evaluation logics and metrics from CPGs and document them by collaborating with knowledge engineers. To perform this evaluation, database engineers should create queries based on the definitions and run them against the clinical database. As a result of data extraction, evaluation outcomes would be delivered to consumers (ie, clinical champions or management leaders), often in the form of Web-based reports.

The problem is that knowledge gaps always exist while transforming logics of CPGs into evaluation-definition documents, developing queries, and generating reports. For example, translating definitions into computationally executable queries may vary by individual knowledge engineers and data developers [17]. Once analytics are delivered, consumers can only see the resulting data, but would lack an understanding of what logics were used to extract the numbers. If analytics are being delivered in a regular manner and evaluation logics are modified, it becomes confusing to know whether a report was made based on an old or new definition. In particular, analytics have become more integrated and automated by integrating data pipelines, report generation, and delivery workflow; this may involve more knowledge translation into the framework and may require a systematic method of management [18,19].

Knowledge gaps are caused by the difficulty of handling too many data points, miscommunication between domain experts and developers, misinterpretation of guidelines, loss of authorship of documents, and revision and update of knowledge sources [20]. This may result in data inconsistency across the analytics pipeline, causing consumers to experience a lack of trust regarding the analytics results. Consumers may be confused and may feel that “these numbers don’t make sense,” but it is difficult to understand the problem with the current analytics process and where it lies in the pipeline.

To address this limitation, we propose an analytics framework whereby data and visualization pipelines are integrated with a knowledge base. A knowledge base is a tool designed to store clinical knowledge content in a systematic way by managing attributes of content authorship, version, and relationship between resources [21,22]. We used a knowledge base as a key component to store CPGs, their adherence definitions, and their executable queries, so that the related documents in their different forms could be managed with metadata and easily shared by domain experts, query developers, and analysts. In addition, we connected data pipelines to executable queries in the knowledge base so that a change of query can be immediately incorporated into the analytics pipelines.

In a pilot study, we adopted locally developed CPGs used in our health care system, resulting in a care process model (CPM). We developed an integrated analytics framework that consists of a knowledge base that employs CPMs in different forms: a commercial data pipelining tool­, Alteryx, connected to our clinical database and a commercial visualization tool, Tableau, to generate reports. We built a dashboard that provides views for adherence by physicians to 21 CPMs. Over an 18-month-long proof-of-concept project, we ran a working group to analyze CPM adherence and to manage definition revisions. We investigated how the use of the knowledge base contributed to the filling of knowledge gaps and automating of analytics pipelines with computational efficiency.

Subject: Care Process Models

In this study, we used CPMs that were created using locally developed CPGs in our health care organization and that were designed to reduce clinical variation, improve quality, and support local preferences [23]. The CPMs were published and managed by clinical programs, which are made up of clinical expert groups consisting of clinical champions, medical directors, nursing administrators, data managers, and data analysts. Over the last 20 years, clinical programs have developed over 120 CPMs that cover a variety of clinical conditions and procedures, such as hypertension, heart failure, breast cancer, appendicitis, and acute myocardial infarction, among others.

CPMs were originally developed as paper or electronic documents containing descriptions of target problems or procedures, logics of decision-making, and recommended actions. Traditionally, CPM implementation was conducted through the involvement and education of care teams and providers. Over the last four years, we have installed a new enterprise-wide electronic health record (EHR) system in our hospitals and clinics in Utah, USA. We have also started developing computationally executable CPMs inside the EHR system using a variety of decision-support components, including order sets, decision-support rules, and care pathways (ie, decision flow and state-based order recommendation tools), among others.

In addition to CPM implementation, there was a strong need from the clinical leadership to monitor and evaluate how providers are complying with CPMs. Analyzing such data may allow us to understand how well-embedded CPMs are within clinical practices and may allow us to gain insights into how to improve best practices within them. However, it has been challenging to quantitatively measure whether CPMs were used as intended after implementation, since key data points for the evaluation are complicated to define and capture. In addition, time-consuming manual data processing to calculate evaluation metrics was required. To address these problems, clinical programs and informatics specialists initiated an effort to build a framework that creates a systematic approach for data extraction and visual analysis within the evolution cycle of CPM development, implementation, and improvement.

To correctly evaluate adherence to CPMs, three types of information should be defined. First, since CPMs are developed to treat patients with certain conditions, a target population (ie, patient cohort) should be identified. Specifically, a combination of inclusion and exclusion criteria should be defined, including patient demographics, diagnosis, lab results, and medications. Second, metrics to quantitatively measure CPM utilization for the defined target population should be defined. A timeline of when to develop the metrics, typically key concepts extracted from logics and actions in CPMs, should be included since it is unrealistic to capture all of the concepts within the CPMs. Examples of key concepts include the following: diagnosed with pneumonia, image ordered for d-dimer, or 1-90 days old with a fever(≥38°C). Third, an adherence formulation is needed that would be used to count credits given to providers or care units who utilized CPMs as intended for target patients.

Although CPMs are well-described guidelines, it is often challenging to derive the key information above, as the nature of the CPMs are composed of complicated, domain-specific knowledge and often contain ambiguity. In addition, it is difficult to connect the derived key concepts to data points—often tables or columns in clinical databases—in real-world data sources. Thus, a multidisciplinary collaboration effort may be required in order to transfer knowledge between the various experts below:

  1. Domain knowledge expert: an expert, author, or publisher of CPMs. This individual would be responsible for interpreting and defining the highest levels of evaluation criteria.
  2. Domain data expert: an individual with both clinical domain and database expertise. This individual would be responsible for translating CPM knowledge and linking concepts to data points in databases.
  3. Informatics expert: an expert in clinical knowledge management. This individual would be responsible for communication between domains.
  4. Database engineer: a database expert. This individual would be responsible for developing and maintaining database tables for CPM relationship information.
  5. Data analyst: a database expert. This individual would be responsible for developing data pipelines for analysis.
  6. Business intelligence developer: an expert in report development.

Problem Statement

The problem we are addressing is that knowledge gaps often exist while translating logics CPMs from domain experts into analytics pipelines. For example, domain experts may define the diagnosis of pneumonia at a high level in the original CPMs. However, the disease should be clearly defined to extract real data from the clinical database, including diagnosis codes in standard terminologies, types of patient visits, clinical context, problem status, and exceptions. Gaps may exist while clarifying such information by mapping incorrect data points, miscommunication, misinterpretation, changes in CPM contents, and changes in data sources. Such inconsistency could result in confusion and distrust of data quality for analytics consumers. In the current analytics environment there is no tool to track and investigate what the gaps are and where they exist.

To address inconsistencies in knowledge translation and improve analytics productivity, we aim to adopt a knowledge base that will help us manage the content across the whole CPM evaluation process. A knowledge base is a component in our EHR system that manages the authoring, review, publication, delivery, and versioning processes that surround clinical knowledge assets for consumption within these frameworks. Typically, clinical knowledge sources may be order sets, decision support rules, nursing protocols, clinical guidelines, and patient education resources. Any knowledge-based content can be a source for a knowledge base, such as concept definitions, formulations, executable queries, and design concepts of visualizations. These assets can be consumed by a human (ie, in the case of narrative clinical care guidelines) or by a computer (ie, rule bases for consumption by inference engines), or they can be aligned for consumption on both ends of the spectrum [21].

There are practical benefits to using a knowledge base for CPM analytics: (1) storing and sharing all the knowledge resources in a centralized place, instead of local storage or exchanging content by emails, to reduce miscommunication and redundancy; (2) being able to manage knowledge resources with unique numeric identifiers and metadata, including author, version, and history of revisions; and (3) making knowledge resources consumable with data analytics tools. In addition, a knowledge base may be useful for end-user data consumers by giving them more contextual information about the data with answers to questions such as “Who defined the definitions of the measures?”, “How were the measures calculated?”, and “What changes have been made?”

Selection of Care Process Models and Deriving Key Measures

As part of a pilot study to evaluate adherence based on the proposed framework, we selected 21 CPMs. Our selection criteria were as follows: (1) clinical utility and popularity of the CPM, (2) whether the CPM was already implemented within our EHR system, (3) the ease with which the CPM allowed the definition of evaluation metrics, and (4) whether data points related to the CPM were fully or partially collected in our clinical information systems. For the selected CPMs, we worked with CPM publishers in clinical programs to derive the three types of information for adherence evaluation defined in a previous section. The adherence rate (%), defined in the equation below, represents physician utilization of designated CPM components for a group of patients who are intended to be treated by the CPM:

Adherence (%) = (number of cases treated using the CPM) / (number of cases intended to be treated by the CPM)

Since the definition above is highly abstracted, there were diverse details regarding how to practically calculate the adherence. For example, to calculate the numerators, we determined what types of decision-support components were used to implement the CPMs (eg, order set) and which were used to connect key concepts to real data points (eg, order set ordering history). We found that 10 of the 21 CPMs (48%) were implemented as physician order sets, or small groups of order sets, whereas the rest were implemented through combinations of care pathways and decision-support rules.

Table 1. Summary of care process model (CPM) adherence definitions.
Clinical program (adherence definitions, n; queries, n)CPM nameCohort-defining conditionDecision-support component
Oncology, 1, 3Lung, breast, or colon cancerDiagnosis and chemotherapyOrder set
Pediatrics, 1, 1Febrile infantVital sign or a reportCare pathway
Neuroscience, 1, 1Acute strokeProblem and admission timeOrder set
Primary care, 4, 4DiabetesAge and diagnosisCare pathway

HypertensionDiagnosisCare pathway

Acute sinusitisDiagnosis and medicationCare pathway

High blood pressureDiagnosisCare pathway
Cardiovascular, 1, 3Heart failureAge, admission, and diagnosisOrder set

Acute myocardial infarctionAge, admission, and diagnosisOrder set

Coronary artery bypass graftAge, admission, and diagnosisOrder set
Musculoskeletal, 1, 1Total hip or knee surgeryProcedureOrder set
Surgical service, 1, 2AppendicitisProcedureOrder set

CholecystectomyProcedureOrder set
Behavioral health, 3, 3DepressionCDSa rule and diagnosisCare pathway

Suicide preventionCDS ruleCare pathway

Mental health integrationClinical document and clinic visitCare pathway
Women and newborn, 1, 2JaundiceNewbornCare pathway

Neonatal hypoglycemiaObservation, rules, and ageCare pathway
Intensive medicine, 4, 4PneumoniaCDS ruleCare pathway

Pulmonary embolismOrder, imaging, diagnosis, and care pathway useCare pathway

Pediatric sepsis in emergencyCDS ruleCare pathway

Pediatric minor head traumaCDS ruleCare pathway

aCDS: clinical decision support.

The denominators were used to identify CPM-specific patient cohorts. For example, the denominator for the adherence rate of the hypoglycemia CPM was defined as a combination of (1) whether certain decision-support rules related to hypoglycemia were fired, (2) whether a patient has a history of hypoglycemia according to the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10), and (3) whether specific nursing documents were recorded.

Below are examples of detailed key concepts to be included in the definitions:

  1. Cohort (denominator): clinical inclusion and exclusion criteria, patient master index, encounter, facility, care unit, decision-support rule firing criteria, clinical form used, etc.
  2. Utilization (numerator): Order set usage, order set title, version, content, orderable items, customized order sets, care pathway title, components in care pathway used, etc.
  3. Adherence: adherence definition, aggregation level by patient, encounter, provider, unit and facility, etc.

A group of experts with different backgrounds reviewed these concepts, documented the definitions, and built queries. Table 1 shows a summary of the derived CPM adherence definitions.

Development of Framework

We developed the analytics framework by integrating several homegrown and commercial tools. For the knowledge base, we used Intermountain’s knowledge repository that our organization has used over the last 10 years. We adopted Alteryx as a tool for data analytics pipelines and Tableau as a tool for visual transformation and Web-based dashboard development. We used our enterprise data warehouse (EDW) as the data source, which is an integrated clinical data repository for research and quality improvement that stores over 100 billion records tied to encounters, lab observations, diagnoses, procedures, medications, and billing over 20 years. We also used CPM-centered database tables in our EDW that were developed by clinical programs, which store condition-specific patient cohorts, quality metrics, and outcomes for clinical research and quality improvement.

Figure 1 depicts the architecture and flow of knowledge in the framework. Domain experts in clinical programs translate paper-based CPMs to adherence definition documents. Structured Query Language (SQL) developers create queries based on the documents and deploy them into the knowledge base on the Web (see Figure 2). Alteryx imports the SQL information from the knowledge base to run against the EDW; Alteryx then exports the cohort and metrics as a Tableau-specific data file to a Tableau server so they can be visually transformed by Tableau to automatically generate charts on the Web. In the dashboard, links are embedded in the charts, enabling users to view the original CPM documents and queries used in the knowledge base.

Figure 3 shows a CPM in various forms during knowledge transformation: human-readable PDF document (top left); logics of the CPM are implemented as an order set in a computerized physician order entry (CPOE; bottom left); adherence definition document (top right); and computerized adherence logic as a SQL (bottom right). Since these knowledge content components in different forms originated from one source, they are semantically linked to each other with authorship, translation record, and version history.

Figure 1. Architecture diagram of the knowledge-based analytics framework for care process model (CPM) adherence. EDW: enterprise data warehouse; SQL: Structured Query Language.
View this figure
Figure 2. Screenshot of the care process model (CPM) knowledge content components uploaded into the online knowledge repository.
View this figure
Figure 3. Evolution of care process model (CPM) knowledge. Top left: an original PDF document; bottom left: a surgical order set inside a computerized physician order entry (CPOE) that implemented logics of the CPM; top right: a narrative text of evaluation criteria; bottom right: Structured Query Language (SQL) to extract adherence metrics. The red boxes and arrows represent the transformation of concepts and logics from one form to another. Source: Cerner PowerChart. Used with permission by Cerner Corporation.
View this figure

Development of Dashboard

We developed a Tableau-based dashboard in a production environment that is accessible to clinicians and researchers in our organization through secured user access. It employs five detailed views for each representative CPM adherence in different contexts and scales.

  1. Main view: this view provides an overview of adherence trends at the highest level. It consists of a bar chart representing the average cumulative rates of CPM utilization and a line chart showing the monthly average over time (see Figure 4). Each CPM is marked with different color. A filter to the right of the bar graph narrows down the data by facility. Average adherence rates range from 0% to 100%, with 0% indicating that no physician used any CPM for the specific condition-based cohort and 100% indicating that all physicians used CPMs for all associated relevant cases.
  2. Facility view: this view is used to monitor the monthly average adherence rate by hospitals. This view consists of a dual chart that represents the percent adherence rate by month and the number of encounters where CPM was used or not used.
  3. Provider view: this view shows the summary of CPM utilization by individual providers. Users can select a provider in the filter on the right, then the view narrows down by patients treated by the selected provider (see Figure 5).
  4. Encounter view: this view provides detailed actions in CPM utilization at the encounter level. A table view includes patient identifier, provider, length of stay, CPM used, and enrolled date and time.
  5. Patient view: this view provides a summary of CPM-based treatments for a patient.

Data Extraction and Early Usage Pattern

Data pipelines were scheduled to run regularly, with different refresh frequencies depending on CPMs and data sources. As of July 1, 2018, the number of patients eligible for 21 CPMs was 230,669 and the number of encounters was 377,507. For those target patients, 7895 providers utilized CPMs at least once. Total adherence rate across all the CPMs during the period was 8.8%.

Figure 4. Main view of the Tableau–based care process model (CPM) utilization dashboard (screenshot).
View this figure
Figure 5. Provider view of the care process model (CPM) utilization dashboard (screenshot).
View this figure
Figure 6. Dashboard usage pattern showing number of daily sessions and number of distinct users. The blue bar represents the number of users that accessed the dashboard and the orange line represents the number of distinct users.
View this figure
Table 2. Care process model (CPM) adherence revision history since dashboard implementation.
Clinical programNumber of revisionsType of revision
Oncology2Amendment of adherence definition; new data source
Pediatrics3Amendment of adherence definition; new data source
Neuroscience2Amendment of adherence definition
Primary care2Amendment of adherence definition
Cardiovascular2Amendment of adherence definition
Musculoskeletal1Amendment of adherence definition
Surgical service1Amendment of adherence definition
Behavioral health4Amendment of adherence definition; new data source; new facility
Women and newborn2Amendment of adherence definition; data-quality improvement
Intensive medicine4Amendment of adherence definition; new CPM component

We analyzed user sessions of the dashboard using a Tableau system monitoring report, as represented in Figure 6. Test and administration users were excluded from the analysis. Overall, the usage pattern was stable with minor seasonal effects and spikes.

Data Provenance: Tracking Revisions of Care Process Model Adherence Definitions

Since implementation, we set up monthly meetings involving clinical programs, CPM implementation leadership, data analysts, and knowledge engineers. These meetings were meant to (1) monitor CPM adherence data, (2) review current adherence definitions and discuss room for improvement, and (3) discuss ways to encourage providers to use CPMs. During the pilot study period, a number of revisions to the definitions were made (see Table 2). Many of the revisions included amendments to the definitions, while some included the addition of new data sources to the EDW or resolving data-quality issues. Definitions were updated with revisions in the knowledge base and the links to the Alteryx pipelines were automatically renewed.

As seen in the bottom portion of Figure 4, the adherence rate steadily increased since implementation. We believe there are two practical reasons for this increase. One is that clinical programs have encouraged their clinicians to utilize CPMs, through CPOE training, physician education, etc. The other reason is that some clinical programs revised their adherence definitions. For example, the oncology program expanded CPM-designed order sets, which resulted in an increase of the numerator. The intensive medicine program revised the definitions of their target patients by narrowing them down to specific facilities, which resulted in a decrease of the denominator.

Comparison With Prior Work

Several studies have used knowledge engineering tools for translation of CPG logics to implement them into practice for the purpose of knowledge standardization and computational automation [4-9]. Unlike in those studies, we used knowledge engineering tools for managing knowledge transformation within analytics processes. Compared with prior work, our original results included (1) validating the usefulness of knowledge management tools within analytics processes, (2) validating the feasibility of integrating knowledge management tools and an analytics framework, and (3) demonstrating the proposed approach using empirical clinical data from local EHR systems.


Although the main contribution of this study is the use of knowledge management, we did not quantitatively analyze the improvement in the consistency of knowledge in transformation or the productivity of analytics pipelines. Rather, we demonstrated it qualitatively, including determining which functionalities of the knowledge base were able to support the consistency of CPM-related knowledge in the development and maintenance phases. We will conduct further analyses in the future as we collect additional data. This will include adding more CPMs with complex clinical settings and practices, including chronic conditions and comorbidities that span multiple encounters, locations, and providers.

In this study, we simplified the adherence definitions to include whether a designated CPM component is used or not used for a target patient, although there may be many variations. We will continue to add more detailed definitions of adherence, including how specific order items or content within CPM components are used. By doing so, we can investigate the mechanism of CPM utilization (ie, use of standard clinical guidelines) and how this can change patient care or clinical workflow.


This case study demonstrated that the proposed analytics framework could accommodate complicated knowledge management and data pipelining for CPM evaluation using a knowledge base, while maintaining computational efficiency. It is expected that the benefits of using a knowledge base will be more significant as we add complicated clinical guidelines into the analytics framework.

Authors' Contributions

JL developed the framework, built the dashboard, performed data analysis, and designed the study. NCH helped design the study, provided knowledge base consultation, collaborated with clinicians, and validated the feasibility of the architecture and the results of data analysis. The authors do not have any funding or financial support associated with this project.

Conflicts of Interest

None declared.

  1. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: Potential benefits, limitations, and harms of clinical guidelines. BMJ 1999 Feb 20;318(7182):527-530 [FREE Full text] [Medline]
  2. Institute of Medicine. In: Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011.
  3. de Clercq PA, Hasman A, Blom JA, Korsten HH. Design and implementation of a framework to support the development of clinical guidelines. Int J Med Inform 2001 Dec;64(2-3):285-318. [Medline]
  4. Rosenfeld RM, Shiffman RN. Clinical practice guideline development manual: A quality-driven approach for translating evidence into action. Otolaryngol Head Neck Surg 2009 Jun;140(6 Suppl 1):S1-S43 [FREE Full text] [CrossRef] [Medline]
  5. Ennis J, Gillen D, Rubenstein A, Worcester E, Brecher ME, Asplin J, et al. Clinical decision support improves physician guideline adherence for laboratory monitoring of chronic kidney disease: A matched cohort study. BMC Nephrol 2015 Oct 15;16:163 [FREE Full text] [CrossRef] [Medline]
  6. Goldstein MK, Hoffman BB, Coleman RW, Musen MA, Tu SW, Advani A, et al. Implementing clinical practice guidelines while taking account of changing evidence: ATHENA DSS, an easily modifiable decision-support system for managing hypertension in primary care. Proc AMIA Symp 2000:300-304 [FREE Full text] [Medline]
  7. Tso GJ, Tu SW, Oshiro C, Martins S, Ashcraft M, Yuen KW, et al. Automating guidelines for clinical decision support: Knowledge engineering and implementation. AMIA Annu Symp Proc 2016;2016:1189-1198 [FREE Full text] [Medline]
  8. Abidi S. A knowledge-modeling approach to integrate multiple clinical practice guidelines to provide evidence-based clinical decision support for managing comorbid conditions. J Med Syst 2017 Oct 26;41(12):193. [CrossRef] [Medline]
  9. Abidi SR, Abidi SSR, Hussain S, Shepherd M. Ontology-based modeling of clinical practice guidelines: A clinical decision support system for breast cancer follow-up interventions at primary care settings. Stud Health Technol Inform 2007;129(Pt 2):845-849. [Medline]
  10. Worrall G, Chaulk P, Freake D. The effects of clinical practice guidelines on patient outcomes in primary care: A systematic review. CMAJ 1997 Jun 15;156(12):1705-1712 [FREE Full text] [Medline]
  11. Marshall DA, Simpson KN, Norton EC, Biddle AK, Youle M. Measuring the effect of clinical guidelines on patient outcomes. Int J Technol Assess Health Care 2000;16(4):1013-1023. [Medline]
  12. Halm E, Atlas S, Borowsky L, Benzer T, Metlay J, Chang Y, et al. Understanding physician adherence with a pneumonia practice guideline: Effects of patient, system, and physician factors. Arch Intern Med 2000 Jan 10;160(1):98-104. [Medline]
  13. Midboe AM, Lewis ET, Paik MC, Gallagher RM, Rosenberg JM, Goodman F, et al. Measurement of adherence to clinical practice guidelines for opioid therapy for chronic pain. Transl Behav Med 2012 Mar;2(1):57-64 [FREE Full text] [CrossRef] [Medline]
  14. Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: Implications for pay for performance. JAMA 2005 Aug 10;294(6):716-724. [CrossRef] [Medline]
  15. Frankel HL, FitzPatrick MK, Gaskell S, Hoff WS, Rotondo MF, Schwab CW. Strategies to improve compliance with evidence-based clinical management guidelines. J Am Coll Surg 1999 Dec;189(6):533-538. [Medline]
  16. James PA, Cowan TM, Graham RP, Majeroni BA, Fox CH, Jaén CR. Using a clinical practice guideline to measure physician practice: Translating a guideline for the management of heart failure. J Am Board Fam Pract 1997;10(3):206-212. [Medline]
  17. Dykes PC. Practice guidelines and measurement: State-of-the-science. Nurs Outlook 2003;51(2):65-69 [FREE Full text] [Medline]
  18. Bodesinsky P, Federico P, Miksch S. Visual analysis of compliance with clinical guidelines. In: Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies. New York, NY: ACM; 2013 Presented at: The 13th International Conference on Knowledge Management and Knowledge Technologies; September 4-6, 2013; Graz, Austria. [CrossRef]
  19. Anderson D, Zlateva I, Khatri K, Ciaburri N. Using health information technology to improve adherence to opioid prescribing guidelines in primary care. Clin J Pain 2015 Jun;31(6):573-579 [FREE Full text] [CrossRef] [Medline]
  20. Trafton JA, Martins SB, Michel MC, Wang D, Tu SW, Clark DJ, et al. Designing an automated clinical decision support system to match clinical practice guidelines for opioid therapy for chronic pain. Implement Sci 2010 Apr 12;5:26 [FREE Full text] [CrossRef] [Medline]
  21. Hulse NC, Galland J, Borsato EP. Evolution in clinical knowledge management strategy at Intermountain Healthcare. AMIA Annu Symp Proc 2012;2012:390-399 [FREE Full text] [Medline]
  22. Sittig DF, Wright A, Meltzer S, Simonaitis L, Evans RS, Nichol WP, et al. Comparison of clinical knowledge management capabilities of commercially available and leading internally developed electronic health records. BMC Med Inform Decis Mak 2011 Feb 17;11:13 [FREE Full text] [CrossRef] [Medline]
  23. James BC, Savitz LA. How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff (Millwood) 2011 Jun;30(6):1185-1191 [FREE Full text] [CrossRef] [Medline]

CDS: clinical decision support
CPG: clinical practice guideline
CPM: care process model
CPOE: computerized physician order entry
EDW: enterprise data warehouse
EHR: electronic health record
ICD-10: 10th revision of the International Statistical Classification of Diseases and Related Health Problems
SQL: Structured Query Language

Edited by H Wu; submitted 22.07.18; peer-reviewed by S Sun, L Huang; comments to author 27.09.18; revised version received 20.11.18; accepted 11.12.18; published 27.02.19


©Jaehoon Lee, Nathan C Hulse. Originally published in JMIR Biomedical Engineering (, 27.02.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Biomedical Engineering, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.