For many family physicians, the idea of a chart audit conjures up images of federal investigators or insurance company representatives descending on their offices to look for evidence of wrongdoing. For the most part, however, a chart audit is not so scary. A chart audit is simply a tool physicians can use to check their own performance, determine how they're doing and identify areas where they might improve. The purpose of this article is to describe some scenarios in which a chart audit might be helpful and to offer step-by-step instructions for doing one.
Chart audits can serve many purposes, from compliance to research to administrative to clinical. You can conduct a chart audit on virtually any aspect of care that is ordinarily documented in the medical record. Practices frustrated with clinical processes that don't work well can use chart audits to document that something is wrong, find the defect in the process and fix it.
Perhaps the most beneficial use for a chart audit is to measure quality of care so that you can improve it. Chart audits are often used as part of a quality improvement initiative. For example, a practice might review charts to see how often a particular vaccine is offered, given or declined. If the audit determines that the vaccine is not being offered or given as recommended, then there is room for improvement. The same practice could review the panels of individual physicians within the group to see if they differ in performance on this measure and to give focus to their improvement efforts (for additional chart audit ideas, see below).
| Preventive care |
| Percentage of women ages 21–64 who have had a Pap smear within the past three years |
| Percentage of adults ages 51–80 who have had colon cancer screening |
| Percentage of children age 2 who have completed all recommended immunizations |
| Percentage of elderly adults with documented fall risk assessment within the past year |
| Chronic disease management |
| Percentage of patients with hypertension whose last blood pressure reading was < 140/90 |
| Percentage of patients with diabetes with an A1C level recorded in the last year |
| Percentage of patients with diabetes whose A1C is < 7.0 |
| Percentage of patients with diabetes with a documented eye exam within the last year |
| Percentage of patients with persistent asthma who are on an anti-inflammatory agent |
A chart audit is one of numerous data sources available for quality improvement efforts. Others include patient surveys, discharge summary reviews, billing/claims data and employee feedback.
Below we describe eight steps to a formal chart audit. Although the process is not necessarily linear, we will discuss each step in the order it might typically occur, using the example of a breast cancer screening audit to illustrate each step. Because the audit will involve reviewing confidential data, it is important to check your institutional guidelines regarding patient confidentiality before you get too far into the planning process.
Step 1: Select a topic. The focus of your audit must be clear, neither too narrow nor too broad, and measurable using data available in the medical record. If possible, choose an area that interests you. You will find that you are more able to recognize nuances in your study when you have personal interest in the topic. Of course, your topic should also be of interest to the practice, perhaps a problem or aspect of care that the providers have identified as needing improvement. The Joint Commission recommends studying issues that are high frequency, high risk or both.
You should also consider early in the process how important external comparison is to your purpose. If it is quite important, then choose a topic that has an existing, well-defined measure and available benchmark data – even one you might not choose otherwise – because this will be more practical than developing your own standard for comparison.
Chart auditing is an iterative process – don't be discouraged if you change directions several times before settling on a topic.
Example: Your practice wants to measure how well it's doing on meeting recommendations for preventive care. Since the insurance carriers in the area are focusing heavily on women's health, the group decides to focus its chart review on screening for breast cancer (mammography).
Step 2: Identify measures. Once you're set on a topic, you need to define exactly what you will measure. Criteria must be outlined precisely, with specific guidelines as to what should be counted as a “yes” (criteria met) and what should be counted as a “no” (not met).
For example, if you decided to review the rate at which foot exams were performed on patients with diabetes in the last year, you would need to decide what qualifies as an adequate foot exam. Is it monofilament testing for sensation? Visual inspection? Palpation of pulses? Many would say all three are necessary for a complete foot exam. If only two of the three are documented, how will you count that?
It may be worthwhile to do a literature review to help you define your measures or consult measures used by insurers or accrediting bodies; adopting measures that have been used successfully in the past will make your work easier. A literature review may also help you identify benchmarks for comparison.
Once you've chosen measures that seem workable, it can be helpful to conduct a pilot audit. Just going through a few charts will help to identify issues that need to be clarified before starting a full audit.
Example: For your audit on breast cancer screening, the group considers several measures, including the following:
After considerable discussion, the group decides to measure whether a mammogram was completed or recommended within the last 24 months.
Step 3: Identify the patient population. To determine which records to review, you need to define the population you want to assess. Characteristics to consider may include age, gender, disease status and treatment status. In many cases, the focus of the audit and even the measure itself will help to define the population. You'll also need to develop specific inclusion or exclusion criteria.
Example: In keeping with the HEDIS breast cancer screening measure that your group decided to follow, your patient population will be women age 40 to 69. Because you'll be looking for evidence of a mammogram in the past 24 months, the lower age limit for the sample will be 42. Only those patients with at least three visits in the last two years and one in the last 13 months will be included. You decide to exclude women who have had bilateral mastectomies or are terminally ill.
Step 4: Determine sample size. A manual audit of all charts meeting your inclusion criteria will not be feasible in most situations. That's where sampling comes in. For an informal, or “quick and dirty,” audit designed to give you a sense of whether a more sophisticated audit is warranted, you may find it useful to sample a minimum of 20 charts. For better results, a common rule of thumb is to try for 10 percent of the eligible charts. Or you may choose to use a convenience sample: the patients from a single day or all the charts on a single shelf in the records room.
If you want to track a measure over time, or if you want your results to be statistically valid, your sample size is critical. If the sample is too small, the random variability will be too large, and the results will be limited in their applicability.
Example: Using the process outlined below, your group determines that its sample should total 81 charts.
Calculating a statistically valid sample size for a chart review follows steps adapted from statistical techniques used for descriptive studies. The process uses a nomogram, or table, to identify the desired number:
1. Estimate the expected proportion within the population that will have the measure of interest.
If you have a benchmark from literature or prior studies, use it. Otherwise, consult with colleagues or experts in the field to determine an estimate. The tables generally require this proportion to be 50 percent or less. If more than 50 percent of the population is expected to have the characteristic, then base your sample size calculation on the proportion without the characteristic.
2. Specify the width of the confidence interval you wish to use.
All empirical estimates based on a sample have a certain degree of uncertainty associated with them. It is necessary, therefore, to specify the desired width of the confidence interval (W). This gives a range of values that you can be confident contains the true value. In most cases, an appropriate width is 0.20 (that is, plus or minus 10 percent).
3. Set the confidence level.
This is a measure of the precision or level of uncertainty. Typically 95 percent is used, meaning that we are 95 percent certain that the interval includes the true value. This is arbitrary, however, and other levels of confidence can be used. The table shown below is for a 95-percent confidence level. The narrower the width of the confidence interval and the higher the confidence level, the larger the sample size.
4. Use the nomogram (below) to estimate sample size.
| Width of the confidence interval (W) | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 |
|---|---|---|---|---|---|
| Expected proportion (P) | |||||
| 0.10 | 138 | 61 | |||
| 0.15 | 196 | 87 | 49 | 31 | |
| 0.20 | 246 | 109 | 61 | 39 | 27 |
| 0.25 | 288 | 128 | 72 | 46 | 32 |
| 0.30 | 323 | 143 | 81 | 52 | 36 |
| 0.40 | 369 | 164 | 92 | 59 | 41 |
| 0.50 | 384 | 171 | 96 | 61 | 43 |
According to HEDIS 2007 Audit Means, Percentiles and Ratios, the NCQA's annual report of health plan performance data, 68.9 percent of women age 40 to 69 had a mammogram during 2006. This makes the expected proportion of those without screening 31.1 percent. We choose a width of the confidence interval of 0.20 (plus or minus 10 percent) and a confidence level of 95 percent. This means that we want to be 95 percent confident that the result falls between 58.9 percent and 78.9 percent. Using the nomogram to determine the sample size, we read down the left column of figures for the expected proportion without the characteristic (0.30 is the closest value to 31.1 percent) and then across to the chosen width of the confidence interval (0.20). When we follow the column down, we find the required sample size (81). If the number required is too large to be completed, we can recalculate with a lower confidence level or wider interval; this will produce a smaller sample size.
Step 5: Create audit tools. To complete your chart audit, you will need instruments on which to record your findings. How they are structured and the details they include will affect the analysis you can do and the eventual usability of your findings. Data should be collected in a format that keeps all individual records separate but allows for easy compiling.
Many chart audits involve the calculation of a rate, percentage, mean or other statistical measurement. An electronic spreadsheet format can be customized to do these calculations for you. For those more comfortable with paper-based systems, a preprinted form that lists the specific items to check in each chart serves well as an audit tool. One form is completed for each chart, and the forms can then be sorted and counted as desired. A separate form can be used to tabulate results.
Creating clear, simple audit tools will make it possible for nonclinical staff to perform many audits effectively. Once you've developed the forms, if someone other than you will be doing the actual chart reviews, go over a few examples together to be sure the reviewer understands the criteria exactly as you intend.
Example: Your group decides to use paper forms for the chart audit (see the completed forms below).
| Patient identification | Inclusion criteria | Exclusion criteria | Mammogram in past 24 months | No mammogram in past 24 months | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Patient name | MRN | Age 42–69 as of 12/31/07 | 3 visits in past 3 years | 1 visit in past 13 months | Bilateral mastectomy | Left practice, terminally ill, expired | Locally | Elsewhere | No discussion documented | Discussed patient declined | Mammogram order not completed |
| Jane D | A2345 | 53 | yes | yes | no | no | yes | no | |||
| Sue S | B2345 | 62 | yes | yes | no | no | yes | no | |||
| Ann J | C2345 | 59 | yes | yes | no | no | no | no | yes | no | no |
| Betty M | D2345 | 65 | yes | yes | yes | no | |||||
| Julie J | E2345 | 57 | yes | yes | no | yes | |||||
| Bonnie B | F2345 | 52 | no | ||||||||
| Alice G | G2345 | 55 | yes | yes | no | no | yes | no | |||
| Kate H | H2345 | 61 | yes | no | |||||||
| Dana T | I2345 | 63 | yes | yes | right side only | no | no | yes | |||
| Doris B | J2345 | 40 | |||||||||
| Helen P | K2345 | 64 | yes | yes | no | no | yes | no | |||
| Evelyn C | L2345 | 51 | yes | yes | no | no | yes | no | |||
| Paula T | M2345 | 49 | yes | yes | no | no | yes | no | |||
| Mary S | N2345 | 69 | yes | yes | no | no | yes | no | |||
Step 6: Collect data. Select the date or dates on which you will collect data. Be sure to coordinate the specifics (date, time and number of charts to be pulled) with the medical records staff. Review each chart to determine if the patient meets the selection criteria. The reviewer should complete one audit tool (paper form or row in the electronic spreadsheet) for each patient that meets the criteria. To protect patient confidentiality, patient names should not be included on the review forms.
Example: You instruct your office staff to pull the charts of roughly 100 adult female patients. Once you've identified 81 that meet the selection criteria, your nursing supervisor fills out the audit tool for each one, reserving questionable cases for physician review.
Step 7: Summarize results. Summarizing the data is a little more complex than just counting up all the data sheets. You must consider how the data will be used and make sure the information is presented in a way that will make it meaningful. Inconsistencies here can produce data that can't be interpreted.
Example: Your breast cancer screening audit results show that 57 percent of your sample received mammograms (see the results table below).
| # | % | |
|---|---|---|
| Total charts reviewed | 100 | |
| Patients included in audit | 81 | |
| Patients who received mammogram | 46 | 57 |
| Received mammogram locally | 25 | 31 |
| Received mammogram elsewhere | 10 | 12 |
| Patients with no documentation of completed mammogram | 35 | 43 |
| Documented declined mammography | 6 | 17 |
| Documented mammogram ordered, not completed | 4 | 13 |
| No documentation of discussion of mammography | 25 | 71 |
Step 8: Analyze and apply results. Once you have compiled your data and calculated the results, you can compare them to local or national benchmarks. There may be multiple benchmarks, depending on your topic and the performance measure you calculated. You should take into account the differences between your population and those you're comparing it with, as appropriate. If the measure is truly important to the group, you may wish to set a performance goal based on what the group feels is appropriate and reasonable and make it the focus of a quality improvement initiative.
Example: At 57 percent, your group's breast screening rate is less than the national benchmark of 68.9 percent. This benchmark is the mean for commercial HMO patients, according to the HEDIS 2007 Audit Means, Percentiles and Ratios, the NCQA's annual report of health plan performance data (view it at http://www.ncqa.org/tabid/334/default.aspx). Of the 35 patient charts that had no documentation of a mammogram, only 10 records showed that the physician had discussed the need for a mammogram with the patient. The challenge is now to drill down to figure out whether the issue was discussed but not documented in those other charts or whether it was simply overlooked. Telephone contact with the 25 identified patients might help you begin to clarify this so that an appropriate intervention can be designed.
Chart audits can be useful tools in improvement and safety efforts. It is essential to define precisely what you want to measure and the criteria by which you will measure it. (If you're floundering, you probably haven't defined this well enough.) Sample sizes can be chosen informally or determined in a statistically valid fashion. Summarize your data in a way that makes sense for the problem you're addressing. Make sure to act on problems you find, and remeasure later to see that your changes made a difference. You and your patients will be glad you did.
