Patient satisfaction has gained greater attention in recent years, partly because of the emphasis on patient-centered care but also because of the increasing number of family physicians leaving small group practices and entering large organizations, many of which systematically use patient surveys to measure patient satisfaction with multiple aspects of their health care. The results are used to calculate scores that enable comparisons among physicians and other providers and with benchmarks.

As the impact of patient satisfaction scores has increased in recent years, so has the importance of correctly calculating and applying them. When scores are posted on the Internet, physicians' reputations are affected. Some clinics also use the scores to help identify physicians whose scores suggest they might need additional training or some other form of administrative intervention. For example, at a meeting last year, clinicians were told that those with patient satisfaction scores in the bottom 25 percent would meet with a human service representative to work on improving their deficiencies. Patient satisfaction scores are a factor in compensation formulas as well.

Valid patient satisfaction scores, appropriately used, may help to improve quality of care and service, but they can have unintended consequences as well, such as diminished physician satisfaction. Seventy-eight percent of clinicians in one study said patient satisfaction scores moderately or severely affected their job satisfaction negatively, and 28 percent said the scores made them consider quitting.¹

Even those outside the medical field have begun scrutinizing the downside of patient satisfaction scores.^2,3 The purpose of this article is to identify potential problems with patient satisfaction scores and suggest some ways of making them more useful.

Samples are not simple

Understanding the potential problems with patient satisfaction scores begins with understanding how the data are gathered, how the scores are calculated, and how this can affect comparability.

Patient satisfaction scores are usually derived from surveys, which are constructed and administered in different ways. How and when questions are asked can affect results. For example, phone surveys tend to glean different results than written surveys. A survey completed at the end of a patient visit provides different information than a survey sent out and completed a few weeks after the visit.

Many surveys ask a series of questions about one area. For instance, a typical patient satisfaction survey may ask 13 questions on clinician interaction and provide five possible responses for each question. Clinicians generally do well, and variation is limited. In Minnesota and parts of some neighboring states, 79 percent of patients gave their doctors a rating of 9 or 10 on a 10-point scale.⁴ This clustering of excellence means that a few patients who are dissatisfied can have a profound impact when an individual score is compared to the aggregate. Sampling methods can affect the value of satisfaction scores and the validity of comparisons. If different numbers of patients are polled for each physician, the resulting scores are misleading. Differences in sample size may not be readily apparent because patient satisfaction is typically presented as a percentage.

A similar problem exists when comparing scores derived from surveys with different numbers of respondents. Physicians with fewer respondents tend to receive the highest or lowest scores, reflecting the higher influence of ratings of excellent and poor performance in a small data set. Physicians with more respondents – and a resulting greater variation of patient experience – are more likely to have average scores. This is a function of statistics, not quality.

Sample size also affects accuracy.⁵ A survey may say it has a confidence level of 95 percent with a margin of error of 5 percent. It seems reasonable that if we are going to judge someone's competency, we should be at least 95 percent confident. Being wrong one in 20 times may still disturb the person being incorrectly judged, however. Margin of error is generally underestimated and increases as scores are based on smaller and smaller groups of patients. Sometimes a sample size is so small that the margin of error actually exceeds the range of scores for the individuals. To meaningfully increase the confidence level of a survey requires a large increase in sample size, which is expensive and is typically not done for the purpose of calculating an individual's score.

Sample size and survey methods must be consistent in order to make accurate comparisons. Sending out a number of surveys and then aggressively following up until you have received a certain percentage will get one type of result. Sending out a large number of surveys and hoping that enough are returned to meet the certain percentage will get another type. In the second case, the sample is not random; rather, self-selected participants have chosen to fill out the survey.

The more variation, the less reliable the score

Patient satisfaction scores also reflect variation in several additional factors unrelated to physician performance that further complicate evaluating and comparing physicians' scores:

People. Studies have identified patient characteristics that are associated with higher scores. Patients who are older, have female providers, achieve positive outcomes, are sicker, or generate higher health care costs give their physicians higher ratings.⁶ Other factors can also confound comparisons. For example, physicians establishing new practices face patient expectations and demands much different than those of their established colleagues. They may be more likely to experience difficult patient interactions or encounter drug-seeking patients than physicians who have developed relationships with their patients over time.

Physicians also interact with patients in myriad ways every day, some of which may prevent some patients and their experiences from being included in satisfaction surveys. For example, patients who communicate with their physician by phone or email rather than being seen for an office visit would be excluded from surveys distributed on paper in the office.

Plus, surveys may not actually measure the true quality of patient interactions. Physicians may deliver less than optimal quality of care and still have excellent patient satisfaction scores, and the converse is also true. In one study, when patients were interviewed after completing a written survey, researchers sometimes found significant disparities between their ratings and the care they received.⁷

“Machinery.” Physicians work within systems that can affect patient satisfaction in ways that are beyond their control. For example, patients may face access problems, long cycle times, or difficulties in making an appointment. System requirements may contribute to overworked teams, rushed visits, electronic health record complexities, and excessive documentation requirements that negatively affect patient interactions.

Measurement. Measuring results within any process will often generate random variation that our brains are wired to explain. Often, these explanations are incorrect. For example, studies involving charts of completely random scores or the results of tasks beyond the control of the subjects found that observers looked for patterns where there were none and even prescribed corrective action.^8,9 Another weakness of satisfaction scores is that they reflect a single point in time. If the same survey is repeated, odds are that the numerical results will not be.

Environment. Physicians work in clinics that can vary significantly in terms of patient population and mission – for instance, tertiary care centers vs. primary care centers. It is not prudent to compare these physicians' patient satisfaction scores, but it is often done.

Taking into account all of these sources of variation, it is important to recognize that a satisfaction score is a perspective, not a truth, about a physician's ability to deliver quality care. It is information, it reflects a subset of daily interactions, and it is dependent on the number of respondents who choose to respond to the survey.¹⁰

Is the improvement real?

Measuring patient satisfaction isn't cheap. Sending out surveys and tabulating responses is expensive, and the cost only increases as organizations seek more accurate results. Some organizations also pay outside firms to independently verify results and to provide strategies for moving the organization's satisfaction scores into the top 10 percent so they can gain a competitive advantage. But does measuring patient satisfaction and acting on the results lead to better performance?

At a recent conference during a discussion about using the Ottawa Ankle Rules to prevent unnecessary imaging, a physician stated that patient satisfaction affected his compensation and observed that ordering radiographs improved patient satisfaction and, therefore, compensation. Financial incentives must be approached with caution, and so should intervention strategies. In a stable system, “regression toward the mean” dictates that the highest and lowest scores will gravitate toward the average in future samples. This means using patient satisfaction scores to rank physicians and putting the lowest performing 15 percent through even a sham program will likely lead to improvement in future surveys – but not necessarily improvement in actual performance.

The best approach to improving satisfaction scores is to devise an intervention, apply it across the organization, and show that the intervention improved the entire system rather than just the presumed poor performers. That approach would involve formal objectives and excellent planning in changing how the organization treats the patient. Improvement – not inspection – is the key to quality improvement of processes.¹¹

So what to do?

Physicians can work to mitigate potential problems with patient satisfaction scores and help their organizations generate the most benefit from an otherwise flawed process by asking the organization to do the following:

1. Aggregate data from all providers to get adequate samples so that meaningful changes can be made if necessary. Identify the processes that are causing issues, perform interventions, and see whether the interventions work (see “Tracking patient satisfaction initiatives”).

2. Do not present satisfaction scores without including statistical information on the margin of error and analysis of means (i.e., how significantly the scores for an individual or group differ from the overall average score).

3. Given the difficulties and expense in making sure satisfaction scoring methods are fair, do not reward or penalize physicians based solely on scores. Physicians should not practice medicine based on poor data, and their salaries should not be determined by it.

4. Examine the lowest few patient satisfaction scores that fall outside a random distribution. Odds are that physicians scoring that low are suffering from burnout or other special circumstances that need attention.

5. Study quality improvement and statistical variation (see “Quality improvement resources”). In most organizations, 85 percent of all outcomes are related to process, and only 15 percent are related to people.¹² Instead of trying to find better methods for scraping burnt toast, learn to fix the toaster.