For several years, Family Practice Management has published the results of electronic health record (EHR) user-satisfaction surveys that two of us (Adler and Edsall) have conducted and analyzed.^1–3 In reviewing survey results, we have repeatedly noticed something interesting. Simpler EHR products designed for small practices have consistently scored better than functionally more sophisticated products designed for practices of 21 doctors or more. In fact, at face value, an almost inverse linear relationship between practice size and user satisfaction has shown up in our surveys; as practice size increases, EHR satisfaction decreases. Nevertheless, many of these products designed for larger practices have won industry awards for their functionality.

We found ourselves wondering whether increased functionality and software complexity drive lower user satisfaction or whether something else is at work. Consequently, we set out to reanalyze data from the 2009 survey to see if we could identify additional factors associated with differences in user satisfaction.

Our approach

For the 2009 survey, as for previous versions, we collected our data by publishing the survey instrument in an issue of FPM, making an online version available through the FPM web site and allowing respondents to self-select.⁴ See the initial survey report for more about the methodology.³ Because of the wide availability of the survey instrument, we accepted responses only from AAFP members as a way of avoiding frivolous responses, multiple responses per individual and other such potential sources of bias. Given this approach, the results should not be considered a statistically accurate picture of EHR use among AAFP members but a more informal collection of responses from a couple thousand of them.

We were able to collect a total of 2,556 responses. Of those, 544 were excluded for various reasons. That left 2,012 responses for analysis. Respondents reported a total of 142 identifiable EHR systems, 22 of which were reported by 13 or more respondents each.

In preparing for the survey, we had hypothesized that the qualities of physician users and their practices may influence their satisfaction as well as the EHR systems used. We therefore included items in the survey to collect data on the following:

• How long respondents had used their EHR systems,

• How skilled they felt they were in using them,

• Whether they were involved in selecting them,

• How many physicians were in their practice,

• Whether they were in a single-specialty family medicine practice or a multispecialty practice,

• Whether their incomes were salary-based, production-based or somewhere in between.

We recruited a statistically skilled colleague (Shields) to help us with the analysis. We analyzed the 2,012 valid responses we received from our 2009 survey using SPSS Statistics, version 15. Because this is not a research journal, we will summarize our analysis rather than show the nitty-gritty details.

We did the analysis in three parts. First, we wanted to know if the above characteristics (variables) had statistically significant relationships with user satisfaction. We performed ANOVA and Scheffé tests on user satisfaction by practice size, by specialties in the practice, by salary type and by involvement in EHR selection. Next, we performed Pearson's Correlation Coefficient analysis on the length of EHR use and EHR user self-assessed skill level. We used a P value of 0.05 to determine statistical significance. We also analyzed satisfaction by EHR vendor. Finally we did a linear regression analysis to identify the relative contributions these user characteristics might make toward overall satisfaction.

Our results

These relationships all turned out to be statistically significant not just at P < 0.05 but even at the P < 0.01 level, that is, the 99-percent confidence level:

• User satisfaction with EHRs varies inversely with practice size – up to a point. Solo physicians are the most satisfied with their EHRs, and average satisfaction decreases as practice size grows to 50 physicians. Interestingly, though, medical groups with 51 or more physicians have satisfaction levels higher than practices ranging from 6 to 50 physicians.

• Family physicians in family-medicine-only practices tend to report higher satisfaction than those in multispecialty practices.

• Physicians in 100-percent productivity-based practices tend to report higher levels of satisfaction than those in salary-based practices.

• Physicians who had been involved in selecting their EHR systems reported greater satisfaction than those who had not.

• A weak positive correlation (17 percent) exists between satisfaction and the length of time one uses an EHR.

• A weak negative correlation (20 percent) exists between satisfaction and self-assessed skill level.

We did confirm that the EHR vendor and/or system does matter – that satisfaction is statistically related to which product one uses.

We recognized that these six variables were not likely to be totally independent from one another – that some probably exhibited covariation. Part of our covariate analysis confirmed the obvious. For example, physicians in larger practices are more likely to have been uninvolved in the selection of their EHR, have non-family-physician colleagues and be salaried. One relationship was not so obvious. In our study, the more experience physicians have with an EHR, the less likely they are to rate themselves as expert users.

We wanted to determine the relative strength of these factors in influencing satisfaction. In other words, we wondered if we could create a predictive model regarding EHR user satisfaction. By doing a stepwise linear regression of the characteristics outlined above, we learned that 17 percent of the variation in satisfaction that we saw in the survey results has nothing to do with which EHR product is used but rather depends on characteristics of the users and their practices as listed above.⁵ Although that doesn't sound like a large percentage, it can make a big difference in individual rankings. Let's explain.

A model: how user characteristics matter

“Some user characteristics that influence satisfaction” is a diagram of the relevant contributions of the four most influential factors. In our survey, the average overall satisfaction score was 3.3, the median score was 4, and the standard deviation in scores was 1.3 points on a scale of 1 to 5 where 1 indicated strong disagreement and 5 indicated strong agreement with the statement “I am highly satisfied with this EHR system.”

Based on our data and model, if we were able to set up a hypothetical situation where all user characteristics remained the same with the exception that no physicians were involved in selecting their EHR, we'd expect a significant drop in satisfaction across the board – but more so for some products than others. For example, 97 percent of the users of the highest ranking vendor for overall satisfaction, Amazing Charts, were involved in selecting their EHR. As you can see from the diagram, lack of involvement in the selection process is associated with a 0.78-point decrease in satisfaction score. So, without physician involvement in selection, we could expect a 0.76 point drop (0.78 x 0.97) in overall satisfaction for Amazing Charts. On the other hand, only 12 percent of the users of another highly rated program, Epic, were involved in EHR selection. For that group, with no physician involvement in selection, you would expect an overall satisfaction score drop of only 0.09 points (0.78 x 0.12). In other words, overall satisfaction would be expected to drop significantly for Amazing Charts and only negligibly for Epic in a scenario where physicians were uninvolved in the selection process. Their relative overall satisfaction ratings would change, as would those of many other vendors.

Conclusions

OK, so what's the message here? One message is that satisfaction isn't solely a function of which vendor physicians use, but is also related to user characteristics such as practice size, length of EHR use, self-assessed EHR skill level and role in the selection process.

Perhaps an even more critical message is that physician involvement in selecting their EHR is of great importance in user satisfaction. This is particularly significant for larger groups, who by their nature tend to have lower EHR satisfaction scores. We believe that physician groups of all sizes should do everything they can to involve all their physicians as much as possible in the EHR selection process.