Chances are that your practice depends on electronic data – for billing, at least, and for pretty much everything if you have an electronic health record (EHR) system. Safeguarding that data is crucial to the success of your practice. But as we have learned through a crippling loss of data, thinking your data is safe is not the same as knowing it is. The purpose of this article is to describe how our loss came about, what we learned from it and, most important, how to prevent this from happening to your practice.

The meltdown

The Family Medicine Residency of Idaho (FMRI) is an independent freestanding corporation affiliated with the University of Washington Residency Network system. FMRI comprises more than 60 providers (family medicine residents, faculty physicians and midlevel providers) handling over 46,000 outpatient visits per year at two clinic sites. When our meltdown occurred, we were about seven months into our EHR implementation and had been documenting essentially all outpatient visits in our EHR for more than four months.

We had been informed that the industry standard was to perform a partial backup each night and a complete backup (during which the system is unavailable) weekly, and these were the instructions given to our IT director. We believed that the backups were occurring, but we did not verify this. In fact, no complete backups were performed for a period of four months, and we didn't realize this until after we suffered a power outage one night at about 2 a.m.

An uninterruptible power supply (UPS) was automatically triggered; it could keep the EHR database server powered for 45 minutes. Due to misconfiguration errors, however, warning systems did not function properly, an orderly shutdown process was not initiated, and the server ran until the UPS battery was exhausted. The sudden, disorderly power loss to the server resulted in severe corruption of our EHR database. All of our data was still on the hard drives, but it had been garbled and was rendered unreadable. And due to our flawed backup scheme, our most recent recoverable data set was now four months old.

Approximately 12,000 patient visits worth of information was lost. Worse, as part of our EHR implementation, we had done much preloading of patient information during the lost four-month period. Literally thousands of hours of provider and nursing time had been spent to enter medication and problem lists, allergies, immunizations, past medical and social history, clinical summaries and visit notes. All of this information was presumed to be lost, and the initial response from our software vendor was that it was most likely nonrecoverable.

Our immediate reaction

We quickly set out to do everything in our power to attempt to recover the lost data. We shipped our EHR server's hard drives across the country to a company that specialized in data recovery. Our most pressing challenge, though, was to continue providing quality clinical care with four months of clinical data unavailable to us. We used every resource we could to piece together the recent clinical history. All patients were given a handout in the waiting room, informing them of the data loss and asking them to fill out a questionnaire that gave us pertinent recent clinical information. We frequently called pharmacies to acquire current medication lists. Being a residency training program, we were fortunate to have Medicare teaching forms that had been filled in by our faculty. These gave us problem lists, assessments and plans, and physical exams for the more complicated patients.

The event seriously affected all areas of our program. Good communication through clinic-wide meetings and frequent e-mails did a lot to maintain morale and to give staff support and direction in dealing with patients, third-party payers, other physicians' offices and attorneys who requested records related to care provided during the loss period.

Fortunately, 11 weeks after our data melt-down, we were able to recover nearly 100 percent of our data. This was the result of a lot of hard work by many people internal and external to our residency program – and a fee of more than $10,000 paid to the data recovery firm. The experience also taught us a lot about the safekeeping of electronic data, and it taught us that we never want to go through anything like it again.

Lessons learned

Computerized practice management and EHR technology have hidden costs, including the price of appropriately protecting data. But these costs are far less than the expense of being forced to scale back or shut down a practice because of a system failure. A high degree of reliability in EHR server hardware is certainly desirable, but as we learned, even highly reliable systems eventually fail. It is not a question of whether a system will fail, but how, when and whether we are prepared to manage the consequences.

For the future, we realized one central goal needed to be the ability to have our EHR system running again within one day following a failure like the one we experienced. And “within one day” was a worst-case scenario; all of our systems needed to be designed to further minimize potential downtime and maintain access to our data.

We used five values to guide our discussion as we set out to formulate a data protection plan:

1. Availability. Because of the mission-critical nature of our work (full-scope outpatient and inpatient family medicine, including obstetrics), we believed that our data should ideally be continuously available 24/7/365 – even in the event of data loss, data corruption or equipment failure. System downtime needed to be minimized. We also needed to broaden our thinking to consider various adverse conditions, including but not limited to an extended regional power outage that might last seven days, burglary, fire, flood and other natural disaster.

2. Redundancy. Increased redundancy – having multilevel, overlapping systems that work both synchronously and in isolation to protect our data while allowing us access to it – should theoretically lead to higher availability.

3. Security. Clearly, our data should always be secured against unauthorized access (theft, intrusion, malicious tampering, etc.). While data security in this sense is beyond the scope of this article, we recognized that we needed to broaden our thinking to include security as it related to our data backup systems (for example, theft of a backup tape from an individual's car or home was a possibility that needed to be protected against). Increasingly redundant systems can theoretically lead to additional avenues for security breaches.

4. Accountability. We believed that everyone should be accountable when it comes to data protection and disaster preparedness. Statements such as, “That's the responsibility of the IT department,” or, “I think they're backing up the data,” became unacceptable. This also included holding the backups themselves accountable, so to speak. In other words, the backups needed to be actively verified in such a way that proved that the data was retrievable and usable.

5. Transparency. We believed that everyone in our organization should have at least a basic overview of the steps being taken to accomplish the above objectives. Naturally, certain details would not be made public for security reasons, but we felt that transparency was imperative to regain and maintain the trust of our physicians, our employees and our patients. All of these stakeholders had invested large amounts of time and effort into working with our EHR, and they needed to be reassured that their work would never again be damaged or destroyed.

Next, it was clear that the oft-quoted and well-meaning (but vague) directive given by EHR vendors to physicians, “Make sure that you back up your data!” needed some additional clarification. We thought we were doing this from the start, and yet a deadly combination of ignorance and complacency put us in a dangerous predicament. We realized that our data needed to be protected in three different dimensions: time, space and method. In other words, data backup needed to occur at multiple times, backup media needed to be secured in different locations, and different backup methods needed to be employed.

Keeping the above five values and three dimensions in mind, we implemented a series of policies designed to safeguard our data and keep it continuously available. The various steps are detailed in “Implementing a three-dimensional backup plan,” page A6. Two critically important points are worth highlighting. First, we now verify that backup tapes are indeed viable by pulling specific tapes from all areas of the backup scheme at least weekly and doing a true test-restore to a spare server. The test restore is the only way to definitively document that backup media will be helpful in an emergency. And second, we no longer assume that backups are being performed correctly by a single individual; rather, we enforce an accountability process which ensures that more than one person is monitoring our data protection procedures.

More work to do

In this article, we have chosen to focus exclusively on protection of the EHR database. However, we feel that the general principles outlined above can be successfully applied to most databases relevant to a family physician's practice. Attention to the five values and three dimensions of data protection will go a long way toward preparing a practice to weather the storms of unexpected system failures.

We recognize that the principles we have outlined in this article are neither perfect nor exclusive of other solutions. They simply constitute our best efforts to understand and address this problem. We are still working out the implementation details in our own plan, and we will no doubt need to make changes as we go. Physicians will need to customize their approaches depending on their own individual tolerance for data loss and EHR downtime, cost constraints (“high availability” systems tend to incur extra expense) and access to knowledgeable IT personnel. Some practices may choose to adopt procedures that are less complex and less costly than ours, while others may choose to be even more stringent.

During our EHR database crisis, open communication and venting of frustration were critical. Accountability, transparency and daily communication were paramount. A sense of “we are all in this together” and a positive, proactive attitude of “getting through this and becoming stronger by it” permeated our culture. In fact, it was this attitude that led to the writing of this article, so that others do not repeat our mistake.