Help! PACS is Down!

There is no question whether your PACS will go down or not, the only question is when, how often and how do you prepare and anticipate it, in other words, how can you minimize the panic factor?

Downtime is generally understood as the period during which a system is non-functional or cannot work. Note that it does NOT include the time that a system is potentially slowing down as it is auto-repairing a failure, such as can be the case for a disk crash which is part of a redundant RAID configuration, or a server failure, which is automatically taken over by a mirrored server. It does not include scheduled downtime either, which is used for software upgrades and maintenance.

According to Mike Cannavo, aka “the PACSman,” a typical RFP for a PACS system requires an uptime of the mystical “five nine’s,” which is 99.999 percent, or 5 minutes/year. However, I have yet to see a PACS that is down only 5 minutes/year, a more realistic number according to Mike is 99.5%, which equates to about 44 hours, which includes scheduled downtime. However, it is critical to have a measure of system performance that constitutes a “downtime,” for example, I would consider retrieval time of an image slowing down to one minute a downtime, while others might not.

What is a typical amount of downtime? I usually ask my PACS system analyst (SA) students when was the last time that their system was down, and I have found that it ranges widely, between one student whose system went down once a week (I would not want to be on call for that system) and those who can’t even remember the last time it was down as it was several years ago. Based on their feedback, it appears that once every six months seems to be the norm and/or average. The average downtime seems to be a couple of hours. Taking into account the numbers from Mike, that seems to be not far off the norm.

Which measures can you implement to take the panic out of a system going down, i.e. considering it being an unscheduled downtime?

1.      Have well-defined downtime procedures, which are visible and have all the users trained on how to use them. The procedures depend on the user, so have a little “cheat-sheet” at their desk telling them what to do. For example, for a technologist at a modality, it might say “select alt PACS” to send images to, for a radiologist it would say “select alt PACS worklist,” text PACS SA, or “Use web viewer,” etc. And as mentioned, train the users so they know what to do.

2.      Have a test system. Surprisingly enough, when I did a poll, I found out that only about two-thirds of users has a test system in place. Not only should there be a test PACS but also a test worklist provider, voice recognition system, and any other critical component. The test system is used to test updates, including patches, train users in new features, and most importantly provide a “life-support” while the system is undergoing scheduled maintenance or experiencing an unscheduled downtime.

3.      Use mirroring. This is different than having a test system, a mirrored system is a fully functional, operational duplicate of the main system, preferably at a different location. For North Texas where I am based, that means sufficiently far away that a tornado would not hit both centers, for southern Louisiana it would mean in another state not subject to the same hurricane or flooding. For California that would mean not on the same fault line subject to an earthquake.

4.      Test your downtime backup. How do you know if your backup solution works? You’ll have to test it, which is a legal requirement for the state of Texas for all government/state institutions. For example, at UT Southwestern in Dallas, they will run their orders from an external system once a year to show it can be done.

5.      Have an alternate workflow for critical areas. One of my students told me that he burns a CD for all cases in the OR and sends them up to the location every day, just in case the system goes down. The same can be done on-demand for critical cases in the ICU in case PACS (or the network) is down. Or, subsequently, one could burn CD’s in the ER for reading at a stand-alone station in radiology.

6.      Have a dual source for the information. Many hospitals used to have a separate web server that stored a copy of the images in a web-accessible format that can be viewed from any PC in case the PACS is down. Unfortunately, from a redundancy perspective, many of these web servers have gone away as PACS systems have integrated those in their main archive. The trend to have a separate VNA as an enterprise archive, however, gives back that duplication.

7.      Have more than one access point. In addition to having multiple sources of the information, having multiple access points is just as important, such as the capability to look at images on PC’s, tablets, or even a phone, not necessarily with the same quality but good enough for an emergency. This is not unheard of, I know of a surgeon who takes a picture with his phone from his view station and shares these with his surgical team on a regular basis.

8.      Reboot, re-initialize on a regular basis. In the early days of Windows implementations there were quite a few hiccups and I remember that we were able to reduce the downtime significantly by auto-rebooting each computer automatically every night at midnight. Software is sometimes funny; there can be “loose threads,” unclaimed or unreleased blocks of memory, or multiple unnecessary background processes running that could impact performance or reliability which is simply cleaned up by a reboot.

9.      Be aware of external factors. One of the most common reasons for system downtime is people cutting through cables, or sharing the wiring closets with plumbing. This is especially common when there are multiple campuses, where there could be someone digging a hole somewhere and impacting power or network availability. Even air-conditioning can bring a system down. Just last week a major brand-new facility here in the north Dallas metroplex had to shut down its server room as the A/C was down. And, for some reason, architects like to position IT in the basement, which obviously is the worst place for flooding and water breaks. Ideally it would be best to locate them on the top floor of a building, but realistically that is in many cases prime real estate.

10.   Constantly monitor your weak points and critical components. When I visited a PACS SA room not too long ago, I saw a monitor on one of the desktops that was scrolling a set of what looked like text strings. Upon asking, he told me that these are his RIS feeds containing all the orders for his PACS. He had no clue as to how to interpret the HL7 order messages, but he knew that as soon as the screen would stop he had a problem, as orders are not coming in anymore. As most of you know, in a typical size department, a one-hour RIS downtime results in a full day of fix-ups at the PACS back-end so he was very keen to monitor that data stream non-stop.

Being down does not have to result in panic. If proper procedures and methods are in place everyone knows what to do and you have time as an imaging and IT professional to fix he problem and get the system back up and running. In addition, having the right infrastructure and architecture as well as tools are essential. But system reliability is a factor, if your system is down once a week you might want to look for another vendor.

DICOM connectivity maturity model

The majority of the PACS installations do not use good practices. The main reason is a lack of
education and knowledge, both from the end-user (PACS administrator) side as well as from the vendor side. Service and support engineers typically want to be in-and-out and avoid any extra work and might not set up the more advanced features of your PACS. The same applies for any new modality connections; the company engineers will set up the bare minimum and not want to bother with any advanced capabilities, which will impact your workflow and operation.

Here are my recommendations for the most important good practices, organized in different steps, each representing an incremental maturity level.

1.      Use common sense AE Titles and standard ports - the DICOM AE-Title is a fixed application level address of your DICOM application. Most devices have one AE-Title, some have multiple (e.g. a PACS might have four, one for its archive, one for the image manager/database, one for the print server and one for the modality worklist provider). Use only upper case characters and the control character “_” (it is case sensitive), use easy to remember mnemonics (e.g. CT_ER), and prefix it with the facility code to make cross-enterprise connectivity easy (e.g. STJGNL for St. Joseph General hospital). Remember you only have 16 characters to encode these, so be creative.

The well-known DICOM port which many use is port 104, however, this might not always be available as some systems limit access to the lower number ranges. To prevent potential conflicts, stick to the “official” DICOM port number, which is assigned by the Internet Assigned Numbers Authority (IANA) as 11112.

2.      Standardize on a preferred Transfer Syntax  - Many modalities have the option to offer Explicit VR when they negotiate transferring images and other objects. Always select this option, don’t use Implicit VR unless you absolutely have to. Explicit VR requires a sender to specify the VR (data type) for all of its header attributes, which is useful for private data. In addition, if you ever want to exchange the images on an exchange media such as a CD, implicit VR could cause issues as it is not allowed on exchange media.

Avoid using Big Endian, which has become less of an issue as most modalities today use Little Endian Byte order, however, if there is an old modality which uses the Unix operating system (and therefore the Big Endian encoding), configure either the sender to only send Little Endian, or configure the receiver to accept only the Little Endian. The reason is that some viewers might not be able to view those, and, similar to the Implicit VR encoding, it is not allowed for exchange media.

Stay away from any proprietary compression encodings, i.e. that are non-DICOM standard, and use the standard JPEG lossless and lossy for your modalities and Wavelet Lossless and Lossy for long term archiving.

3.      Avoid any proprietary and raw image objects (SOP Classes) - There are two types of proprietary objects, the first type is easy to recognize as it carries a proprietary unique identification (UID), which means that they are easy to detect. Many PACS systems will refuse to archive them, and I have incompatibility issues with these if someone puts them on a CD for image exchange, for example, it is not uncommon for private CR objects to end up on CD’s. Other applications are the 3-D private ultrasound or angio/cardio applications. The second group of proprietary objects are the ones that are “standard” on the surface, i.e. it is a standard image object such as a CT or secondary capture, but under the cover it is really another object, which became popular with the onset of the new breast tomo-synthesis modalities. The latter created a lot of incompatibility issues, one vendor would be able to recognize that the CT image was really a mammography but anyone else would get really confused and hanging protocols as well as pre-fetching algorithms would fail.

With regard to archiving raw data, I come across institutions that archive their raw data for mammography images. The raw data is only used to run a CAD algorithm, and unless you are an academic institution and want to save this raw data to test future new CAD algorithms, this would more than double your mammography image storage requirements. Only keep the image data, and don’t archive the raw data.

4.      Archive only thick slices and enhanced “anything” image types (SOP Classes) - CT scanners are increasing their number of slices, a 64-slice is becoming more or less the standard. However for cardiac applications, one might have a 320- or even up to 640-slice detector. The number of thin slices can easily run into the thousands. These thin slices are great for 3D imaging as they allow access to “orthogonal” voxels, i.e. having identical resolution in any direction to allow for lossless MPR, and 3D rendering. However, these thin slices don’t really provide additional clinical information that can’t be seen by averaging multiple thin slices into thick slices and stacking them for review by the radiologist. Therefore, keep the thin slices for let’s say a month so you can do additional 3-D views if you wish, but only permanently store these reconstructed 3D’s with the thick, averaged slices and save multiple factors of storage.

The new CT’s, MR’s and also the angio and cardio (“anything”) modalities typically support two types of image formats, the traditional format, e.g. CT or MR image storage and the “enhanced” versions. The enhanced image headers are more comprehensive, have more information, are much more standard with regard to the different series extending them to multiple “dimensions” and information in the header uses standard encoding for most clinical terms. In short, the enhanced versions are a more complete standard encoding between multiple vendors. The hanging protocols are more robust based on elaborate header information. Last but not least, these objects are multiframe, which means that the protocol overhead to transfer a study is more effective as it happens in a single transaction instead of having to use thousands of transactions back and forth to exchange one of these studies.

5.      Use the DICOM workflow services, i.e. Storage Commitment, Modality Performed Procedure Step (MPPS) and Instance Availability (IA) - Almost any modality supports Storage Commitment to provide a secure “hands-off” between a modality and PACS. Using this prevents images from falling between the cracks and getting lost. The same applies for the hands-off between a PACS and VNA, or cloud storage. It is just a matter of configuring this feature. The same applies for using MPPS, use it where you can as it provides a hands-off between the modality and PACS allowing for studies to appear on a radiologist worklist as soon as they are available, and automating any updates caused by changes in the procedures. IA is even better for identifying that processing and/or reading can be done as a next step. Also make sure your PACS and VNA support the new IHE profile IOCM (Image Object Change Management) to synchronize any changes in patient demographics, which will save your PACS administrator a lot of extra work.

6.      Use Structured Reports instead of a screen capture for CT dose and Ultrasound measurements, and switch off embedded US annotations - It is easy for a radiologist to look at the CT screen containing the dose information and record that as part of the report (which has become standard practice), however, a digital interface to the reporting system would allow this information to be automatically captured, saving time, increasing accuracy, and having much more detailed information in these reports that can be used for quality reporting and improvement.

Use Structured Reports for measurements from your ultrasound instead of re-recording all these measurements which is time consuming and prone to errors. Using the true digital interface to a reporting system is much faster and more accurate.

Switch off embedded ultrasound annotations. There is absolutely no need to have the patient name “burned-in” the image data as the DICOM header has this information. Burned-in data is very hard to get rid of in case corrections have to be made to the name, or the data has to be anonymized for whatever reason.

In conclusion, none of these features are new, they are proven, established more than 5 years ago, available in pretty much all of the systems out there, and activating them is just a matter of configuration. If you do so, you will be able to provide a more robust and more efficient infrastructure to your medical imaging department. And, as an extra bonus, you could claim to be compliant with a certain level of DICOM connectivity maturity. 100 percent compliance might be hard, but I would argue that 80 percent compliance for each level would qualify.