Different levels of AI applications in diagnostic imaging

There are different levels where AI can be applied in radiology as well as other diagnostic imaging applications, depending on the step in the workflow from acquisition to interpretation, post processing and analysis.

The first level is at the Image Acquisition level. For example, one of the challenges with doing a CT scan is to have the patient center coincide with the center of the radiation beam, which will result in optimal dose distribution and corresponding image quality. In addition to the patient being centered, the distribution of the radiation dose, depending on the body part is also important, e.g. a lower dose for the head than for the pelvis. Instead of having a technologist making an educated guess, the machine can assist with this and automate the positioning process, again to optimize dose which means not using more than necessary.

De-noising of images is also an important feature. Typically, with lower radiation techniques, more noise is created, which can compromise a diagnosis and ultimately patient care. This is especially true for screening, where there is no direct indication to perform a CT study and limiting dose is important. An algorithm can be taught what noise looks like in a typical low-dose image and uses that knowledge to apply image processing to remove the noise to allow a lower dose technique to be used. The same principle is used to remove common artifacts such as created by metal parts in an X-ray. If the algorithm is taught how a typical artifact shows up in an image, it could remove it or, at a minimum, reduce it thus improving image quality and contributing to a better diagnosis.

An important feature for AI would be regulating the workflow, i.e. determining which cases should be considered “urgent” aka STAT based on automatic abnormality detection. These cases would be bumped to the top of the worklist to be seen by the radiologist.

The opposite is true as well, some of the images could be considered totally “clear,” i.e. having no indication and therefore not needing to be seen by a radiologist. This is useful in mass-screenings, e.g. for TB among immigrants, or black lung disease for people working in coal mines. These “normal” cases could be eliminated from a worklist.

The next level of AI is at the post-processing and reading level. CAD is probably the most common form of AI, where an image is marked using an annotation indicating a certain finding, which serves as a “second opinion.”

AI can also increase the productivity dramatically by assisting in creating a report. Macro’s can be used to automatically create sentences for common findings, again based on learning what phrases a user would typically use for a certain indication.

Standard measurements such as used for obstetrics can be automated.  The algorithm can detect the head and indicate automatically its circumference and diameter which are standard measurements to indicate growth.

One of the labor-intensive activities is the annual contouring of certain anatomical parts such as the optical nerve in skull images. This contouring is used by radiation therapy software to determine where to minimize radiation to prevent potential damage. Automating the contouring process could potentially save a lot of time.

Automatic labeling of the spine vertebrae for the radiologist also saves time, which could also improve accuracy. This time savings might only be seconds, but it would add up when a radiologist is reviewing a large number of such cases.

Determining the age of a patient based on the x-ray such as of a hand is a good example of quantification, another example is the amount of calcium in a bone indicating potential osteoporosis.

Some of the indications are characterized by a certain number of occurrences within a particular region, for example the number of “bad cells” indicating cancer in a certain area when looking at a tissue specimen through a microscope, or, in the case of digital pathology, displayed on a monitor. Labeling particular cells and automatic counting them offers a big time savings for a pathologist.

One of the frequent complaints heard about the workstation functionality is that the hanging protocols, i.e. how the images are organized for a radiologist are often cumbersome to configure and do not always work. AI can assist in having “self-learning” hanging protocols based on radiologist preferences and also be more intelligent in determining the body part to determine what hanging protocol is applicable.

As AI becomes integrated in the workflow, the expectation is that it is “always-on,” meaning that it is seamlessly operates in the background, without a user having to push any buttons or launch a separate application to have an AI “opinion.”

One of the challenges is also to make sure that relevant prior studies are available, which might need to be retrieved from local and/or remote image sources, for example from a VNA or cloud. AI can assist by learning what prior studies are typically used as a comparison and do an intelligent discovery of where they might be archived.

Not only do radiologists want to see prior imaging studies, but also additional medical information that might be stored in an Electronic Health Record or EMR such as lab results, patient history, medications, etc. Typically, a radiologist would have access to that information, especially as most PACS systems are migrating to become EMR driven, however for teleradiology companies, the lack of access to EMR data is a major issue, where AI might be able to assist.

AI is just starting to make an impact, we have only seen the tip of the iceberg, but it is clear that there can be major improvements made using this exciting technology.

The evolution of PACS through the years.

PACS systems have evolved quite a bit over the past 25 years, this essay provides the background of where PACS started, where we are now and where we are headed. I am covering the four essential PACS components, i.e. the P for Picture (viewing ), A for Archiving and image and information management, C for Communication and S for System.

Regarding the P for pictures: In the first generation view stations, the software was not as sophisticated, it had only basic functionality, and the viewers were thick clients, meaning that the images had to be downloaded to the local workstation and all of the processing was done locally. These view stations were mimicking a alternator, both in size and functionality, mostly displaying images in a landscape format.

By the second generation, radiologists discovered that they did not really needed 8 monitors but can view cross sectional studies using “stacking” and virtually integrating the3-D in their mind. The viewers added more sophisticated hanging protocols, aka DDP’s or Default Display Protocols, which refers back to how films were ”hanged” on a light box. How the images are sorted can depend on the modality, (e.g. Mammography), body part (e.g. Chest or extremity), specialty (e.g. Neuro) and individual preferences. Re-arranging images and sorting through literally hundreds of them in case of a cross-sectional study such as a CT or MRI is a burden for the radiologist and takes time. Inconsistent display can also be cause for medical errors, imagine that the new study is always displayed on the top of a monitor and the prior one on the bottom and that for some reason, this is reversed, this could cause the radiologist to report the wrong study. Voice aka Speech Recognition has become routine. Some studies, initially mammography, are subjected to Computer Aided Diagnosis which creates a “second opinion” for the radiologist by marking the images with CAD marks for clinical findings.

The 3^rd generation workstations are accommodating different specialties in addition to radiology such as cardiology, ophthalmology, dermatology, and others, commonly referred to as “ologies”. The viewer becomes a Universal viewer which instead of a thick client is now a thin client which does not leave any trace of patient information after the user has logged out, aka a “zero-footprint”. Some modalities create images and/or studies with huge file sizes in excess to 1 GigaByta, which makes it more efficient to do what is called “server-side” rendering whereby the viewer functions as a remote window to a server which performs the processing.

The fourth generation of viewers implement web services that also allow for mobile access, i.e. look at the images from a mobile device whether it is a tablet or smart phone using the DICOMWeb protocol. What used to be called CAD is now replaced with Artificial Intelligence or AI which spans many more detections of various diseases in addition to automating the workflow for the radiologist. As an example, AI can detect a critical finding and automatically bump the study to the top of the worklist. It can also remember and learn physician preferences and support his or her workflow.

The next component of the PACS is the Archiving and image and information management. The early generations of PACS systems were limited by cost of archive media. Most systems would archive studies with a certain age on a second or third tier, slower and less expensive media such as Magnetic optical disks, tape, or even store it off-line.

In the second generation, the big Storage Area Networks and Networked Attached Storage Devices were introduced having multiple arrays of inexpensive disks called RAID’s which is still the most common configuration. Because of some natural disasters and hardware failures, most hospitals learned the hard way that redundancy and backup is critical so most of these archive systems by now have at least one mirrored copy and a sound backup. CD’s become the standard for image exchange between physicians.

In the 3^rd generation, data migration as well as life cycle management is becoming a major issue. Many hospitals are replacing their PACS vendor and find out that it is really hard, costly and lengthy to migrate their images to another archive from a different vendor. They were looking for remote storage solutions, i.e. SSP’s, or buying a Vendor Neutral Archive (VNA) to take control over their image archive and not being dependent and locked in by a single PACS vendor. Some hospitals went all the way and deconstructed their PACS by buying workstations, workflow managers, routers in addition to their VNA and built their own PACS more or less from scratch, Cloud providers are making an in-road, and life cyscle management becomes important as not every hospital wants to store all the studies for ever but want to implement retention rules.

The fourth generation will see a shift to virtual storage, i.e. you won’t know or need to know where the images are archived, whether it is in the cloud or local, in which case it is most likely on solid state memory, providing very fast and reliable access. Images are now archived from anywhere in the enterprise, whether it is from a camera in the ER, to a Pint Of Care (POC) Ultrasound at the bed site or a video camera from physical therapy. The boundaries between documents and images is getting blurred, some store everything on one server, some use two distinct information management systems. Cyber security is a major concern, as malware is becoming a real threat and ransomware already has caused major downtimes, requiring strict security policies and mechanisms to protect the data.

The communication part of PACS has gone some major changes as well. Initially, each PACS had its own dedicated network, because sending images over the existing infrastructure would bring down the complete network. Speeds were up to about 100 Megabit/second, which was OK for the relative small image and study sizes. The second generation networks were upgraded to fiber instead of copper wire allowing speeds in excess of 1 Gigabit/Second. Network technology advanced allowing the PACS networks to be part of the overall hospital infrastructure by reconfiguring the routers and creating Virtual Local Area Networks aka VLAN’s. The third generation of network technology starts to replace the CD’s exchange with cloud based image exchange using brokers, i.e. having a 3^rd party taking care of your information delivery to patients as well as physicians. In the fourth generation, we see the introduction of Webservices in the form FHIR and DICOMWeb allowing for distribution on mobile devices, we are needing to create new profiles to deal with encounter based imaging instead of order based imaging using universal worklists and of course, security is becoming a major threat requiring firewalls, the use of DMZ’s to screen your outside connections and cyber security monitoring tools.

The fourth component of the PACS is the “System” component which mostly includes workflow support, of which there was initially very little. In the second generation, there has been a shift from PACS driven to RIS driven worklists and IHE starts to make an impact by defining multiple use cases with their corresponding HL7, DICOM and other standards. In the 3^rd generation, the annual IHE connectathons have made a major impact as it provides a neutral testing ground for proving that these IHE profiles really work. The worklists at the radiologist are becoming EMR driven and orders are placed using a Centralized Physician Order Entry (CPOE) system, often at the EMR. The last generation we see the use of cross enterprise information exchange starting to take place using IHE standards such as XDS, in a secure manner making sure that consents are in place and that authentication and audit trails are being utilized in the form of ATNA standards. Patients are also able to upload their information from the Personal Health Records (PHR) and wearables.

As you can see, we have come a long way since the early PACS days and we still have a bright future ahead of us. I am sure in another 5 years there will be some more changes to come.