6: Iterative and deep learning reconstruction basics

Assumptions made to derive the FBP algorithm

The problems with the FBP algorithm are due to several “idealized” assumptions made when deriving the algorithm. Fig. 6.2 has been used by Hsieh (2008) to illustrate the nature of these assumptions, which include:

The above assumptions of the CT system have been used successfully to derive the FBP algorithm. Such a CT system model “clearly does not represent physical reality” (Hsieh et al., 2013); therefore, other algorithms that model the CT system accurately are needed. These algorithms must consider accurate modeling of the CT system optics, the noise (noise model), and the image properties (image model). The highlights of modeling will be described later in the chapter. This is an important point to note, as accurate modeling results in improvements in image quality and noise reduction when compared with the FBP analytical algorithm.

Noise reduction techniques

The increasing use of CT has resulted in a corresponding increase in radiation dose to the patient and concerns about the potential biological effects of radiation exposure from CT examinations (Brenner & Hall, 2007; Pearce et al., 2012). Subsequently, these concerns have resulted in increasing awareness of various efforts to reduce the dose to the patient in CT scanning (Seeram, 2014; Seeram 2018).

It is well known that the use of low-dose techniques can produce noisy images; therefore, it is essential to consider various approaches to noise reduction, especially in low-dose CT imaging. Fig. 6.3 shows four strategies to reduce the noise during CT image reconstruction. These include projection space denoising, analytical FBP denoising using standard reconstruction kernels (see Chapter 5), image space denoising, and the use of iterative reconstruction (IR) (Ehman et al., 2014). Whereas image space denoising and FBP-based analytic reconstruction use digital filters (e.g., smoothing filters, edge enhancement filters; see Chapter 2), projecting space denoising (filtering before image reconstruction) uses photon statistics to reduce noise through a statistical noise model or adaptive noise filtering (Ehman et al., 2014)

IR algorithms have been developed to reduce image noise in low dose CT, and at the same time to preserve image quality (namely, spatial resolution and low-contrast detectability [LCD]) and reduce artifacts due to the presence of metal implants, beam-hardening effects, and photon starvation (Ehman et al., 2014). These artifacts (and more) are common when using the traditional FBP algorithm (Hsieh et al., 2013; Zeng & Zamyatin, 2013).

The fundamental differences between the FBP and IR algorithms are illustrated in Fig. 6.4.

IR algorithms use better defined mathematical estimates (modeling) and multiple iterations to produce images with less noise at low dose exposures compared with the FBP algorithm. It is visually clear that the FBP image is much noisier than the IR image. The word iteration simply refers to a calculation process that uses a series of operations repeated several times.

There are several IR algorithms commercially available from CT scanner manufacturers, based on different physical methods and concepts. These algorithms must consider accurate modeling of the CT system optics, noise (noise model), and the image properties (image model).

In generally, IR algorithms differ in the way the measured projection data and the artificial raw data are compared and the method in which correction is applied to the current estimate.

IR algorithms have been placed in the following three categories, as illustrated in Fig. 6.5:

IR algorithms without modeling: Fundamental concepts

A generalized or typical IR process is illustrated in a flowchart in Fig. 6.6, and consists of the following steps:

A comparison of images reconstructed using FBP and an IR algorithm is shown in Fig. 6.7.

The accuracy of the estimate mentioned previously depends on what is referred to as modeling the CT system acquisition components and the object being imaged. These will be briefly described in the following section.

Modeling approaches in IR algorithms: An overview

The term modeling is used to refer to the characteristics of the CT imaging system and the imaged object, and includes what has been popularly referred to as system optics (optics model), noise statistics (noise model), and object and physics models. These different system models are illustrated in Fig. 6.8, where the points to which they are applied during the CT IR process are shown.

Whereas the optics model or system optics addresses the x-ray source, image voxel, and detectors, and so on, the noise model takes into consideration the x-ray photon statistics (x-ray flux at the source and detector). The object model, on the other hand, deals with radiation attenuation through the patient, whereas the physics model examines the physics of CT data acquisition. The interested reader should refer to Beister et al. (2012) and Hsieh et al. (2013) for a more detailed account of these models.

Whereas some IR algorithms model the system optics, others model the system noise statistics, physics, and objects. As noted by Hsieh et al. (2013), a “fully modeled IR is mor computationally intensive than analytical reconstruction methods.” Furthermore, Hsieh (2008) also points out that “the computational intensity on the modeling of the noise portion of the system is not nearly as big as the modeling of the system optics. Therefore, by focusing first on the modeling of the noise properties and the scanned object, we provide significant benefit for those examinations that may experience limitations due to noise in the reconstructed images, as a result of lower dose exams, large patients, thinner slices, etc. Phrased differently, we gain the dose benefit by lowering the noise in the reconstructed images, so that the scanning technique can be reduced with equivalent noise.”

Iterative methods without modeling compute an initial estimate of the image from the projection data set (using the FBP image), subsequently improving the image (current estimate) using several iterations (a calculation process using a series of operations repeated several times) (Patino et al., 2015).

Model-based IR methods

As early as 2012, Beister et al. (2012), provided a description of model-based IR methods as follows: “...model-based methods try to model the acquisition process as accurately as possible. The acquisition is a physical process in which photons with a spectrum of energies are emitted by the focus area of the anode of the x-ray tube. Subsequently, they travel through the object and are either registered within the area of the detector pixel, scattered outside of the detector, or are absorbed by the object. The better the forward projection is able to model this process, the better the artificial image can be matched to the acquired raw data. Better matching artificial raw data leads to better correction terms for the next image update step and, in consequence to an improved image quality in the reconstruction data.”

Statistical and model-based methods are outside the scope of this book. The interested reader should refer to the paper by Beister et al. (2012) for a comprehensive description of the core principles of these two methods. A graphic illustration of the statistical iterative reconstruction algorithm (ASiR) and a model-based IR algorithm (Veo compared with the FBP analytical reconstruction algorithm) is shown in Fig. 6.9.

Recall that the goals of IR algorithms compared with the FBP algorithm are to improve image quality (namely, spatial resolution or sharpness of the image), reduction of noise, better low-contrast detectability (LCD), and artifact reduction. Such image quality improvement is shown in Fig. 6.10. This image quality improvement and reduction of dose depend on accurate modeling. For the improvement of image quality, it is important that the system optics are modeled accurately in the reconstruction process. For reduced dose, better LCD, and artifact reduction, it is mandatory that the noise statistics, the physics, and the object are modeled accurately in the reconstruction process.

Examples of IR algorithms from CT vendors

Several IR algorithms are currently available from major CT vendors such as General Electric (GE) Healthcare, Siemens Healthineers, Philips Healthcare, and Canon Medical Systems (formerly Toshiba Medical Systems). Each vendor offers propriety algorithms that provide improved image quality and noise reduction in low-dose CT imaging (dose reduction) compared with the FBP algorithm. Examples of these IR algorithms are listed in Table 6.1.

Furthermore, a literature review on the performance evaluation of these IR algorithms between 2012 to 2015 (Qiu & Seeram, 2016) concluded that “the results share the general consensus that IR algorithms do faithfully reduce radiation dose and improve image quality in CT examinations, when compared with the FBP algorithm. The use of IR algorithms definitively reduces objective image noise even at reduced radiation dose levels, while subjective image quality, in terms of spatial resolution and low contrast detectability, is improved or preserved with the use of IR algorithms as long as the radiation dose is not reduced excessively. However, the use of IR algorithms of excessively high iteration levels should be avoided, as the produced smoothing effect can negatively impact the subjective image quality. The use of IR algorithms may also be unable to preserve image detail in denser structures such as bone, although further research may be needed to validate this observation. With the rapid improvement in technology, IR algorithms will likely become the preferred CT image reconstruction method in the foreseeable future.” Today, IR algorithms have become commonplace in CT image reconstruction.

Deep learning reconstruction: An artificial intelligence-based approach-a new era

Artificial Intelligence (AI) is gaining widespread attention in healthcare and especially in medical imaging. One definition of AI given in Chapter 1 is that AI is the “effort to automate intellectual tasks normally performed by humans” (Chollet, 2018). As noted by Chartrand (2017), AI is “devoted to creating systems to perform tasks that ordinarily require human intelligence.” There are two subsets of AI, Machine Learning (ML), and Deep Learning (DL) which is a subset of ML, as illustrated in Fig. 6.11. Whereas ML is the use of algorithms that “are trained to perform tasks by learning patterns from data rather than by explicit programming” (Erickson et al., 2017), DL uses algorithms that are “characterized by the use of neural networks with many layers” (Chartrand et al., 2017). Furthermore, DL is based on the use of “a special type of artificial neural network (ANN) that resembles the multilayered human cognition system” (Lee et al., 2017). Additionally, the neural network many layers refer to “layers of mathematical equations and millions of connections and parameters that get trained and strengthened based on the desired output” (Hsieh et al., 2019).

Humans think through the use of a complex network of real neurons (units of the nervous system) that make use of chemical and electrical signals. The conceptual difference between real neurons and artificial neurons is illustrated in Fig. 6.12. As shown in Figure 6.12A, the neuron is made up of a cell body, dendrites, axons, and synapses, and incoming signals are transmitted from one neuron to the next via synapses. In Figure 6.12B, the ANN is characterized by interconnected artificial neurons, where “each artificial neuron implements a simple classifier model that outputs a decision signal based on the weighted sum of evidences. Hundreds of these basic computing units are assembled together to establish the ANN. The weights of the network are trained by a learning algorithm, such as back propagation, where pairs of input signals and desired output decisions are presented, mimicking the condition where the brain relies on external sensory stimuli to learn to achieve specific tasks” (Lee et al., 2017). The elements of AI, including DL will be discussed further in Chapter 20.

Deep learning CT image reconstruction: Basic concepts

DL algorithms are now being applied to CT image reconstruction, which has been referred to as Deep Learning Reconstruction (DLR), or Deep Learning Image Reconstruction (DLIR). The ultimate goal of developing DLR is to provide improved image quality, dose performance, and reconstruction speed compared to previous image reconstruction algorithms, especially in low-dose CT scanning (Jensen et al., 2020). Two are currently available from CT vendors, Canon Medical Systems (Boedeker, 2019) and General Electric (GE) Healthcare (Hsieh et al., 2019) and were approved by the Food and Drug Administration in 2019. While Canon’s DLR CT reconstruction algorithm is referred to as the Advanced Intelligent Clear-IQ Engine (AiCE) DLR algorithm, the DLR algorithm from GE is referred to as TrueFidelity.

Limitations of FBP and IR algorithms

In Fig. 6.13, the influence of low-dose CT and the choice of the image reconstruction algorithm on image quality is illustrated. It is clear that in a high-dose CT situation, using the FBP algorithm the image quality is excellent; however, in a low-dose CT with the FBP algorithm, the image quality is poor. The solution to this problem is to use IR algorithms to provide improved image quality (compared to the low-dose FBP situation) using low-dose CT. IR delivers less image noise and artifact, “however, the smoothing artifact imparts a plastic-like, blotchy image appearance that has been observed in all iterative reconstructions particularly at higher levels. The modified appearance is considered a limitation of these techniques, and it affects the evaluation of CT scan images and, arguably, the interpretation of imaging findings” (Kim et al., 2019).

In an effort to provide excellent image quality using low dose CT, DLR algorithms have been introduced into the domain of CT image reconstruction. This development has often been referred to as a new era in CT image reconstruction (Hsieh et al., 2019).

Generalized framework for deep learning reconstruction

The location of a DL neural network training engine in CT is illustrated in Fig. 6.14. There are several types of deep learning neural networks, or deep neural network (DNN) for short, that are used for different purposes. For example, the Convolutional Neural Networks (CNNs) are neural networks that are used “primarily for classification of images, clustering of images and object recognition. CNNs enable unsupervised construction of hierarchical image representations. CNNs are used to add much more complex features to it so that it can perform the task with better accuracy” (Shukla & Iriondo, 2020). CNNs are not within the scope of this book; however, Do et al. (2020) provides a very brief description of a CNN as follows: “CNN is one of the artificial neural networks that can be characterized by a convolutional layer, which differs from other neural networks. A typical CNN is composed of a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer is the core of a CNN. In mathematical terms, “convolution” refers to the mathematical combination of two functions to produce a third function. When used in a CNN, convolution means that a kernel (or filter = small matrix) is applied to the input data to produce a feature map” (see Chapter 2 for the basics of convolution). CNNs have been applied exclusively in CT image reconstruction for low-dose computed tomography (LDCT) image denoising, and have been labelled DLIR (Chen et al., 2017; Kang et al., 2018; Shan et al., 2018).

In general, there are at least three phases for deep learning image reconstruction in CT (Hsieh et al., 2019) shown in Fig. 6.15: (1) design phase, (2) training and optimization phase, and (3) verification phase. Only the latter two will be highlighted in this chapter

Training and optimization

As seen, in Figure 6.14, both low-dose CT images created from the low-dose CT sinogram as well as high-dose CT images created from the high-dose sinogram (obtained from training data sets) are sent into the DL reconstruction (DLR) training engine where “learning” of features of the image occurs, and image parameters are adjusted, to provide an output image.

The training phase for GE’s TrueFidelity (Fig. 6.16) involves the input of a “low dose sinogram through the Deep Neural Network and comparing the output image to a ground truth image – a high dose version of the same data. These two images are compared across multiple image parameters such as image noise, low contrast resolution, low contrast detectability, noise texture, etc. The output image reports the differences to the network via backpropagation which then strengthens some equations and weakens others and tries again. This process is repeated till there is accuracy between the output image and the ground truth image” (Hsieh et al., 2019). Additionally:

• • The DL neural network training engine uses thousands of ground truth training data sets to create a mathematically exact ground truth CT image (reference image). In deep learning, the term is used to check the accuracy of the deep leaning algorithm results using factual data (objective data) fed into the DL engine.
• • The output CT image is then compared with the ground truth image (reference image) on image quality metrics such as, for example, image noise, spatial resolution, contrast resolution, low contrast detectability, noise texture, CT number accuracy, and anatomical features, to check for differences. The engine repeats the “learning” process until the output CT image matches the ground truth image (reference image) (Hsieh et al. 2019).

The training phase for Canon’s AiCE, illustrated in Fig. 6.17, involves the following elements (Lenfant et al., 2020; Boedeker, 2019):

• • First, a full dose CT sinogram is reconstructed by the model-based IR (MBIR) algorithm and a hybrid-IR algorithm using different scan conditions to create reference data sets (high-quality output and a low-quality input, respectively). These are fed into the CNN, where “learning” of features of the image occurs and image parameters (such as noise, spatial and contrast resolution, low contrast detectability, noise texture, and so on) are adjusted, to provide an output image.
• • Secondly, the DLR engine applies mathematical techniques to determine the amount of error (estimation error) between the output image and the reference datasets (ground truth reference) from which high-dose reference images are created from thousands of phantom and patient images using high tube current and the MBIR algorithm, and
• • Thirdly, the output image and the reference image are compared using the estimation error, and the CNN acts to decrease any “discrepancy” through the use of forward-backward propagation (Table 6.2) to obtain optimization (Lenfant et al., 2020; Boedeker, 2019).

TABLE 6.2

Examples of Various Terms and Concepts Used in DL Neural Networks

Terms/Concepts	Brief Descriptions
Ground Truth	A term used in various fields to refer to data and/or methods related to a greater consensus or to reliable values/aspects that can be used as references. In deep learning (DL), it usually refers to “correct labels” prepared by experts.
Data Curation	Organization and integration of data collected from various sources to add value to the existing data. Data curation includes all the processes necessary for principled and controlled data creation, maintenance, and management. Medical imaging data curation may include data anonymization, checking the representative of the data, unification of data formats, minimizing noise of the data, annotation, and creation of structured metadata such as clinical data associated with imaging data.
Data Augmentation	Acquiring large amounts of good-quality data is often difficult in clinical radiology. Data augmentation is an approach that alters the training data in a way that changes the data representation while keeping the label the same. It provides a way to artificially expand a dataset and maximize the usefulness of a well-curated image dataset. Popular augmentations of image data include blurring or skewing an image, modifying the contrast or resolution, flipping or rotating the image, adjusting zoom, and changing the location of a lesion
Training, Validation, and Test Data	Training data are used to train and optimize the parameters of the model. The validation data are used to monitor the performance of the model during the training and to search for the best model. The test data are used to finally evaluate the performance of the developed model.
Parameter versus Hyperparameter	A parameter refers to a variable that is automatically adjusted during the model training. A “hyperparameter” is a variable used to design the deep learning model and is a variable to be set before training the model.
CNN	CNN is one of the artificial neural networks that can be characterized by a convolutional layer, which differs from other neural networks. A typical CNN is composed of a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer is the core of a CNN. In mathematical terms, “convolution” refers to the mathematical combination of two functions to produce a third function. When used in a CNN, convolution means that a kernel (or filter = small matrix) is applied to the input data to produce a feature map.
Cost Function (Loss Function) Cost (Loss)	This is the difference (not necessarily a mathematical difference) or compatibility between the true values (ground truth) and the values predicted by the model. A cost function (loss function) is a function comprising the true value and the predicted value and it is used to measure cost.
Forward Propagation and Back Propagation	Forward propagation is a process used to calculate the predicted value from input data through the model. Back propagation is a process used to adjust each parameter of the model to minimize the cost.

(These terms have been quoted from an article by Do, S., Song, K.D., Chung, J.W. (2020). Basics of deep learning: a radiologist’s guide to understanding published radiology articles on deep learning. Korean Journal of Radiology, 21, 33–41. Reproduced under Creative Commons License.)

Verification or performance

The final phase is verification or performance of the DLIR engine to “reconstruct clinical and phantom cases it has never seen before, including extremely rare cases designed to push the network to its limits, confirming its robustness” (Hsieh et al., 2019). Image quality metrics used for the evaluation includes, for example, noise reduction, noise texture, and contrast-to-noise ratio (CNR). The results to date indicate that DLIR outperforms IR algorithms on these parameters (Hsieh et al., 2019; Boedeker, 2019; Shan et al., 2019; Akagi et al., 2019) including fast reconstruction speed to meet the needs of clinical CT examinations.

The DLIR engine also features different levels of reconstruction, such as low, medium, and high, to address the amount of noise reduction. Fig. 6.18 shows the image noise reduction for the FBP, an IR algorithm (ASiR-V), and the DLIR on low, medium, and high levels of image reconstruction. This is important in both the development of clinical protocols and the radiologist preference for fast reconstruction speeds to meet the needs of the clinical CT examination (Hsieh et al., 2019). Furthermore, other studies have shown that DLR:

• • “significantly improved image noise and the overall image quality...and offered an additional significant radiation dose reduction while allowing slices to be twice as thin as compared to hybrid-IR. Thus, DLR can yield better image quality that hybrid-IR while reducing radiation dose” (Lenfant et al., 2020)
• • “On DLR images, the image noise was lower, and high-contrast spatial resolution and task-based detectability were better than on images reconstructed with other state-of-the art techniques. DLR also outperformed other methods with respect to task-based detectability” (Higaki et al., 2020)
• • “Deep learning can be used to improve the image quality of clinical scans with image noise reduction” (Nakamura et al., 2020)
• • “New DLIR algorithm reduced noise and improved spatial resolution and detectability without changing the noise texture. Images obtained with DLIR seem to indicate a greater potential for dose optimization than those with hybrid IR” (Greffier et al., 2020)

Direct application in clinical environment (hospitals/clinics)

When using the CT scanner with DL reconstruction algorithm in the clinical environment (hospital or clinic), the flow of the reconstruction process using DL algorithms is illustrated in Fig. 6.19. First the CT scanner produces a low-dose CT data set (low-dose sinogram) which goes through the first image reconstruction using either the FBP algorithm or an IR algorithm. Subsequently, these reconstructed images are sent into the deep convolution neural network engine for further processing. The result is a high-quality CT image created from the original low-dose input CT data set. A visual comparison of images reconstructed with the FBP, IR, and DL reconstruction algorithms is shown in Fig. 6.20.

Review questions

Answer the following questions to check your understanding of the materials studied.