medical image computing medical image computing medical image computing (mic) is an interdisciplinary field at the intersection of computer science, information engineering, electrical engineering, physics, mathematics and medicine. this field develops computational and mathematical methods for solving problems pertaining to medical images and their use for biomedical research and clinical care. the main goal of mic is to extract clinically relevant information or knowledge from medical images. while closely related to the field of medical imaging, mic focuses on the computational analysis of the images, not their acquisition. the methods can be grouped into several broad categories: image segmentation, image registration, image-based physiological modeling, and others. data forms medical image computing typically operates on uniformly sampled data with regular x-y-z spatial spacing (images in 2d and volumes in 3d, generically referred to as images). at each sample point, data is commonly represented in integral form such as signed and unsigned short (16-bit), although forms from unsigned char (8-bit) to 32-bit float are not uncommon. the particular meaning of the data at the sample point depends on modality: for example a ct acquisition collects radiodensity values, while an mri acquisition may collect t1 or t2-weighted images. longitudinal, time-varying acquisitions may or may not acquire images with regular time steps. fan-like images due to modalities such as curved-array ultrasound are also common and require different representational and algorithmic techniques to process. other data forms include sheared images due to gantry tilt during acquisition; and unstructured meshes, such as hexahedral and tetrahedral forms, which are used in advanced biomechanical analysis (e.g., tissue deformation, vascular transport, bone implants). segmentation a t1 weighted mr image of the brain of a patient with a meningioma after injection of an mri contrast agent (top left), and the same image with the result of an interactive segmentation overlaid in green (3d model of the segmentation on the top right, axial and coronal views at the bottom). segmentation is the process of partitioning an image into different meaningful segments. in medical imaging, these segments often correspond to different tissue classes, organs, pathologies, or other biologically relevant structures. medical image segmentation is made difficult by low contrast, noise, and other imaging ambiguities. although there are many computer vision techniques for image segmentation, some have been adapted specifically for medical image computing. below is a sampling of techniques within this field; the implementation relies on the expertise that clinicians can provide. atlas-based segmentation: for many applications, a clinical expert can manually label several images; segmenting unseen images is a matter of extrapolating from these manually labeled training images. methods of this style are typically referred to as atlas-based segmentation methods. parametric atlas methods typically combine these training images into a single atlas image, while nonparametric atlas methods typically use all of the training images separately. atlas-based methods usually require the use of image registration in order to align the atlas image or images to a new, unseen image. shape-based segmentation: many methods parametrize a template shape for a given structure, often relying on control points along the boundary. the entire shape is then deformed to match a new image. two of the most common shape-based techniques are active shape models and active appearance models. these methods have been very influential, and have given rise to similar models. image-based segmentation: some methods initiate a template and refine its shape according to the image data while minimizing integral error measures, like the active contour model and its variations. interactive segmentation: interactive methods are useful when clinicians can provide some information, such as a seed region or rough outline of the region to segment. an algorithm can then iteratively refine such a segmentation, with or without guidance from the clinician. manual segmentation, using tools such as a paint brush to explicitly define the tissue class of each pixel, remains the gold standard for many imaging applications. recently, principles from feedback control theory have been incorporated into segmentation, which give the user much greater flexibility and allow for the automatic correction of errors. subjective surface segmentation: this method is based on the idea of evolution of segmentation function which is governed by an advection-diffusion model. to segment an object, a segmentation seed is needed (that is the starting point that determines the approximate position of the object in the image). consequently, an initial segmentation function is constructed. the idea behind the subjective surface method is that the position of the seed is the main factor determining the form of this segmentation function. convolutional neural networks (cnn's): the computer-assisted fully automated segmentation performance has been improved due to the advancement of machine learning models. cnn based models such as segnet, unet, resnet, aatsn, transformers and gans have fastened the segmentation process. in the future, such models may replace manual segmentation due to their superior performance and speed. however, there are some other classification of image segmentation methods which are similar to above categories. moreover, we can classify another group as "hybrid" which is based on combination of methods. registration ct image (left), pet image (center) and overlay of both (right) after correct registration. image registration is a process that searches for the correct alignment of images. in the simplest case, two images are aligned. typically, one image is treated as the target image and the other is treated as a source image; the source image is transformed to match the target image. the optimization procedure updates the transformation of the source image based on a similarity value that evaluates the current quality of the alignment. this iterative procedure is repeated until a (local) optimum is found. an example is the registration of ct and pet images to combine structural and metabolic information (see figure). image registration is used in a variety of medical applications: studying temporal changes. longitudinal studies acquire images over several months or years to study long-term processes, such as disease progression. time series correspond to images acquired within the same session (seconds or minutes). they can be used to study cognitive processes, heart deformations and respiration. combining complementary information from different imaging modalities. an example is the fusion of anatomical and functional information. since the size and shape of structures vary across modalities, it is more challenging to evaluate the alignment quality. this has led to the use of similarity measures such as mutual information. characterizing a population of subjects. in contrast to intra-subject registration, a one-to-one mapping may not exist between subjects, depending on the structural variability of the organ of interest. inter-subject registration is required for atlas construction in computational anatomy. here, the objective is to statistically model the anatomy of organs across subjects. computer-assisted surgery. in computer-assisted surgery pre-operative images such as ct or mri are registered to intra-operative images or tracking systems to facilitate image guidance or navigation. there are several important considerations when performing image registration: the transformation model. common choices are rigid, affine, and deformable transformation models. b-spline and thin plate spline models are commonly used for parameterized transformation fields. non-parametric or dense deformation fields carry a displacement vector at every grid location; this necessitates additional regularization constraints. a specific class of deformation fields are diffeomorphisms, which are invertible transformations with a smooth inverse. the similarity metric. a distance or similarity function is used to quantify the registration quality. this similarity can be calculated either on the original images or on features extracted from the images. common similarity measures are sum of squared distances (ssd), correlation coefficient, and mutual information. the choice of similarity measure depends on whether the images are from the same modality; the acquisition noise can also play a role in this decision. for example, ssd is the optimal similarity measure for images of the same modality with gaussian noise. however, the image statistics in ultrasound are significantly different from gaussian noise, leading to the introduction of ultrasound specific similarity measures. multi-modal registration requires a more sophisticated similarity measure; alternatively, a different image representation can be used, such as structural representations or registering adjacent anatomy. a recent study employed contrastive coding to learn shared, dense image representations, referred to as comirs (contrastive multi-modal image representations) which enabled the registration of multi-modal images where existing registration methods often fail due to a lack of sufficiently similar image structures. it reduced the multi-modal registration problem to a mono-modal one, in which general intensity based, as well as feature-based, registration algorithms can be applied. the optimization procedure. either continuous or discrete optimization is performed. for continuous optimization, gradient-based optimization techniques are applied to improve the convergence speed. visualization volume rendering (left), axial cross-section (right top), and sagittal cross-section (right bottom) of a ct image of a subject with multiple nodular lesions (white line) in the lung. visualization plays several key roles in medical image computing. methods from scientific visualization are used to understand and communicate about medical images, which are inherently spatial-temporal. data visualization and data analysis are used on unstructured data forms, for example when evaluating statistical measures derived during algorithmic processing. direct interaction with data, a key feature of the visualization process, is used to perform visual queries about data, annotate images, guide segmentation and registration processes, and control the visual representation of data (by controlling lighting rendering properties and viewing parameters). visualization is used both for initial exploration and for conveying intermediate and final results of analyses. the figure "visualization of medical imaging" illustrates several types of visualization: 1. the display of cross-sections as gray scale images; 2. reformatted views of gray scale images (the sagittal view in this example has a different orientation than the original direction of the image acquisition; and 3. a 3d volume rendering of the same data. the nodular lesion is clearly visible in the different presentations and has been annotated with a white line. atlases medical images can vary significantly across individuals due to people having organs of different shapes and sizes. therefore, representing medical images to account for this variability is crucial. a popular approach to represent medical images is through the use of one or more atlases. here, an atlas refers to a specific model for a population of images with parameters that are learned from a training dataset. the simplest example of an atlas is a mean intensity image, commonly referred to as a template. however, an atlas can also include richer information, such as local image statistics and the probability that a particular spatial location has a certain label. new medical images, which are not used during training, can be mapped to an atlas, which has been tailored to the specific application, such as segmentation and group analysis. mapping an image to an atlas usually involves registering the image and the atlas. this deformation can be used to address variability in medical images. single template the simplest approach is to model medical images as deformed versions of a single template image. for example, anatomical mri brain scans are often mapped to the mni template as to represent all the brain scans in common coordinates. the main drawback of a single-template approach is that if there are significant differences between the template and a given test image, then there may not be a good way to map one onto the other. for example, an anatomical mri brain scan of a patient with severe brain abnormalities (i.e., a tumor or surgical procedure), may not easily map to the mni template. multiple templates rather than relying on a single template, multiple templates can be used. the idea is to represent an image as a deformed version of one of the templates. for example, there could be one template for a healthy population and one template for a diseased population. however, in many applications, it is not clear how many templates are needed. a simple albeit computationally expensive way to deal with this is to have every image in a training dataset be a template image and thus every new image encountered is compared against every image in the training dataset. a more recent approach automatically finds the number of templates needed. statistical analysis statistical methods combine the medical imaging field with modern computer vision, machine learning and pattern recognition. over the last decade, several large datasets have been made publicly available (see for example adni, 1000 functional connectomes project), in part due to collaboration between