ims - microscopy fusion is demonstrated through three different applications : ( i ) predicting ion distributions to a spatial resolution that exceeds that of the measured ion images ( sharpening ) ; ( ii ) predicting ion distributions in tissue areas that were not measured by ims ( out - of - sample prediction ) ; and ( iii ) discovery of biological patterns that are difficult to retrieve from the source modalities separately ( enrichment ) .
the results cover seven distinct case studies ( supplementary table 1 ) , each describing a multi - modal tissue imaging experiment with microscopy and maldi ims from the same or an adjacent tissue section .
the sharpening of ion images provides a means of predicting molecular tissue content to a higher spatial resolution and introduces the general modeling procedure that underlies all our fusion applications . in case study 1 , the tissue distribution of ion m / z 762.5 ( identified as pe(16:0/22:6 ) ) is measured through an ims experiment performed at 100 m spatial resolution ( fig .
3b ) is taken from the same tissue section stained with h&e after the ims acquisition .
our two - phase fusion method ( see online methods and supplementary figs . 1 , 2 , and
3 ) combines the information from both modalities , builds a model to capture any cross - modality relationships it can detect , and employs that model and the microscopy measurements to predict the distribution for all ions at 10 m resolution . the spatial prediction for ion m / z 762.5
additionally , we compare the 100 m measurement and the fusion - based 10 m prediction to an ion image for m / z 762.5 actually acquired at 10 m on an adjacent tissue section ( fig .
the ims acquisition parameters are identical for ( a ) and ( d ) , except for increased laser power required to compensate for reduced signal intensities at 10 m pixel widths .
an example by an alternative up - sampling approach , bilinear interpolation ( supplementary fig .
the online methods provide details on the model building and evaluation process ( supplementary figs . 5 and 6 ) . the ims - microscopy model obtained by pls regression between ims and microscopy - derived variables , contains a sub - model for each ims variable ( supplementary fig .
model performance is therefore specific to each ims variable , and needs to be evaluated to ascertain for which ion peaks good prediction ( and useful fusion ) is possible .
this percentage score summarizes how well the measured ion distribution can be predicted using the given microscopy observations .
a value of 100% indicates that the cross - modality relationships allow complete reconstruction of the measured intensity distribution using only variables from the other technology .
a value close to 0% signifies that no cross - modality relationship could be found or modeled for this ion , and that fusion - driven prediction is not an option for it .
most measured ion images can be predicted at least partially using microscopy , and in our examples we rely on fusion only for ions with a score above 75% .
we additionally introduce the absolute residuals image and the 95% confidence interval image as measures of prediction performance in function of tissue location ( supplementary fig . 6 ) . while m / z 762.5 and 747.5 ( a combination of three nominally isobaric lipids , supplementary fig .
9 ) exhibit a strong cross - modality relationship to h&e microscopy , m / z 766.5 ( pe(18:0/20:4 ) ) and 715.6 ( pe - cer(d16:1/22:0 ) ) from the same ims experiment are examples of ions for which no such relationships could be found and thus prediction is not recommended ( supplementary figs . 7 and 8) .
the access to information from different technologies allows fusion to predict beyond hard constraints inherent to a particular sensor type .
this advantage , for example , enables the image sharpening application to push beyond the physical limits of the ims laser by predicting at a spatial resolution below the wavelength of the laser . as an example
, we apply the fusion procedure to a mouse brain sample on which both source modalities have been pushed to their current state - of - the - art in spatial resolution for the instrumentation used in this experiment ( fig .
/ z 646.4 ( nominal isobars cer(d18:1/24:1 ) and second c isotope of cerp(d18:1/18:0 ) ) and 788.5 ( nominal isobars ps(18:0/18:1 ) and pe(40:7 ) ) were successfully predicted with 75% and 76% scores at 330 nm resolution ( fig .
4c ) , using a 355 nm wavelength laser to acquire the source ims measurements .
4a ) , while the textural information was encoded at 330 nm resolution in an h&e stained microscopy image of an adjacent section ( fig .
additional results in the supplementary information demonstrate how fusion can predict to any spatial resolution between the native ims and microscopy resolutions ( supplementary figs .
it is also independent of the tissue type used or the molecule type measured ( supplementary figs . 12 and 18 ) , and the method is shown to operate with tissue stains other than h&e ( supplementary figs . 1316 , 17 ) .
fusion is shown to provide useful integration and prediction even for ion peaks that report multiple unresolved ion species ( supplementary fig .
the behavior of the method is verified against a gold standard of known cross - modal and modality - specific patterns ( supplementary fig .
this application of the fusion process develops the capability of predicting ion distributions in areas not measured by ims , using a model built from areas where data from both modalities are available . as an example
, the ion distribution for m / z 10,516 is predicted in non - ims measured mouse brain areas ( fig .
first , an ims - microscopy model is built on a sub - area of the tissue where ims measurements ( in the mw range of interest ) are available at 100 m resolution and h&e - stained microscopy at 5 m resolution .
subsequently , the model is used to predict both inside the ims - measured area as well as the area outside , where only microscopy is available .
the predicted pattern for m / z 10,516 outside the modeled area was successfully confirmed via ims measurements external to this case study . although prediction at the native ims resolution of 100 m is possible ( supplementary fig .
21 ) , the availability of microscopy at 5 m resolution enables additional sharpening of the ion predictions to 5 m both inside and outside the modeled area ( fig . 5 ) .
the discovery of cross - modality relationships enables the separation of a measured tissue signal into a part that can be predicted by the other technology and a part that is modality - specific .
for measured signals with strong cross - modal support , this information can be used to filter technology - specific noise from genuine tissue signal . this process can substantially increase the signal - to - noise ratio of observations , and enables discovery of patterns that might otherwise be missed . each ion image measured by ims
can be considered a superposition of noise variation on top of a biological or sample signal pattern .
the fusion model tries to write each ion image as a linear combination of microscopy - derived patterns , and for ion images with strong cross - modal support , the model succeeds in reproducing most if not all of the biological or sample signal pattern that way .
since the noise tends to be sensor - specific , it is highly unlikely that fusion finds a link to an identical pattern in the other technology . as a result ,
modality - specific noise variation will often not survive the modeling and prediction process , effectively de - noising the predictions . since the biological signal often reports an underlying tissue structure that can modulate measurements in both modalities , finding a relationship across technologies is not uncommon and when that happens the presence of real tissue variation in the predictions is more readily observed .
it is not necessarily so that all technology - specific variation is noise , since a measurement might be reporting a biological feature that can only be detected by one of the sensors .
however , that situation can be detected by a reduced reconstruction score and can even be spatially confined to a tissue sub - area using for example the absolute residuals image . for measured variables and tissue areas with strong cross - modal support , fusion enables enrichment of genuine tissue signal and removal of modality - specific noise .
multi - modal enrichment can cause a feature to be discovered or recognized as biological or sample - based when in a single - modality analysis it would be mistaken for noise or go unnoticed .
propagation through the fusion process into the predictions indicates that these features are corroborated by measurements in both modalities . from a microscopy viewpoint ,
, fusion provides an increased confidence in the genuine nature of these features , so that they may be recognized from faint signals in the noise .
fusion separates features with support across technologies from those supported by only a single sensor type .
it shows a localized intensity drop , which might be perceived as biological since it covers multiple pixels .
however , in microscopy ( acquired post - ims ) there is no visual trace at this location and even the fusion process finds no cross - modal evidence to support this drop as a real tissue feature . although lack of cross - modal corroboration does not necessarily mean a feature is noise , since it can be a biological feature detectable by only one of the technologies , it does reduce confidence in it being a genuine tissue feature and suggests here an ims acquisition anomaly .
fusion - based filtering can also be used as a global method for de - noising ion images with strong cross - modal support . as an example , an ion image for m / z 5,666 measured at 100 m resolution in mouse brain is fused with an h&e - stained microscopy image at 5 m resolution ( supplementary fig . 20 ) .
a similar cleanup effect and increase in s / n is also observed in predictions for other ions from this data set ( supplementary figs .
the sharpening of ion images provides a means of predicting molecular tissue content to a higher spatial resolution and introduces the general modeling procedure that underlies all our fusion applications . in case study 1 , the tissue distribution of ion m / z 762.5 ( identified as pe(16:0/22:6 ) ) is measured through an ims experiment performed at 100 m spatial resolution ( fig .
3b ) is taken from the same tissue section stained with h&e after the ims acquisition .
our two - phase fusion method ( see online methods and supplementary figs . 1 , 2 , and 3 ) combines the information from both modalities , builds a model to capture any cross - modality relationships it can detect , and employs that model and the microscopy measurements to predict the distribution for all ions at 10 m resolution . the spatial prediction for ion m / z 762.5
3c ) . additionally , we compare the 100 m measurement and the fusion - based 10 m prediction to an ion image for m / z 762.5 actually acquired at 10 m on an adjacent tissue section ( fig .
3d ) . the ims acquisition parameters are identical for ( a ) and ( d ) , except for increased laser power required to compensate for reduced signal intensities at 10 m pixel widths .
an example by an alternative up - sampling approach , bilinear interpolation ( supplementary fig .
the online methods provide details on the model building and evaluation process ( supplementary figs . 5 and 6 ) . the ims - microscopy model obtained by pls regression between ims and microscopy - derived variables , contains a sub - model for each ims variable ( supplementary fig .
model performance is therefore specific to each ims variable , and needs to be evaluated to ascertain for which ion peaks good prediction ( and useful fusion ) is possible .
this percentage score summarizes how well the measured ion distribution can be predicted using the given microscopy observations .
a value of 100% indicates that the cross - modality relationships allow complete reconstruction of the measured intensity distribution using only variables from the other technology . a value close to 0% signifies that no cross - modality relationship could be found or modeled for this ion , and that fusion - driven prediction is not an option for it .
most measured ion images can be predicted at least partially using microscopy , and in our examples we rely on fusion only for ions with a score above 75% .
we additionally introduce the absolute residuals image and the 95% confidence interval image as measures of prediction performance in function of tissue location ( supplementary fig .
6 ) . while m / z 762.5 and 747.5 ( a combination of three nominally isobaric lipids , supplementary fig .
9 ) exhibit a strong cross - modality relationship to h&e microscopy , m / z 766.5 ( pe(18:0/20:4 ) ) and 715.6 ( pe - cer(d16:1/22:0 ) ) from the same ims experiment are examples of ions for which no such relationships could be found and thus prediction is not recommended ( supplementary figs . 7 and 8) .
the access to information from different technologies allows fusion to predict beyond hard constraints inherent to a particular sensor type .
this advantage , for example , enables the image sharpening application to push beyond the physical limits of the ims laser by predicting at a spatial resolution below the wavelength of the laser .
as an example , we apply the fusion procedure to a mouse brain sample on which both source modalities have been pushed to their current state - of - the - art in spatial resolution for the instrumentation used in this experiment ( fig .
/ z 646.4 ( nominal isobars cer(d18:1/24:1 ) and second c isotope of cerp(d18:1/18:0 ) ) and 788.5 ( nominal isobars ps(18:0/18:1 ) and pe(40:7 ) ) were successfully predicted with 75% and 76% scores at 330 nm resolution ( fig .
4c ) , using a 355 nm wavelength laser to acquire the source ims measurements .
4a ) , while the textural information was encoded at 330 nm resolution in an h&e stained microscopy image of an adjacent section ( fig .
additional results in the supplementary information demonstrate how fusion can predict to any spatial resolution between the native ims and microscopy resolutions ( supplementary figs .
it is also independent of the tissue type used or the molecule type measured ( supplementary figs . 12 and 18 ) , and the method is shown to operate with tissue stains other than h&e ( supplementary figs . 1316 , 17 ) .
fusion is shown to provide useful integration and prediction even for ion peaks that report multiple unresolved ion species ( supplementary fig .
finally , using a synthetic multi - modal data set , the behavior of the method is verified against a gold standard of known cross - modal and modality - specific patterns ( supplementary fig .
this application of the fusion process develops the capability of predicting ion distributions in areas not measured by ims , using a model built from areas where data from both modalities are available . as an example
, the ion distribution for m / z 10,516 is predicted in non - ims measured mouse brain areas ( fig .
first , an ims - microscopy model is built on a sub - area of the tissue where ims measurements ( in the mw range of interest ) are available at 100 m resolution and h&e - stained microscopy at 5 m resolution .
subsequently , the model is used to predict both inside the ims - measured area as well as the area outside , where only microscopy is available .
the predicted pattern for m / z 10,516 outside the modeled area was successfully confirmed via ims measurements external to this case study .
although prediction at the native ims resolution of 100 m is possible ( supplementary fig .
21 ) , the availability of microscopy at 5 m resolution enables additional sharpening of the ion predictions to 5 m both inside and outside the modeled area ( fig .
the discovery of cross - modality relationships enables the separation of a measured tissue signal into a part that can be predicted by the other technology and a part that is modality - specific . for measured signals with strong cross - modal support
, this information can be used to filter technology - specific noise from genuine tissue signal .
this process can substantially increase the signal - to - noise ratio of observations , and enables discovery of patterns that might otherwise be missed . each ion image measured by ims
can be considered a superposition of noise variation on top of a biological or sample signal pattern .
the fusion model tries to write each ion image as a linear combination of microscopy - derived patterns , and for ion images with strong cross - modal support , the model succeeds in reproducing most if not all of the biological or sample signal pattern that way .
since the noise tends to be sensor - specific , it is highly unlikely that fusion finds a link to an identical pattern in the other technology . as a result ,
modality - specific noise variation will often not survive the modeling and prediction process , effectively de - noising the predictions . since the biological signal often reports an underlying tissue structure that can modulate measurements in both modalities , finding a relationship across technologies is not uncommon and when that happens the presence of real tissue variation in the predictions is more readily observed .
it is not necessarily so that all technology - specific variation is noise , since a measurement might be reporting a biological feature that can only be detected by one of the sensors .
however , that situation can be detected by a reduced reconstruction score and can even be spatially confined to a tissue sub - area using for example the absolute residuals image . for measured variables and tissue areas with strong cross - modal support , fusion enables enrichment of genuine tissue signal and removal of modality - specific noise .
multi - modal enrichment can cause a feature to be discovered or recognized as biological or sample - based when in a single - modality analysis it would be mistaken for noise or go unnoticed .
propagation through the fusion process into the predictions indicates that these features are corroborated by measurements in both modalities . from a microscopy viewpoint ,
, fusion provides an increased confidence in the genuine nature of these features , so that they may be recognized from faint signals in the noise .
fusion separates features with support across technologies from those supported by only a single sensor type .
it shows a localized intensity drop , which might be perceived as biological since it covers multiple pixels .
however , in microscopy ( acquired post - ims ) there is no visual trace at this location and even the fusion process finds no cross - modal evidence to support this drop as a real tissue feature . although lack of cross - modal corroboration does not necessarily mean a feature is noise , since it can be a biological feature detectable by only one of the technologies , it does reduce confidence in it being a genuine tissue feature and suggests here an ims acquisition anomaly .
fusion - based filtering can also be used as a global method for de - noising ion images with strong cross - modal support . as an example , an ion image for m / z 5,666 measured at 100 m resolution in mouse brain
is fused with an h&e - stained microscopy image at 5 m resolution ( supplementary fig .
a similar cleanup effect and increase in s / n is also observed in predictions for other ions from this data set ( supplementary figs .
the image fusion process combines different image types , each measuring a different aspect of the content of a tissue sample , and predicts the tissue content in an integrated way as if all aspects were observed concurrently . modeling between technologies
since this report examines ims - microscopy fusion , ims - derived variables ( peak intensities ) are designated response variables and microscopy - derived variables ( e.g. hue , texture , entropy ) are used as predictor variables .
the resulting mathematical model attempts to approximate each ion peak distribution in the ims source as a linear combination of microscopy - derived patterns .
since each ims variable has its own relationship to the microscopy patterns , the fusion model contains a sub - model for each variable ( supplementary fig .
5 ) and predictive performance will be variable - specific . building the best possible model
does not ensure a good prediction in all cases , since there may not be a cross - modality relationship to exploit for a particular peak , or the complexity of the relationship can not be described accurately using the present model .
it is therefore imperative that the model - building step is followed by an evaluation step that assesses for each m / z its prediction reliability .
the first indicator is the reconstruction score , which reports how close a prediction using microscopy measurements gets to the measured ion distribution , given the model .
the second indicator is the absolute residuals image , which shows where in the tissue microscopy - driven prediction approximates the ims measurements well , and where it does not .
a third indicator conveys prediction robustness as a function of location by calculating a 95% confidence interval ( ci ) image using bootstrapping ( see online methods references 41 and 42 ) .
this ci - image highlights in which tissue areas the prediction is more robust and in which areas the prediction might be model - specific and less trustworthy .
the prediction of molecular distributions in tissue areas for which no ims is available is a form of out - of - sample prediction ( fig .
5 ) . in molecular imaging studies where comprehensive measurement of the entire tissue sample is not practical or economical , or in studies where the number of samples is very large
it can also provide an evidence - supported view into the probable content of tissue areas or samples that need to be saved for other analysis techniques , e.g. in multi - branched drug discovery workflows .
anomaly detection , where the measured content of a diseased tissue sample is directly compared to the content predicted to be there on the basis of a model trained on normal tissue . in high - dimensional data sets
, this would immediately highlight differences and provide a facile path to pathology - derived anomalies .
however , it is important to keep the conditions for the ( microscopy ) measurements between these different areas as similar as possible and to keep the model from over - fitting on the training data .
fusion - based predictions show an enrichment of patterns supported by multiple technologies and an attenuation of modality - specific patterns .
the enrichment effect is inherently present in fusion - based predictions and can be used to increase confidence in observations by cross - modal corroboration ( supplementary figs .
the ability of fusion to aid in separating true signal variation from instrumental noise variation by integrating with another data source , has immediate value for increasing measurement sensitivity and s / n without the need to physically adjust the instrument .
the modeling aspects that influence the reliability of fusion - driven predictions are addressed in the online methods .
a subsequent section specifically highlights how the empirical nature of the image fusion method , namely mining for cross - modality relationships rather than pre - defining them , ensures broad applicability to imaging modalities and sensor types beyond the data sources illustrated here .
we also shortly discuss how the fusion process enables bi - directional communication and corroboration between different data sources , and how fusion with ims can contribute to microscopy interpretation and the pathological recognition process .
furthermore , to enable the readers to experiment on their own data sets , we provide a full implementation of the image fusion framework as a command line utility that can be downloaded at http://fusion.vueinnovations.com .
this package includes an example ims - microscopy multi - modal data set . in conclusion ,
image fusion enables the creation of novel predictive imaging modalities that combine the advantages of different sensor types to deliver insights that can not be normally obtained from the separate technologies alone .
the modeling of cross - modality relationships provides predictive paths to new biological understanding through the integration of observations from different measurement principles .
it also allows these fused modalities to circumvent sensor - specific limitations ( e.g. ims resolution via sharpening ) , to attenuate technology - specific noise sources ( e.g. matrix artifact attenuation by biological signal enrichment ) , and to predict observations in the absence of measurements ( e.g. prediction of ion intensity in non - ims measured areas ) . the predictive applications of image fusion can contribute in those cases where conducting a physical measurement is unpractical ( e.g. measurement time , instrument stability ) , uneconomical ( e.g. laser wear , detector deterioration ) , or even not feasible due to instrumental limitations ( e.g. spatial resolution beyond laser capabilities , low s / n ) .
multi - modal studies have become widespread and the complementarity between different technologies is well appreciated .
this study shows that the fusion of microscopy and mass spectrometry can provide remarkable results , and lays the groundwork for more advanced modeling across technologies .
these prediction methods can fulfill an instrumental role in realizing the full potential of multi - modal pattern discovery , particularly in the molecular mapping of tissue .
 a new predictive imaging modality is created through the  fusion of two distinct technologies : imaging mass spectrometry ( ims ) and microscopy . 
 ims - generated molecular maps , rich in chemical information but having coarse spatial resolution , are combined with optical microscopy maps , which have relatively low chemical specificity but high spatial information . 
 the resulting images combine the advantages of both technologies , enabling prediction of a molecular distribution both at high spatial resolution and with high chemical specificity . 
 multivariate regression is used to model variables in one technology , using variables from the other technology . 
 several applications demonstrate the remarkable potential of image fusion : ( i )  sharpening of ims images , which uses microscopy measurements to predict ion distributions at a spatial resolution that exceeds that of measured ion images by ten times or more ; ( ii ) prediction of ion distributions in tissue areas that were not measured by ims ; and ( iii ) enrichment of biological signals and attenuation of instrumental artifacts , revealing insights that are not easily extracted from either microscopy or ims separately . 
 image fusion enables a new multi - modality paradigm for tissue exploration whereby mining relationships between different imaging sensors yields novel imaging modalities that combine and surpass what can be gleaned from the individual technologies alone . 
Results
Sharpening ion distributions by microscopy
Predicting molecule distributions in non-IMS measured tissue
Discovery through multi-modal enrichment
Discussion
Supplementary Material