In direct imaging at high contrast the bright glare produced by the host star makes the detection and the characterization of substellar companions particularly challenging. In spite of the use of an extreme adaptive optics system combined with a coronagraphic mask to strongly attenuate the starlight contamination dedicated postprocessing methods combining several images recorded with the pupil tracking mode of the telescope are needed to reach the required contrast. In that context we recently proposed to combine the statisticsbased model of PACO with a deep learning approach in a threestep algorithm. First the data are centered and whitened locally using the PACO framework to improve the stationarity and the contrast in a preprocessing step. Second a convolutional neural network CNN is trained in a supervised fashion to detect the signature of synthetic sources in the preprocessed science data. Finally the trained network is applied to the preprocessed observations and delivers a detection map. A second network is trained to infer locally the photometry of detected sources. Both deep models are trained from scratch with a custom data augmentation strategy allowing to generate a large training set from a single spatiotemporospectral dataset. This strategy can be applied to process jointly the images of observations conducted with angular and eventually spectral differential imaging ASDI. In this proceeding we present in a unified framework the key ingredients of the deep PACO algorithm both for ADI and ASDI. We apply our method on several datasets from the the IRDIS imager of the VLTSPHERE instrument. Our method reaches in average a better tradeoff between precision and recall than the comparative algorithms.
['Astrophysics - Instrumentation and Methods for Astrophysics']
Combining statistical learning with deep learning for improved exoplanet detection and characterization
{'No Header': '<!-- image -->', 'Combining statistical learning with deep learning for improved exoplanet detection and characterization': "Olivier Flasseur a , Th'eo Bodrito b , Julien Mairal c , Jean Ponce b,d , Maud Langlois a , and Anne-Marie Lagrange e,f  \na Centre de Recherche Astrophysique de Lyon, CNRS, Univ. de Lyon, Univ. Claude Bernard Lyon 1, ENS de Lyon, France b D'epartement d'Informatique de l' ' Ecole Normale Sup'erieure (ENS-PSL, CNRS, Inria), France c Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France d Courant Institute of Mathematical Sciences, Center for Data Science, New York Univ., USA e Laboratoire d' ' Etudes Spatiales et d'Instrumentation en Astrophysique, Observatoire de Paris, Univ. PSL, Sorbonne Univ., Univ. Paris Diderot, France f Univ. Grenoble Alpes, Institut de Plan'etologie et d'Astrophysique de Grenoble, France", 'ABSTRACT': 'In direct imaging at high contrast, the bright glare produced by the host star makes the detection and the characterization of sub-stellar companions particularly challenging. In spite of the use of an extreme adaptive optics system combined with a coronagraphic mask to strongly attenuate the starlight contamination, dedicated post-processing methods combining several images recorded with the pupil tracking mode of the telescope are needed to reach the required contrast.  \nAmong the large variety of post-processing methods of the field, the PACO algorithm capturing locally the spatial correlations of the data with a multi-variate Gaussian model shown better detection sensitivity than standard methods of the field (e.g., cADI, PCA, TLOCI). However, there is a room for improvement to increase the detection sensitivity due to the approximate fidelity of the statistical model embedded in PACO with respect to the observations.  \nFurther author information: Send correspondence to Olivier Flasseur.  \nE-mail: olivier.flasseur@univ-lyon1.fr, Telephone: +33 (0)4 78 86 83 70.  \nIn that context, we recently proposed to combine the statistics-based model of PACO with a deep learning approach in a three-step algorithm. First, the data are centered and whitened locally using the PACO framework to improve the stationarity and the contrast in a preprocessing step. Second, a convolutional neural network (CNN) is trained in a supervised fashion to detect the signature of synthetic sources in the preprocessed science data. Finally, the trained network is applied to the preprocessed observations and delivers a detection map. A second network is trained to infer locally the photometry of detected sources. Both deep models are trained from scratch with a custom data augmentation strategy allowing to generate a large training set from a single spatio-temporo-spectral dataset. This strategy can be applied to process jointly the images of observations conducted with angular, and eventually spectral, differential imaging (A(S)DI). In this proceeding, we present in a unified framework the key ingredients of the deep PACO algorithm both for ADI and ASDI.  \nWe apply our method on several datasets from the the IRDIS imager of the VLT/SPHERE instrument. Our method reaches, in average, a better trade-off between precision and recall than the comparative algorithms.  \nKeywords: High angular resolution - techniques: image processing - methods: numerical - methods: statistical - methods: data analysis.', '1. INTRODUCTION': 'The future thirty meters class telescopes (e.g., ELT, GMT, TMT) will enable exploring much deeper the inner environment of nearby solar-type stars than existing facilities. This goal raises several challenges from a data science point of view, including: (i) approaching the ultimate performance of the instruments by an optimal extraction of the signals of the sought objects, (ii) capturing a highly spatially structured nuisance component subject to strong temporal fluctuations, and (iii) building a model of the nuisance component from several datasets to bypass the limits of ADI at very short angular separations. In that context, data-driven approaches combining statistical modeling with deep learning could be highly valuable to deal with the complexity of high-contrast observations.  \nIn previous works, we have developed the PACO algorithm [8, 5, 7, 11, 10] dedicated to the post-processing of A(S)DI observations [14, 16] for exoplanet detection and characterization by direct imaging at high contrast. PACO captures locally the spatial correlations of the nuisance component (i.e., speckles plus other sources of noise) with a scaled mixture of multi-variate Gaussian models. Its parameters are estimated in a data-driven fashion at the scale of a patch of a few tens of pixels. PACO delivers reliable detection confidences with an improved sensitivity with respect to the classical processing methods of the field (e.g., cADI [14, 12], PCA [18, 1], TLOCI [15]). However, it remains room for improvement, especially at short angular separations.  \nVery recently, we proposed a new algorithm, dubbed deep PACO [9, 6], improving the detection sensitivity of PACO. The proposed method combines the statistics-based model of PACO with a deep learning model in a three steps procedure. First, the data are centered and whitened locally using the PACO framework to improve the stationarity and the contrast in a pre-processing step. Second, a convolutional neural network is trained from scratch, in a supervised fashion, to detect the signature of synthetic sources in the pre-processed science data. Finally, the trained network is applied to the pre-processed observations and delivers a detection map. Photometry of detected sources can be estimated by a second deep neural network. Both models are trained from scratch with a custom data augmentation strategy allowing to generate large training sets from a single spatio-temporo-spectral dataset.  \nWe review the main ingredients of the deep PACO algorithm in Sect. 2. Section 3 presents comparative results obtained on VLT/SPHERE data. Finally, Sect. 4 draws our main conclusions and briefly discusses some future research directions.', '2.1 Direct Model of the Observations': 'A stack r ∈ R N × T × L of high-contrast images is composed of N -pixels frames, recorded at times t ∈ J 1; T K and in spectral channels ℓ ∈ J 1; L K . The contribution r ℓ ∈ R N × T writes:  \nr ℓ = f ℓ + P ∑ p =1 α p,ℓ h ℓ ( ϕ p ) , (1)  \nwhere f ℓ ∈ R N × T and h ℓ ∈ R N × T are respectively the contribution of the nuisance component (i.e., speckles and other additive sources of noise) and of any point-like source taking the form of the off-axis point-spread function (PSF), at spectral channel ℓ . The contribution of a source p ∈ J 1; P K is centered at location F t,ℓ ( ϕ p ) in the t -th image of the ℓ -th channel, where ϕ p is its initial location on an image at a time t ref and spectral channel λ ref of reference. F t,ℓ is a deterministic function accounting for the apparent motion of the sought sources induced by ASDI.', '2.2 Statistical Modeling of the Nuisance Component': 'As in [10], we model the fluctuations of the nuisance component f by a statistical model whose parameters are estimated locally (in this work, at a scale of non-overlapping square patches of K pixels). The locality of the model allows to account for the non-stationarity of the nuisance. We note P the set of pixel locations on which the parameters of the statistical model should be evaluated. The distribution of a patch of nuisance f n,t,ℓ centered at pixel n ∈ P , at time t , and channel ℓ is modeled by a scaled multi-variate Gaussian: N ( m n,ℓ , σ 2 n,t,ℓ C n ). The sample estimators { ̂ m n,ℓ , ̂ σ 2 n,t,ℓ , ̂ S n } of { m n,ℓ , σ 2 n,t,ℓ , C n } are obtained from the T L patches r n ∈ R K × T × L through the maximum-likelihood:  \n         ̂ m n,ℓ = ( ∑ t ̂ σ -2 n,t,ℓ ) -1 ∑ t ̂ σ -2 n,t,ℓ r n,t,ℓ , ̂ S n = 1 T L ∑ t,ℓ ̂ σ -2 n,t,ℓ ( r n,t,ℓ -̂ m n,ℓ )( r n,t,ℓ -̂ m n,ℓ ) ⊤ , ̂ σ 2 n,t,ℓ = K -1 ( r n,t,ℓ -̂ m n,ℓ ) ⊤ ̂ S -1 n ( r n,t,ℓ -̂ m n,ℓ ) , (2)  \nThe number of samples involved in the computation of ̂ S n being lower than the number K of pixels in a patch, the sample covariance ̂ S n is very noisy. As in [10], we regularize it by shrinkage [13, 4] to form ̂ C n as follows:  \n̂ C n = (1 -̂ ρ n ) ̂ S n + ̂ ρ n ̂ F n , (3)  \nwhere ̂ F n is a diagonal matrix with only the sample variances. The weight ̂ ρ n sets a bias-variance trade-off and it is estimated in data-driven fashion:  \n̂ ρ n = tr ( ̂ S 2 n ) +tr 2 ( ̂ S n ) -2 ∑ K k =1 [ ̂ S n ] 2 kk ( Q +1) ( tr ( ̂ S 2 n ) -∑ K k =1 [ ̂ S n ] 2 kk ) , (4)  \nwith Q = ( ∑ t,ℓ ̂ σ -2 n,t,ℓ ) 2 / ( ∑ t,ℓ ̂ σ -4 n,t,ℓ ) the equivalent number of patches involved in the computation of ̂ C n in the presence of the scaling factors { ̂ σ 2 n,t,ℓ } t =1: T,ℓ =1: L . Given the statistics of the nuisance, the dataset r is preprocessed by centering and whitening to form ˜ r ∈ R N × T × L with attenuated spatial structures:  \n˜ r n,t,ℓ = W n,t,ℓ r n,t,ℓ = ̂ σ -1 n,t,ℓ ̂ L ⊤ n ( r n,t,ℓ -̂ m n,ℓ ) , (5)  \nsuch that ̂ L n ̂ L ⊤ n = ̂ C -1 n .', '2.3 Detection by Supervised Deep Learning': 'We aim to produce a detection map ̂ y in [0; 1] M , where each pixel-value represents the pseudo-probability that a source is centered at that location at time t ref . In order to perform supervised training of our model, we generate S pairs { ( r [ s ] ; y [ s ] } s =1: S of synthetic samples/ground truths, resulting from the massive injection of point-like sources. The trained model is data-dependent, i.e. it differs for each A(S)DI dataset due to the high variability of the nuisance component from one observation to the other. We thus resort to data-augmentation to generate a large training basis from a single dataset: we apply a random permutation of the T images of each spectral channel for each training sample s . Synthetic sources are injected inside the temporally permuted data following the direct model (1) to form the intermediate datasets { s r ∈ R N × T × L } s =1: S :  \ns r [ s ] ℓ = P [ s ] f ℓ + P [ s ] ∑ p =1 α [ s ] p,ℓ h ℓ ( ϕ [ s ] p ) , (6)  \nwith P the operator performing a random permutation of the temporal frames. For each synthetic source p and if L > 2, we select randomly a synthetic SED α p ∈ R L from a custom library of 10,000 sub-stellar spectra generated with ExoREM [3]. Before injection of synthetic sources, the initial dataset r is pre-processed to form ˜ r . After injection of synthetic sources, the set S [ s ] of locations impacted by the signal of the fiducial sources is determined. Outside S [ s ] , the pre-processed images are obtained from the temporal permutation of ˜ r . Inside S [ s ] , the statistics of the nuisance and the pre-processed images are updated given the contamination of the P [ s ] injected sources to form { q r ∈ R N × T × L } s =1: S such that:  \nq r [ s ] n,t,ℓ = { W n,t,ℓ s r [ s ] n,t,ℓ , for n ∈ S [ s ] ∩ P , P [ s ] ˜ r n,t,ℓ , for n ∈ P -S [ s ] ∩ P . (7)  \nFinally, the apparent motion of the injected sources are compensated to co-align their signals within { ( r [ s ] ∈ R N × T × L } s =1: S :  \n( r [ s ] t,ℓ = D t,ℓ q r [ s ] t,ℓ , (8)  \nwith D t,ℓ the operator performing a rotation of the image at time t and spectral channel ℓ by the opposite of the parallactic angle at time t .  \nAt training time, the network parameters are optimized by minimizing the Dice2 loss [19]:  \nL [ s ] = 1 -∑ m y [ s ] m ̂ y [ s ] m + ϵ ∑ m y [ s ] m + ̂ y [ s ] m + ϵ -∑ m (1 -y [ s ] m )(1 -̂ y [ s ] m + ϵ ) ∑ m 2 -y [ s ] m -̂ y [ s ] m + ϵ , (9)  \nwhere { y [ s ] ; ̂ y [ s ] } is a set of ground truth and predicted detection maps, and ϵ is a small stability parameter. At validation time, we compute from a predicted detection map ̂ y [ s ] ∈ [0; 1] M thresholded at τ ∈ [0; 1], the true positive rate (TPR) and the false discovery rate (FDR). From these two quantities, receiver operating curves (ROCs) are built and the area under ROCs (AUC) is derived as an overall measurement of the trade-off between precision and recall.  \nConcerning the architecture, we use a U-Net with a ResNet18 as encoder backbone ∗ . The network weights are optimized from scratch with AMSGrad. The number of epochs, the number of samples per epoch, the batch size, the weight decay, and the learning rate are discussed in details in [9, 6].', '3. RESULTS ON VLT/SPHERE DATA': 'For our experiments, we have selected three datasets from the VLT/SPHERE-IRDIS imager [2]. The performance of deep PACO are compared with the cADI, PCA, and PACO algorithms. The details about the observing  \nFigure 1. Detection maps obtained with cADI, PCA, PACO, and deep PACO on three datasets (HIP 88399 -2015-05-10-, HD 95086 -2021-03-11-, HD 95086 -2015-05-05-) from the VLT/SPHERE-IRDIS instrument.  \n<!-- image -->  \nparameters, the pre-reduction of the recorded datasets, and the setting of the comparative algorithms are discussed in [9, 6].  \nFigure 1 shows the detection maps obtained with the comparative algorithms. The detected peaks are classified as true, missed, or false detections based on a thresholding of the output map at τ = 5 for the algorithms producing a signal-to-noise map (i.e., cADI, PCA, PACO), and at τ = 0 . 5 for deep PACO producing a pseudo-probability map. For the four considered algorithms, the results are obtained by processing the first spectral channel, i.e. exploiting solely ADI. For the proposed approach, we compare ADI and ASDI by also considering a joint processing of the two spectral channels of the dual band observations. These results illustrate the ability of deep PACO to reach, in average, a better detection sensitivity and, at the same time, to avoid some spurious false alarms produced by the comparative algorithms. The joint spectral processing allows to further improve the trade-off between precision and recall.', '4. CONCLUSION': 'As a proof of concept, we applied our method on several datasets from the VLT/SPHERE-IRDIS instrument. We compared the deep PACO method against some state-of-the-art algorithms of the field, including PACO. With ADI, our method reaches in average a better trade-off between precision and recall than the comparative algorithms. A joint processing of spatio-temporo-spectral observations obtained with ASDI allows to further improve the detection performance.  \nOn the methodological side, we are currently investigating ways to improve the deep learning stage of the proposed algorithm by jointly modeling the typical distribution of the nuisance component from multiple datasets. This methodology would strongly differs from reference differential imaging (RDI, [17]) in the sense that a  \nhighly non-linear model would be learned from the archived observations and would exploit several prior domain knowledge. On the application side, we will consider to adapt and to apply the deep PACO algorithm on simulated ELT high-contrast data (e.g., from ELT/HARMONI) to ground its benefits in the context of the future giant telescopes.', 'ACKNOWLEDGMENTS': "This work is supported by the French National Programs (PNP and PNPS) and by the Action Sp'ecifique Haute R'esolution Angulaire (ASHRA) of CNRS/INSU co-funded by CNES. This project is also supported in part by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (COBREX; grant agreement n ° 885593). The work of TB and JP was supported in part by the Inria/NYU collaboration, the Louis Vuitton/ENS chair on artificial intelligence and the French government under management of Agence Nationale de la Recherche as part of the Investissements d'avenir program, reference ANR19-P3IA0001 (PRAIRIE 3IA Institute). The work of JM was supported in part by the ERC grant number 714381 (SOLARIS project) and by ANR 3IA MIAI@Grenoble-Alpes (ANR-19-P3IA-0003).  \nThe raw data used in this article are freely available on the ESO archive facility at http://archive.eso.org/eso/eso archive main.html. They were pre-reduced with the SPHERE Data Centre, jointly operated by OSUG/IPAG (Grenoble), PYTHEAS/LAM/CESAM (Marseille), OCA/Lagrange (Nice), Observatoire de Paris/LESIA (Paris), and Observatoire de Lyon/CRAL (Lyon, France). This work was also granted access to the HPC resources of IDRIS under the allocation 2022-AD011013643 made by GENCI.", 'References': "- [1] Adam Amara and Sascha P Quanz. 'PYNPOINT: an image processing package for finding exoplanets'. In: Monthly Notices of the Royal Astronomical Society 427.2 (2012), pp. 948-955.\n- [2] J-L Beuzit et al. 'SPHERE: the exoplanet imager for the Very Large Telescope'. In: Astronomy & Astrophysics 631 (2019), A155.\n- [3] Benjamin Charnay et al. 'A self-consistent cloud model for brown dwarfs and young giant exoplanets: comparison with photometric and spectroscopic observations'. In: The Astrophysical Journal 854.2 (2018), p. 172.\n- [4] Yilun Chen et al. 'Shrinkage algorithms for MMSE covariance estimation'. In: IEEE Transactions on Signal Processing 58.10 (2010), pp. 5016-5029.\n- [5] Olivier Flasseur et al. 'An unsupervised patch-based approach for exoplanet detection by direct imaging'. In: IEEE International Conference on Image Processing . 2018, pp. 2735-2739.\n- [6] Olivier Flasseur et al. 'deep PACO: Combining statistical models with deep learning for exoplanet detection and characterization in direct imaging at high contrast'. In: under minor review in Monthly Notices of the Royal Astronomical Society (2023).\n- [7] Olivier Flasseur et al. 'Exoplanet detection in angular and spectral differential imaging: local learning of background correlations for improved detections'. In: SPIE Astronomical Telescopes + Instrumentation . Vol. 10703. International Society for Optics and Photonics. 2018, 107032R.\n- [8] Olivier Flasseur et al. 'Exoplanet detection in angular differential imaging by statistical learning of the nonstationary patch covariances - The PACO algorithm'. In: Astronomy & Astrophysics 618 (2018), A138.\n- [9] Olivier Flasseur et al. 'Exoplanet detection in angular differential imaging: combining a statistics-based learning with a deep-based learning for improved detections'. In: Adaptive Optics Systems VIII . Vol. 12185. SPIE. 2022, pp. 1154-1167.\n- [10] Olivier Flasseur et al. 'PACO ASDI: an algorithm for exoplanet detection and characterization in direct imaging with integral field spectrographs'. In: Astronomy & Astrophysics 637 (2020), A9.\n- [11] Olivier Flasseur et al. 'Robustness to bad frames in angular differential imaging: a local weighting approach'. In: Astronomy & Astrophysics 634 (2020), A2."}