# Improved period variations of 32 contact binaries with rapidly decreasing
periods
in the Galactic Bulge

Kyeongsoo Hong Korea Astronomy and Space Science Institute, Daejeon 34055,
Republic of Korea  Jae Woo Lee Korea Astronomy and Space Science Institute,
Daejeon 34055, Republic of Korea  Astronomical Institute, Faculty of
Mathematics and Physics, Charles University in Prague, 180 00 Praha 8, V
Holešovičkách 2, Czech Republic  Dong-Jin Kim Korea Astronomy and Space
Science Institute, Daejeon 34055, Republic of Korea  Jang-Ho Park Korea
Astronomy and Space Science Institute, Daejeon 34055, Republic of Korea  Hye-
Young Kim Department of Astronomy and Space Science, Chungbuk National
University, Cheongju 28644, Republic of Korea  Pakakaew Rittipruk National
Astronomical Research Institute of Thailand, Chiang Mai 50200, Thailand
Cheongho Han Department of Physics, Chungbuk National University, Cheongju
28644, Republic of Korea Kyeongsoo Hong<EMAIL_ADDRESS>
###### Abstract

We present detailed analyses of updated eclipse timing diagrams for 32 contact
binary merger candidates in the Galactic bulge. The photometric data was
obtained from 2016 to 2021 using the Korea Microlensing Telescope Network
(KMTNet) with the 1.6 m telescopes located at three southern sites (CTIO,
SAAO, and SSO). The times of minimum lights were determined by applying the
binary star model to full light curves created at half-year intervals from the
observations. The orbital period variations of the binary systems were
analyzed using the $O-C$ diagrams from our new timings with the others
published in the literature (Hong et al.), which are based on the OGLE
observations from 2001 to 2015. As results, the orbital periods and period
decreasing rates of 32 binary systems were located to be in the ranges of
0.370$-$1.238 days and from $-3.0$ to $-13.1\times 10^{-6}$ day yr-1,
respectively. Out of these stars, 24 systems show a combination effect of a
parabola and a light travel time caused by a third body and their outer
orbital periods are in the range of 9.1$-$26.5 yr, respectively. We propose
that all of our merger candidates need additional monitoring observations to
study a luminous red nova (LRN) progenitor.

Contact binary stars(297); Eclipsing binary minima timing method(443);
Eclipsing binary stars (444); Galactic bulge(2041); Stellar mergers(2157)

##  1 Introduction

The merger of low-mass contact eclipsing binaries (CEBs) with a rapidly
decaying orbital period is associated with the transient events of luminous
red novae (LRNe), which are located between classical novae and supernovae
(Soker & Tylenda, 2003; Kulkarni et al., 2007; Tylenda et al., 2011; Ivanova
et al., 2013; Pejcha, 2014; Pejcha et al., 2016, 2017; Pastorello et al.,
2019). The mechanisms of merger can be triggered by the Darwin instability
(Darwin, 1879), which occurs when the spin angular momentum of the system is
more than a third of its orbital angular momentum ($J_{\rm orb}\leqslant
3J_{\rm spin}$, Hut, 1980; Rasio, 1995; Li & Zhang, 2006). This happens when
the mass of the secondary component and the mass ratio are extremely small
(Soker & Tylenda, 2006; Wadhwa et al., 2021) and/or possibly in combination
with other effects such as mass and angular momentum loss from the vicinity of
the $L_{2}$ point (Rasio, 1995; Stępień & Gazeas, 2012; Pejcha et al., 2016).

Recently, Howitt et al. (2020) summarized the plateau period and distance
estimates for the recognized 13 LRNe. Among them, LRN V1309 Sco is the only
known confirmed merger event, based on archive data from the OGLE by Tylenda
et al. (2011). He reported that the precursor of the LRN was a W UMa-type
contact binary with an orbital period of 1.44 days and that it showed an
exponential period decay before the eruption. Nandez et al. (2014) presented
that the initial mass of V1309 Sco with a mass ratio of $\sim 0.1$ was
composed of 1.52 $M_{\odot}$ primary and 0.16 $M_{\odot}$ secondary. Ferreira
et al. (2019) suggested that the LRN V1309 Sco will evolve into a blue
straggler state. Such stellar merger events occur about once every 10 yr in
our galaxy (Kochanek et al., 2014). The CEB precursors provide us an excellent
opportunity for understanding stellar merging scenarios because they provide
the fundamental physical parameters such as masses, radii, luminosities, etc.

The potential stellar merger candidates were explored using the methods such
as identification of EBs with either a striking period decreasing rate
(Pietrukowicz et al., 2017; Gazeas et al., 2021; Hong et al., 2022), and
extremely low-mass ratio below $\sim$ 0.1 (Li et al., 2017; Caton et al.,
2019; Wadhwa et al., 2021; Li et al., 2022; Liu et al., 2023). Especially,
Hong et al. (2022) presented the study for the orbital period variations of
14,127 CEBs in the Galactic bulge based on the OGLE observations between 2001
and 2015. In this study, we selected a total of 132 CEBs with period
decreasing rates of higher than $-3\times 10^{-6}$ day yr-1 in the paper of
Hong et al. (2022), which satisfy the necessary criterion for a precursor
candidate by Molnar et al. (2017). The main goal of this study is to find out
the potential merger candidates using new eclipse timings of the selected CEBs
from the Korea Microlensing Telescope Network (KMTNet) observations between
2016 and 2021 in the Galactic bulge. In Section 2, the observational data of
the KMTNet and OGLE are described. Section 3 presents the results of eclipse
timing analysis for the stellar merger candidates. The summary and discussion
of our results are presented in Section 4.

##  2 Data from observations

###  2.1 KMTNet

The photometric data of the selected CEBs between 2016 and 2021 in the
Galactic bulge were collected from the KMTNet data archive. The KMTNet
observations were monitored with an exposure time of 2 minutes using the 1.6 m
wide field optical telescopes at three sites; the Siding Spring Observatory in
Australia (KMTA), Cerro Tololo Interamerican Observatory in Chile (KMTC), and
the South African Astronomical Observatory in South Africa (KMTS). Each CCD
mosaic camera of the telescopes consisted of four 9k $\times$ 9k chips with a
4 deg2 field of view. Twenty-seven fields of the KMTNet were covered to be
nearly 96 deg2 and they observed with cadences from $\Gamma=2$ hr-1 to
$\Gamma=8$ hr-1 (Kim et al., 2018, Figure 12). In this study, we used the
light curves in $I$ band because most KMTNet observations are performed in $I$
band. All preprocessing and photometry were performed though the KMTNet
pipeline (cf. Kim et al., 2016, 2018), which used the difference image
analysis (DIA, Alard & Lupton, 1998).

###  2.2 OGLE

The OGLE-III&IV observations were carried out between 2001 and 2015 toward the
Galactic bulge using the 1.3 m telescope located at the Las Campanas
Observatory in Chile (Udalski et al., 2008, 2015). The sky coverages of the
OGLE-III (2001$-$2009) and OGLE-IV (2010$-$2015) were about 92 deg2 and 182
deg2 of the Galactic bulge, respectively. Most of the observations were taken
in $I$ band, and some $V$ band images were acquired for the color measurement.

##  3 Eclipse timing Variations

The 132 CEBs with a high period decreasing rate of $\geq-3\times 10^{-6}$ day
yr-1 were selected from the paper of Hong et al. (2022), who presented the
period studies of 14,127 CEBs in the Galactic bulge. From them, a total of 111
CEBs with small scatters and sufficient data points in the KMTNet observations
were selected to determine their eclipse timings. We examined whether the
period changes of these systems could be represented by a downward parabola
and/or a light-travel-time (LTT) effect using the additional eclipse timings.
The downward parabolic variation can be explained by a mass transfer from the
more massive to the less massive star and/or angular momentum loss (AML) by a
magnetic stellar wind. The LTT effect can be caused by an additional companion
in the system. The OGLE and KMTNet observations with a time span of about 20
years allowed us to determine the period change rates ($\dot{P}$) for the
selected CEBs through the observed minus computed ($O-C$) diagram, which is an
efficient tool for determining subtle changes in periodic astrophysical
phenomena (Rovithis-Livaniou, 2020). For this, we performed the approach
described below.

> 1\. It is difficult to obtain the times of minimum light by the method of
> Kwee & van Woerden (1956) from the KMTNet survey data with cadences of
> $\Gamma=2-8$ hr-1. Therefore, we used the determination method of eclipse
> timings for the selected CEBs using the light curve synthesis method (see
> Hong et al., 2019, 2022). In order to obtain the binary parameters, the
> light curves in $I$ band formed by the OGLE and KMTNet observations in 2001
> to 2021 were analyzed using the Wilson-Devinney differential correction code
> (Wilson & Devinney, 1971; Van Hamme & Wilson, 2007, hereafter WD). In the WD
> runs, the initial binary parameters were taken from the paper of Hong et al.
> (2022). We adjusted only the orbital ephemeris parameters (the epoch
> $T_{0}$, the period $P$, and the period change rate d$P$/d$t$). The OGLE and
> KMTNet observations with the model light curves for 32 CEBs are presented in
> Figures 1 and 2, where CEBs have been identified as merger candidates using
> the following procedures. The basic parameters for the 32 CEBs are listed in
> Table 1. The resulting period change rates (d$P$/d$t$) are presented in the
> last columns of Tables 2-3.

> 2\. The full light curves with an interval of half a year were formed from
> the KMTNet observations of 2016$-$2021, and analyzed with the binary
> parameters by the WD code. After testing intervals of 90, 180, and 360 days,
> the interval was selected by considering the information loss caused by the
> binning process, the increase in error and the outlier occurrence rate due
> to the decreasing bin size. At that point, only the orbital ephemeris
> parameters were adjusted. Then the primary eclipse timings and their
> uncertainties were determined from the epoch ($T_{0}$), and the error in
> each light curve solution calculated by the WD program, respectively. The
> individual eclipse timings for 8 CEBs with a parabola, and 24 CEBs with two
> variations, are listed in Tables 4 and 5, respectively.

> 3\. In addition to these, the times of minimum light for the selected CEBs
> have been collected from the paper of Hong et al. (2022), who used the OGLE-
> III&IV archive data from 2001 to 2015. In order to obtain a mean light
> ephemeris, the initial epochs and periods were adopted from Hong et al.
> (2022), and we applied a linear least-square fit to all eclipse timings, as
> follows:
>
> $C_{1}=T_{0}+PE.$ (1)
>
> Here, $T_{0}$ is the reference epoch, $E$ denotes the epoch number for a
> given cycle of the binary system, and $P$ is the orbital period. Although
> the selected CEBs show downward parabolic variations in their $O-C$ diagrams
> by Hong et al. (2022), additional variations in the diagrams can be
> displayed by adding new eclipse timings from the KMTNet. Therefore, we
> examined whether the $O-C$ diagrams could be explained by a quadratic
> ephemeris or the combination of a quadratic term and an LTT ephemeris caused
> by the influence of an additional component, using the following equations:
>
> $C_{2}=T_{0}+PE+AE^{2},$ (2)
>
> where, $A$ denotes the coefficient of secular period change, and the
> parameters in Equation (2) were solved by a least-squares quadratic fitting.
>
> $C_{3}=T_{0}+PE+AE^{2}+\tau_{3}.$ (3)
>
> The LTT effect $\tau_{3}$ in Equation (3) depends on five parameters (Irwin,
> 1952, 1959): $a_{\rm AB}$sin$i_{\rm out}$, $e_{\rm out}$, $\omega_{\rm
> out}$, $n$, and $T_{\rm peri}$. Here, $a_{\rm AB}$ denotes the projected
> semi-major axis of the inner binary system, and $i_{\rm out}$, $e_{\rm out}$
> and $\omega_{\rm out}$ are the inclination, the eccentricity and the
> longitude of the periastron of the eclipsing pair around the barycentre of
> the three-body system, respectively. The $n$ and $T_{\rm peri}$ denote the
> true anomaly and the time of the periastron passage, respectively. The eight
> variables of the ephemeris in Equation (3) were evaluated using the
> Levenberg-Marquardt algorithm (Press et al., 1992). To select the best-fit
> model, we compared two different models using the reduced $\chi^{2}$ of the
> Equation (2) and (3). As the results, a total of 32 CEBs with a period
> decreasing rate higher than $-3\times 10^{-6}$ day yr-1 were identified,
> where the eclipse timing diagrams of 8 and 24 CEBs were explained by a
> parabolic variation and by a combination of a parabolic plus LTT ephemeris,
> respectively.

###  3.1 Parabolic Variation

The eclipse timing diagrams of 32 CEBs with a parabolic variation are plotted
in the upper panel of Figures 3-5, where the red curves in Figure 3 and blue
dashed lines in Figures 4-5 represent the quadratic terms. The resulting
period decreasing rates of 32 CEBs are listed in the fourth column of Table 2
and the third column of Table 3. As one can see in Table 2, the period
decreasing rates of 8 CEBs from eclipse timing analysis are in good agreement
with those obtained by the WD code, while the difference between
$\dot{P}_{(O-C_{3})}$ and $\dot{P}_{\rm WD}$ for about half of the 24 CEBs in
Table 3 are outside their uncertainties. All of the selected CEBs were in an
orbital period range of 0.370$-$1.238 days and in a period decreasing rate
range between $-3.0$ and $-13.1\times 10^{-6}$ day yr-1. In our samples, the
system OGLE-BLG-ECL-169991 with an orbital period of 1.176 days had the
highest period decreasing rate ($\dot{P}$) of $-1.31(9)\times 10^{-5}$ day
yr-1.

###  3.2 Parabolic plus LTT effect

The sinusoidal variations in the eclipse timing diagram can be explained by
either a LTT effect due to an additional companion in a binary system (Irwin
1952, 1959) or a magnetic activity cycle (Applegate 1992; Lanza et al. 1998).
Out of 32 CEBs, the best-fitting parameters of 24 CEBs with a quadratic term
plus LTT ephemeris are listed in Table 3 together with other related
quantities. The eclipse timing diagrams of 24 CEBs calculated based on
Equation (3) are displayed in the top panels of Figures 4-5, where the red
solid lines represent the full contribution of Equation (3). The middle and
bottom panels show the LTT orbit of the third body and the residuals from the
full ephemeris, respectively. The outer orbital periods of 24 CEBs are in the
ranges of 9.1$-$26.5 yr.

##  4 Summary and Discussion

Hong et al. (2022) presented a study of period variations for 14,127 CEBs in
the Galactic bulge based on the OGLE observations between 2001 and 2015. Of
them, a total 132 CEBs had period decreasing rates larger than $-3\times
10^{-6}$ day yr-1. The decreasing rate is the necessary criterion for a
precursor candidate by Molnar et al. (2017). In this study, we examined the
data to find suitable merger candidates using the updated eclipse timing
diagrams of 111 CEBs, which had small scatters and enough data points based on
the additional KMTNet observations from 2016 to 2021. The eclipse timings for
the selected systems were obtained using the binary solutions from the WD
code, which determined the minimum epochs without the impact of spot activity
or distorted light curves (Maceroni & van’t Veer, 1994; Lee et al., 2014). The
$O-C$ diagrams of all selected CEBs showed parabolic variations. Out of these
candidates, a total of 24 CEBs exhibit a quadratic term plus LTT ephemeris.
The orbital periods and period decreasing rates of all selected CEBs were
located within the ranges of 0.370$-$1.238 days and from $-3.0$ to
$-13.1\times 10^{-6}$ day yr-1, respectively. Their outer orbital periods of
24 CEBs are in the range of 9.1$-$26.5 yr. As a results, a total of 32
potential binary merger candidates were found to satisfy the essential
criterion. These systems need constant monitoring observations until the
binary merging, or a revision of their period variations.

Among our candidates, OGLE-BLG-ECL-169991 with the highest period decreasing
rate of $-1.31\times 10^{-5}$ day yr-1 has a relatively long orbital period of
1.176 days. Its period is similar to the LRN 1309 Sco with an orbital period
of 1.43 days before eruption. Wadhwa et al. (2021) listed the most important
theoretical factors for the dynamic instability of binary systems to be the
merger, such as the relationship between orbital and spin angular momentum
(Rasio, 1995), the degree of contact (Rasio & Shapiro, 1995; Li & Zhang,
2006), and angular momentum loss (Stępień & Gazeas, 2012). The Darwin
instability happens when the mass ratio is less than the theoretical limit
(Yang & Qian, 2015). The instability mass ratio limit for contact binaries has
been predicted to be below 0.1 by several investigators (Rasio, 1995; Li &
Zhang, 2006; Arbutina, 2007, 2009; Yang & Qian, 2015; Wadhwa et al., 2021,
2023), and many systems with an extremely low mass ratio have been detected,
such as: V1187 Her ($q\sim 0.044$; Caton et al., 2019), ZZ PsA ($q\sim 0.078$;
Wadhwa et al., 2021), CW Lyn ($q\sim 0.067$; Gazeas et al., 2021). This makes
it necessary to check the mass ratio of OGLE-BLG-ECL-169991 using the
spectroscopic observations.

We thank the OGLE and KMTNet teams for all of the observations. This research
has made use of the KMTNet system operated by the Korea Astronomy and Space
Science Institute (KASI), and the data were obtained at three host sites of
CTIO in Chile, SAAO in South Africa, and SSO in Australia. This research was
supported by the KASI grant 2023-1-832-03 and by grants (2019R1A2C2085965 and
2022R1I1A1A01053320) from the National Research Foundation (NRF) of Korea.

## References

  * Alard & Lupton (1998) Alard, C. & Lupton, R. H. 1998, ApJ, 503, 325
  * Arbutina (2007) Arbutina, B. 2007, MNRAS, 377, 1635
  * Arbutina (2009) Arbutina, B. 2009, MNRAS, 394, 501
  * Caton et al. (2019) Caton, D., Gentry, D. R., Samec, R. G., et al. 2019, PASP, 131, 054203
  * Darwin (1879) Darwin, G. H. 1879, The Observatory, 3, 79
  * Ferreira et al. (2019) Ferreira, T., Saito, R. K., Minniti, D., et al. 2019, MNRAS, 486, 1220
  * Gazeas et al. (2021) Gazeas, K. D., Loukaidou, G. A., Niarchos, P. G., et al. 2021, MNRAS, 502, 2879
  * Hong et al. (2019) Hong, K., Lee, J. W., Kim, S.-L., et al. 2019, AJ, 158, 185
  * Hong et al. (2022) Hong, K., Woo Lee, J., Park, J.-H., et al. 2022, AJ, 163, 157
  * Howitt et al. (2020) Howitt, G., Stevenson, S., Vigna-Gómez, A., et al. 2020, MNRAS, 492, 3229
  * Hut (1980) Hut, P. 1980, A&A, 92, 167
  * Irwin (1952) Irwin, J. B. 1952, ApJ, 116, 211
  * Irwin (1959) Irwin, J. B. 1959, AJ, 64, 149
  * Ivanova et al. (2013) Ivanova, N., Justham, S., Avendano Nandez, J. L., et al. 2013, Science, 339, 433
  * Kim et al. (2016) Kim, S.-L., Lee, C.-U., Park, B.-G., et al. 2016, Journal of Korean Astronomical Society, 49, 37. doi:10.5303/JKAS.2016.49.1.37
  * Kim et al. (2018) Kim, D.-J., Kim, H.-W., Hwang, K.-H., et al. 2018, AJ, 155, 76
  * Kochanek et al. (2014) Kochanek, C. S., Adams, S. M., & Belczynski, K. 2014, MNRAS, 443, 1319
  * Kulkarni et al. (2007) Kulkarni, S. R., Ofek, E. O., Rau, A., et al. 2007, Nature, 447, 458
  * Kwee & van Woerden (1956) Kwee, K. K. & van Woerden, H. 1956, Bull. Astron. Inst. Netherlands, 12, 327
  * Lee et al. (2014) Lee, J. W., Park, J.-H., Hong, K., et al. 2014, AJ, 147, 91
  * Li et al. (2017) Li, K., Hu, S., Chen, X., et al. 2017, PASJ, 69, 79
  * Li et al. (2022) Li, K., Gao, X., Liu, X.-Y., et al. 2022, AJ, 164, 202
  * Li & Zhang (2006) Li, L. & Zhang, F. 2006, MNRAS, 369, 2001
  * Liu et al. (2023) Liu, X.-Y., Li, K., Michel, R., et al. 2023, MNRAS, 519, 5760
  * Maceroni & van’t Veer (1994) Maceroni, C. & van’t Veer, F. 1994, A&A, 289, 871
  * Molnar et al. (2017) Molnar, L. A., Van Noord, D. M., Kinemuchi, K., et al. 2017, ApJ, 840, 1
  * Nandez et al. (2014) Nandez, J. L. A., Ivanova, N., & Lombardi, J. C. 2014, ApJ, 786, 39
  * Pastorello et al. (2019) Pastorello, A., Mason, E., Taubenberger, S., et al. 2019, A&A, 630, A75
  * Pejcha (2014) Pejcha, O. 2014, ApJ, 788, 22
  * Pejcha et al. (2016) Pejcha, O., Metzger, B. D., & Tomida, K. 2016, MNRAS, 455, 4351
  * Pejcha et al. (2017) Pejcha, O., Metzger, B. D., Tyles, J. G., et al. 2017, ApJ, 850, 59
  * Pietrukowicz et al. (2017) Pietrukowicz, P., Soszyński, I., Udalski, A., et al. 2017, Acta Astron., 67, 115
  * Press et al. (1992) Press, W. H., Teukolsky, S. A., Vetterling, W. T., et al. 1992, Numerical Recipes (Cambridge: Cambridge Univ. Press)
  * Rasio (1995) Rasio, F. A. 1995, ApJ, 444, L41
  * Rasio & Shapiro (1995) Rasio, F. A. & Shapiro, S. L. 1995, ApJ, 438, 887
  * Rovithis-Livaniou (2020) Rovithis-Livaniou, H. 2020, Galaxies, 8, 78
  * Soker & Tylenda (2003) Soker, N., & Tylenda, R. 2003, ApJ, 582, L105
  * Soker & Tylenda (2006) Soker, N. & Tylenda, R. 2006, MNRAS, 373, 733
  * Stępień & Gazeas (2012) Stępień, K. & Gazeas, K. 2012, From Interacting Binaries to Exoplanets: Essential Modeling Tools, 282, 456
  * Tylenda et al. (2011) Tylenda, R., Hajduk, M., Kamiński, T., et al. 2011, A&A, 528, A114
  * Udalski et al. (2008) Udalski, A., Szymanski, M. K., Soszynski, I., et al. 2008, Acta Astron., 58, 69
  * Udalski et al. (2015) Udalski, A., Szymański, M. K., & Szymański, G. 2015, Acta Astron., 65, 1
  * Van Hamme & Wilson (2007) Van Hamme, W., & Wilson, R. E. 2007, ApJ, 661, 1129
  * Wadhwa et al. (2021) Wadhwa, S. S., De Horta, A., Filipović, M. D., et al. 2021, MNRAS, 501, 229
  * Wadhwa et al. (2023) Wadhwa, S. S., Arbutina, B., Tothill, N. F. H., et al. 2023, arXiv:2306.15190
  * Wilson & Devinney (1971) Wilson, R. E., & Devinney, E. J. 1971, ApJ, 166, 605
  * Yang & Qian (2015) Yang, Y.-G. & Qian, S.-B. 2015, AJ, 150, 69

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
Figure 1: $I$ light curves of 16 binary systems from the OGLE-III (OGIII),
OGLE-IV (OGIV) and KMTNet surveys. The points and solid lines represent the
observations and the theoretical light curves from the WD code, respectively.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
Figure 2: Same as Figure 1, for 16 more systems.

|  |  |
---|---|---|---
|  |  |
Figure 3: Eclipse timing diagrams of eight CEBs with a parabolic variation.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
Figure 4: Eclipse timing diagrams of 16 CEBs with a parabolic plus LTT effect.

|  |  |
---|---|---|---
|  |  |
Figure 5: Same as Figure 5, for 8 more systems.

Table 1: Basic Parameters of 32 CEBs

Object ID | R.A. | Decl. | $I$ | $V-I$ | $P^{a}$ | $T_{0}^{a}$
---|---|---|---|---|---|---
(OGLE-BLG-) | (J2000) | (J2000) | (mag) | (mag) | (day) | (HJD+2450000)
ECL-069847 | 17:42:03.15 |  $-$22:42:46.3 | 17.910 | 1.677 | 0.6666801(26) | 2457000.4720(32)
ECL-088600 | 17:44:57.63 |  $-$34:17:24.9 | 18.410 | 1.810 | 0.5090498(07) | 2457000.1046(10)
ECL-112600 | 17:48:13.97 |  $-$34:12:18.7 | 17.401 | 1.753 | 1.2378942(36) | 2457000.3488(17)
ECL-115269 | 17:48:36.87 |  $-$33:32:31.1 | 16.472 | 2.299 | 1.0832214(23) | 2457000.8871(16)
ECL-151250 | 17:52:36.28 |  $-$30:42:53.2 | 17.127 | 2.175 | 0.9717356(09) | 2457000.7534(06)
ECL-161150 | 17:53:26.14 |  $-$31:12:37.5 | 17.454 | 2.211 | 0.8734009(12) | 2457000.8062(09)
ECL-169991 | 17:54:09.00 |  $-$32:05:55.0 | 17.104 | 1.965 | 1.1757013(27) | 2457000.8180(15)
ECL-186358 | 17:55:32.98 |  $-$29:19:16.8 | 18.164 | … | 0.4410626(09) | 2457000.3277(20)
ECL-192939 | 17:56:06.40 |  $-$29:29:21.4 | 18.206 | 2.032 | 0.5318869(08) | 2457000.4773(15)
ECL-212252 | 17:57:44.48 |  $-$31:03:40.9 | 18.765 | … | 0.4132157(04) | 2457000.2266(07)
ECL-218355 | 17:58:16.47 |  $-$28:31:11.8 | 18.344 | 2.225 | 0.4423432(05) | 2457000.2507(08)
ECL-218952 | 17:58:19.54 |  $-$28:08:21.9 | 16.662 | 2.211 | 0.9275454(07) | 2457000.9101(05)
ECL-220028 | 17:58:24.67 |  $-$29:57:42.0 | 16.705 | 1.492 | 0.9207411(06) | 2457000.7851(04)
ECL-224851 | 17:58:50.40 |  $-$29:00:13.1 | 18.460 | 1.294 | 0.3723072(09) | 2457000.1468(20)
ECL-231595 | 17:59:26.45 |  $-$31:09:49.2 | 17.386 | 1.953 | 0.5903635(06) | 2457000.2168(08)
ECL-241388 | 18:00:18.16 |  $-$28:13:40.4 | 16.905 | 1.723 | 0.8478846(14) | 2457000.7616(11)
ECL-256434 | 18:01:38.90 |  $-$32:15:24.3 | 18.134 | 1.451 | 0.4576949(08) | 2457000.2088(14)
ECL-260102 | 18:01:59.20 |  $-$28:09:04.1 | 17.934 | 1.454 | 0.7524082(05) | 2457000.1329(03)
ECL-264774 | 18:02:25.72 |  $-$30:07:51.9 | 17.861 | 1.693 | 0.4728997(09) | 2457000.3677(13)
ECL-272254 | 18:03:07.25 |  $-$28:15:19.0 | 15.549 | 1.168 | 0.8535252(04) | 2457000.8072(02)
ECL-272282 | 18:03:07.43 |  $-$27:34:04.2 | 17.075 | 1.845 | 0.9639203(12) | 2457000.2507(08)
ECL-273301 | 18:03:12.84 |  $-$27:38:38.8 | 17.456 | 1.888 | 0.4475261(07) | 2457000.2124(11)
ECL-287793 | 18:04:32.28 |  $-$26:53:29.4 | 17.208 | 1.358 | 0.5318892(06) | 2457000.2048(06)
ECL-288727 | 18:04:37.09 |  $-$27:05:54.1 | 17.352 | 1.768 | 0.5338122(04) | 2457000.2634(04)
ECL-288979 | 18:04:38.47 |  $-$29:58:13.5 | 17.993 | 1.328 | 0.3699377(07) | 2457000.3433(12)
ECL-289109 | 18:04:39.30 |  $-$29:05:08.8 | 17.052 | 1.363 | 0.4565416(03) | 2457000.0349(03)
ECL-292951 | 18:05:00.38 |  $-$26:57:52.2 | 16.669 | 1.444 | 0.5956858(05) | 2457000.4901(06)
ECL-293210 | 18:05:02.07 |  $-$26:57:45.3 | 17.807 | 1.428 | 0.4549985(05) | 2457000.2194(07)
ECL-314826 | 18:07:08.24 |  $-$30:22:00.3 | 17.601 | 1.630 | 0.7990017(18) | 2457000.7918(20)
ECL-319624 | 18:07:38.73 |  $-$27:28:18.3 | 17.079 | … | 0.6078540(06) | 2457000.1025(08)
ECL-339477 | 18:09:50.66 |  $-$25:58:29.2 | 16.812 | 1.535 | 1.1381204(25) | 2457001.1215(15)
ECL-346785 | 18:10:40.50 |  $-$25:11:59.9 | 16.351 | 1.628 | 1.0501516(14) | 2457000.8382(09)

  * Note: The coordinates, $I$, and $V-I$ are taken from the OGLE-IV catalogue by Soszyński et al. (2016).

  * a Value taken from Hong et al. (2022).

Table 2: The Secular Period Change Rates of 8 CEBs Object ID | $P^{a}$ | $T_{0}^{a}$ | $\dot{P}_{\rm(O-C_{2})}$ | $\dot{P}_{\rm WD}^{b}$
---|---|---|---|---
(OGLE-BLG-) | (day) | (HJD+2450000) | (day year-1) | (day year-1)
ECL-069847.out | 0.6666749(3) | 2457000.4626(11) |  $-4.00(10)$E$-06$ |  $-3.89(9)$E$-06$
ECL-112600.out | 1.2378877(6) | 2457000.3483(7) |  $-5.42(19)$E$-06$ |  $-5.40(11)$E$-06$
ECL-115269.out | 1.0832219(4) | 2457000.8888(9) |  $-4.17(15)$E$-06$ |  $-4.14(11)$E$-06$
ECL-169991.out | 1.1757003(3) | 2457000.8113(6) |  $-1.31(13)$E$-05$ |  $-1.30(9)$E$-05$
ECL-231595.out | 0.5903635(1) | 2457000.2160(4) |  $-3.00(4)$E$-06$ |  $-2.96(3)$E$-06$
ECL-241388.out | 0.8478777(3) | 2457000.7575(6) |  $-5.82(13)$E$-06$ |  $-5.73(10)$E$-06$
ECL-314826.out | 0.7990036(4) | 2457000.7909(8) |  $-6.78(12)$E$-06$ |  $-5.12(6)$E$-06$
ECL-346785.out | 1.0501509(4) | 2457000.8377(6) |  $-3.70(11)$E$-06$ |  $-3.58(8)$E$-06$
Table 3: The Fitted Parameters for the Parabolic $plus$ LTT ephemeris of 24 CEBs Object ID | Period | $\dot{P}_{\rm(O-C_{3})}$ | $P_{3}$ | $\omega_{\rm out}$ | $e_{\rm out}$ | MHJD |  $a_{\rm AB}$sin$i_{\rm out}$ | $f(m)$ | $\dot{P}_{\rm WD}$
---|---|---|---|---|---|---|---|---|---
(OGLE-BLG-) | (day) | (day year-1) | (year) | (degree) |  | (HJD+2450000) | (AU) |  | (day year-1)
ECL-088600 | 0.50905239(15) |  $-3.53(3)$E$-6$ | 12.4(4) | 229(8) | 0.39(17) | 7419.5(90.9) | 1.32(19) | 0.015(2) |  $-3.57$(7)E$-06$
ECL-151250 | 0.97177086(46) |  $-4.91(10)$E$-6$ | 26.5(1.0) | 189(3) | 0.32(5) | 6120.8(62.8) | 7.05(22) | 0.497(24) |  $-2.16$(8)E$-06$
ECL-161150 | 0.87338659(31) |  $-5.19(6)$E$-6$ | 17.2(5) | 25(4) | 0.00(13) | 6401.0(72.5) | 2.64(21) | 0.062(5) |  $-6.19$(9)E$-06$
ECL-186358 | 0.44106076(18) |  $-3.29(4)$E$-6$ | 12.6(5) | 286(12) | 0.77(13) | 8165.4(79.7) | 1.56(27) | 0.024(4) |  $-3.27$(5)E$-06$
ECL-192939 | 0.53189577(18) |  $-3.39(4)$E$-6$ | 14.3(3) | 49(7) | 0.63(14) | 9350.8(74.5) | 1.97(20) | 0.037(4) |  $-3.26$(4)E$-06$
ECL-212252 | 0.41321226(21) |  $-3.91(5)$E$-6$ | 9.1(6) | 50(13) | 0.00(31) | 7071.1(123.6) | 1.13(27) | 0.018(4) |  $-4.04$(6)E$-06$
ECL-218355 | 0.44233597(30) |  $-4.57(9)$E$-6$ | 11.9(4) | 290(7) | 0.55(12) | 9445.0(84.1) | 2.47(31) | 0.106(14) |  $-5.33$(6)E$-06$
ECL-218952 | 0.92756241(41) |  $-4.75(13)$E$-6$ | 12.5(5) | 275(5) | 0.00(11) | 8182.2(66.3) | 2.02(17) | 0.053(5) |  $-4.90$(8)E$-06$
ECL-220028 | 0.92074865(38) |  $-3.84(8)$E$-6$ | 15.7(7) | 219(6) | 0.34(14) | 7758.7(88.4) | 2.12(25) | 0.039(5) |  $-4.56$(8)E$-06$
ECL-224851 | 0.37231230(24) |  $-4.54(5)$E$-6$ | 23.0(7) | 8(2) | 0.66(5) | 8155.4(46.9) | 8.68(51) | 1.234(82) |  $-2.66$(6)E$-06$
ECL-256434 | 0.45769408(12) |  $-3.32(4)$E$-6$ | 16.0(5) | 198(9) | 0.36(34) | 2753.9(120.7) | 0.89(15) | 0.003(1) |  $-3.57$(5)E$-06$
ECL-260102 | 0.75241242(11) |  $-4.75(3)$E$-6$ | 12.2(6) | 233(7) | 0.11(16) | 7030.6(90.0) | 0.55(07) | 0.001(1) |  $-4.46$(5)E$-06$
ECL-264774 | 0.47290437(12) |  $-3.90(3)$E$-6$ | 10.8(2) | 301(5) | 0.35(10) | 7901.4(51.1) | 1.54(15) | 0.031(3) |  $-3.78$(8)E$-06$
ECL-272254 | 0.85353540(17) |  $-5.82(4)$E$-6$ | 16.7(2) | 292(2) | 0.32(04) | 9018.7(32.4) | 2.53(10) | 0.058(2) |  $-7.00$(5)E$-06$
ECL-272282 | 0.96391717(45) |  $-3.42(12)$E$-6$ | 19.1(5.7) | 38(31) | 0.00(72) | 6678.8(594.9) | 0.43(20) | 0.001(1) |  $-3.79$(8)E$-06$
ECL-273301 | 0.44753733(23) |  $-3.13(6)$E$-6$ | 17.2(6) | 213(4) | 0.00(12) | 6875.6(73.8) | 3.42(23) | 0.135(10) |  $-2.60$(7)E$-06$
ECL-287793 | 0.53189423(12) |  $-3.48(3)$E$-6$ | 12.1(3) | 71(7) | 0.22(14) | 9650.9(80.9) | 0.96(11) | 0.006(1) |  $-3.23$(6)E$-06$
ECL-288727 | 0.53381448(16) |  $-3.18(4)$E$-6$ | 15.5(1.4) | 336(7) | 0.68(12) | 8179.3(88.4) | 1.26(17) | 0.008(1) |  $-2.71$(6)E$-06$
ECL-288979 | 0.36994185(17) |  $-3.40(4)$E$-6$ | 11.5(7) | 210(9) | 0.00(25) | 6422.7(102.5) | 1.36(23) | 0.019(4) |  $-3.23$(6)E$-06$
ECL-289109 | 0.45654460(18) |  $-2.99(4)$E$-6$ | 11.5(1.0) | 182(17) | 0.00(63) | 6813.5(201.8) | 0.73(23) | 0.003(1) |  $-3.06$(4)E$-06$
ECL-292951 | 0.59568544(14) |  $-3.02(4)$E$-6$ | 9.4(4) | 17(8) | 0.21(24) | 5650.6(77.2) | 0.85(12) | 0.007(1) |  $-2.77$(6)E$-06$
ECL-293210 | 0.45499924(15) |  $-3.47(4)$E$-6$ | 11.1(3) | 276(7) | 0.16(13) | 8431.1(76.3) | 1.33(17) | 0.019(3) |  $-3.41$(7)E$-06$
ECL-319624 | 0.60785853(27) |  $-3.43(6)$E$-6$ | 13.1(4) | 27(10) | 0.00(30) | 9552.8(132.0) | 1.48(25) | 0.019(3) |  $-3.56$(6)E$-06$
ECL-339477 | 1.13814860(38) |  $-3.58(8)$E$-6$ | 17.8(2) | 340(3) | 0.08(10) | 8952.2(53.7) | 3.47(21) | 0.132(8) |  $-3.42$(14)E$-06$
Table 4: List of Eclipse Timings for 8 CEBs showing a Parabolic Variations

Object ID | HJD | Error | Cycle | $O-C_{1}$ | $O-C_{2}$ | Source | $N_{\rm obs}$
---|---|---|---|---|---|---|---
(OGLE-BLG-) | (+2450000) | (day) |  | (day) | (day) |  |
ECL-069847 | 2454.9030 | 0.0052 | $-6818$ | $-0.17020$ | $-0.00031$ | OGLE | 49
| 2820.2652 | 0.0034 | $-6270$ | $-0.14578$ | $-0.00211$ | OGLE | 49
| 5378.4205 | 0.0026 | $-2433$ | $-0.02212$ | $-0.00049$ | OGLE | 126
| 5726.4325 | 0.0033 | $-1911$ | $-0.01442$ | $-0.00107$ | OGLE | 100
| 6098.4515 | 0.0037 | $-1353$ | $0.00006$ | $0.00675$ | OGLE | 106
| 6459.7948 | 0.0034 | $-811$ | $0.00556$ | $0.00796$ | OGLE | 89
| 6841.8006 | 0.0042 | $-238$ | $0.00664$ | $0.00685$ | OGLE | 56
| 7177.8039 | 0.0072 | $266$ | $0.00574$ | $0.00600$ | OGLE | 64
| 7858.4587 | 0.0032 | $1287$ | $-0.01456$ | $-0.00850$ | KMTNet | 134
| 7980.4650 | 0.0029 | $1470$ | $-0.00973$ | $-0.00184$ | KMTNet | 126
| 8221.7964 | 0.0053 | $1832$ | $-0.01465$ | $-0.00239$ | KMTNet | 82
| 8225.8157 | 0.0054 | $1838$ | $0.00463$ | $0.01697$ | KMTNet | 88
… |  |  |  |  |  |  |

(This table is available in its entirety in machine-readable form.)

Table 5: List of Eclipse Timings of 24 CEBs with the parabolic $plus$ LTT
ephemeris

Object ID | HJD | Error | Cycle | $O-C_{1}$ | $O-C_{2}$ | $O-C_{3}$ | Source | $N_{\rm obs}$
---|---|---|---|---|---|---|---|---
(OGLE-BLG-) | (+2450000) | (day) |  | (day) | (day) | (day) |  |
ECL-088600 | 2448.4717 | 0.0037 | $-8941.0$ | $-0.16670$ | $0.00208$ | $0.00209$ | OGLE | 46
| 2821.1195 | 0.0027 | $-8209.0$ | $-0.14760$ | $-0.00746$ | $-0.00233$ | OGLE | 55
| 3169.3382 | 0.0022 | $-7525.0$ | $-0.12290$ | $-0.00704$ | $0.00032$ | OGLE | 125
| 3541.9905 | 0.0027 | $-6793.0$ | $-0.09920$ | $-0.00690$ | $-0.00088$ | OGLE | 115
| 3901.4081 | 0.0025 | $-6087.0$ | $-0.07480$ | $-0.00268$ | $0.00054$ | OGLE | 165
| 4268.4583 | 0.0021 | $-5366.0$ | $-0.05350$ | $0.00047$ | $0.00059$ | OGLE | 144
| 4631.4313 | 0.0019 | $-4653.0$ | $-0.03720$ | $0.00144$ | $-0.00124$ | OGLE | 166
| 4913.4606 | 0.0048 | $-4099.0$ | $-0.02460$ | $0.00378$ | $-0.00073$ | OGLE | 35
| 5381.8087 | 0.0029 | $-3179.0$ | $-0.00750$ | $0.00725$ | $0.00060$ | OGLE | 139
| 5727.9783 | 0.0021 | $-2499.0$ | $0.00430$ | $0.01171$ | $0.00437$ | OGLE | 242
| 6102.6414 | 0.0022 | $-1763.0$ | $0.00250$ | $0.00449$ | $-0.00251$ | OGLE | 291
| 6463.5623 | 0.0019 | $-1054.0$ | $0.00310$ | $0.00236$ | $-0.00298$ | OGLE | 263
… |  |  |  |  |  |  |  |

(This table is available in its entirety in machine-readable form.)


# Learning Real-World Image De-Weathering with Imperfect Supervision

Xiaohui Liu1, Zhilu Zhang1, Xiaohe Wu1, Chaoyu Feng, Xiaotao Wang, LEI LEI,
Wangmeng Zuo1

###### Abstract

Real-world image de-weathering aims at removing various undesirable weather-
related artifacts. Owing to the impossibility of capturing image pairs
concurrently, existing real-world de-weathering datasets often exhibit
inconsistent illumination, position, and textures between the ground-truth
images and the input degraded images, resulting in imperfect supervision. Such
non-ideal supervision negatively affects the training process of learning-
based de-weathering methods. In this work, we attempt to address the problem
with a unified solution for various inconsistencies. Specifically, inspired by
information bottleneck theory, we first develop a Consistent Label Constructor
(CLC) to generate a pseudo-label as consistent as possible with the input
degraded image while removing most weather-related degradations. In
particular, multiple adjacent frames of the current input are also fed into
CLC to enhance the pseudo-label. Then we combine the original imperfect labels
and pseudo-labels to jointly supervise the de-weathering model by the proposed
Information Allocation Strategy (IAS). During testing, only the de-weathering
model is used for inference. Experiments on two real-world de-weathering
datasets show that our method helps existing de-weathering models achieve
better performance. Codes are available at
https://github.com/1180300419/imperfect-deweathering.

##  1 Introduction

Figure 1: Examples of various inconsistent types between the degraded (top)
and original GT (bottom) images. The pairs in (a)$\sim$(c) show
inconsistencies in color and illumination. (d) and (e) show that the pairs are
not aligned owing to object movement. (f) shows inconsistent textures due
because of snow melting. The inconsistent parts are highlighted in red boxes.

The effectiveness of outdoor computer vision systems _e.g._ monitoring systems
and autonomous driving is inevitably impacted by weather conditions. Image de-
weathering, which aims at removing undesirable image degradation caused by
adverse weather, _e.g._ rain, fog, and snow, is significant in improving the
performance of the subsequent tasks. Recently, data-driven learning-based de-
weathering methods have experienced notable progress. However, capturing ideal
real-world pairs for training neural networks is nearly impossible since it is
not feasible to capture a scene simultaneously with and without weather-
related artifacts.

To circumvent this problem, some existing de-weathering methods (Li, Tan, and
Cheong 2020; Valanarasu, Yasarla, and Patel 2022; Wang, Ma, and Liu 2023)
initially construct paired data by simulating weather degradation and
subsequently undergo supervised learning. However, synthesized degraded images
have limited realism in modeling complex and variable weather characteristics
(_e.g._ raindrop shapes and fog concentration). Consequently, models trained
using synthetic pairs typically face challenges in effectively generalizing to
realistic severe weather scenarios, even leading to artifacts and the loss of
original details. Other methods (Wu et al. 2021; Wei et al. 2021; Ye et al.
2021) address the lack of real-world pairs by employing semi-supervised or
unsupervised manners. While some of these methods (Rai et al. 2022; Huang, Yu,
and He 2021) have demonstrated promising results, challenges remain in
separating intricate overlaps of foreground and background information in
real-world degraded images (Li et al. 2022).

Recognizing the aforementioned limitations, the recent works (Ba et al. 2022;
Zhang et al. 2023) still pin their hopes on acquiring real-world image pairs.
They relax the strict requirement for simultaneous acquisition of ideal pairs
and instead retrieve degraded and clean images from live streams of landscape
scenes on YouTube within the shortest possible time window. And the conclusion
shows that such a manner indeed bridges the domain gap better than synthetic
(Li et al. 2016; Hu et al. 2019; Li, Cheong, and Tan 2019) and semi-real (Wang
et al. 2019) datasets. It is worth noting that, despite the adoption of strict
filtering conditions during dataset collection to ensure spatial position and
illumination consistency between the ground-truth (GT) and degraded images,
certain inconsistencies persist, which will have a detrimental impact on the
learning process of the de-weathering model.

Actually, the inconsistencies also occur in some other low-level vision
(_e.g._ super-resolution (Zhang et al. 2022; Wang et al. 2023), deblurring (Li
et al. 2023), learnable ISP (Zhang et al. 2021b), and etc (Feng et al. 2023))
tasks. And their works usually utilize optical flow estimation (Sun et al.
2018) and color correction approaches (He, Sun, and Tang 2012) for aligning
the network output and GT in position and color, respectively. However, these
solutions are less effective when being applied to de-weathering due to the
complexity and diversity of inconsistent types (see Fig. 1).

In this work, we suggest a novel perspective to alleviate the adverse impact
of imperfect supervision in real-world de-weathering datasets, and propose a
unified solution for a variety of inconsistencies. The main motivation is from
the information bottleneck (IB) theory (Tishby and Zaslavsky 2015). Therein,
during network training, it has been observed that the mutual information
between the learned features and target increases monotonically, while that
between the learned features and input increases first and then decreases
gradually, as shown in Fig. 2. Taking advantage of this property, we can
develop a Consistent Label Constructor (CLC) for generating a pseudo-label
whose non-weather content is as consistent as possible with the input degraded
image while degradations are mostly removed. Considering the characteristics
of the dataset itself, multiple adjacent frames of the current input can be
also fed into CLC to further enhance the ability of the pseudo-label. Noted
that although the pseudo label is better than the original imperfect label in
terms of consistency preservation, it may not be as satisfactory as the
original label in terms of degradation removal. Therefore, we combine the two
labels and present an Information Allocation Strategy (IAS) for supervising
the de-weathering model, which draws on the advantages and discards the
disadvantages of both. During inference, only the de-weathering model is used
for removing weather-related degradations.

Experiments are conducted with RainRobust (Ba et al. 2022) and Restormer
(Zamir et al. 2022) models on GT-Rain-Snow (Ba et al. 2022; Zhang et al. 2023)
and WeatherStream (Zhang et al. 2023) datasets. With the proposed method, the
quantitative and qualitative results of the existing de-weathering models are
effectively improved, while not increasing any inference cost.

The main contributions are as follows:

  * •

In real-world de-weathering datasets, the content inconsistencies between
degraded and labeled images hurt the performance of de-weathering models, and
we propose a unified and novel solution for various inconsistencies.

  * •

We first feed multiple frames into Consistent Label Constructor (CLC) to
generate a pseudo-label consistent with input content, and then combine the
pseudo- and original labels for supervising the de-weathering model by the
Information Allocation Strategy (IAS).

  * •

Experiments demonstrate that our proposed method enhances the performance of
existing de-weathering models without incurring additional inference costs.

##  2 Related Work

###  2.1 De-Weathering Methods

Deraining. Traditional deraining methods (Garg and Nayar 2005, 2007; Chen and
Hsu 2013) often delve into handcrafted image priors to remove rain streaks.
However, the manually designed priors heavily depend on empirical observations
and fail to capture the intrinsic properties of clean images accurately. In
contrast, recent advancements in deep learning-based approaches, utilizing
neural convolutional networks (CNNs) (Fu et al. 2017; Jiang et al. 2020) or
Transformers (Qin et al. 2021; Chen et al. 2023b), have achieved remarkable
rain removal performance by effectively learning prior knowledge from paired
data. Nonetheless, these methods typically rely on synthetic data and show
limited generalizability when applied to real-world images. To address this
limitation, several studies have explored semi-supervised (Yasarla, Sindagi,
and Patel 2021) or unsupervised (Wang et al. 2022; Ye et al. 2022) frameworks
for handling real rainy images, given the scarcity of real-world deraining
datasets. However, these approaches often encounter challenges in achieving
satisfactory deraining results.

Desnowing. Early works usually employ priors derived from rainfall-driven
features to model snow particles. Desnownet (Liu et al. 2018) is the first
CNN-based approach introduced for snow removal. Recently, DDMSNet (Zhang et
al. 2021a) further presents a deep dense multiscale network incorporated
semantic and geometric priors to enhance snow removal. To address the
challenges posed by diverse scenes, a multi-scale projection transformer (Chen
et al. 2023a) is proposed, which employs a multi-path approach to involve
various snow degradation features and utilizes self-attention operation to
integrate comprehensive scene context information for clean reconstruction.

Dehazing. Previous attempts (He, Sun, and Tang 2010; Li et al. 2019) primarily
rely on the parameter estimation of the atmospheric scattering model by the
handcrafted priors on haze-free images. With the advent of deep learning
techniques, learning-based methods have been developed rapidly by exploring
the deployment of CNNs and Transformers effectively (Guo et al. 2022). Despite
performing excellently well on synthetic data, the above approaches typically
suffer from constrained generalization in handling real-world hazy images
(Ancuti et al. 2021) . As an alternative, some studies endeavor to employ GANs
(Goodfellow et al. 2014) within the unpaired or unsupervised framework (Zhu et
al. 2017; Chen et al. 2022b; Ye et al. 2022) for real image dehazing. However,
GANs often generate images with artifacts, which adversely affects the
subsequent model training.

Unified De-Weathering. To remove various adverse weather conditions via a
single set of pre-trained parameters, All-in-One (Li, Tan, and Cheong 2020)
deploys multiple encoders associated with different weather types for image
restoration. However, it suffers from the model inefficiency problem. To
overcome this, TransWeather (Valanarasu, Yasarla, and Patel 2022) presents a
transformer-based structure comprising a single encoder-decoder network with
learnable weather type queries. In contrast, (Chen et al. 2022a) draw
inspiration from knowledge distillation, and leverage multiple well-trained
teacher networks to distill the student model for multiple weather effects.
Meanwhile, (Wang, Ma, and Liu 2023) introduce a multi-task learning way for
deraining and desnowing by designing multiple heads. Although the
aforementioned works achieve encouraging results in various weather types with
a unified framework, they are typically trained on synthetic data, leading to
degraded performance in real-world all-weather removal.

###  2.2 De-Weathering Datasets

Single weather removal, such as deraining (Li et al. 2016; Yang et al. 2017;
Fu et al. 2017; Zhang and Patel 2018; Li, Cheong, and Tan 2019; Zhang,
Sindagi, and Patel 2019; Hu et al. 2019), dehazing (Li et al. 2018a; Ancuti et
al. 2019, 2021) and desnowing (Liu et al. 2018; Chen et al. 2020b, 2021) has
made substantial advancements due to the numerous publicly accessible
datasets. As for the de-weathering task, due to the absence of dedicated
datasets, prevailing methods often explore the integration of existing
weather-specific datasets. For example, some works (Valanarasu, Yasarla, and
Patel 2022; Li, Tan, and Cheong 2020) sample images from Snow100K (Liu et al.
2018), Raindrop (Qian et al. 2018) and Outdoor-Rain (Li, Cheong, and Tan 2019)
to facilitate the training of the model. However, those weather-specific
datasets are commonly composed of synthetic paired data given certain
degradation.

Recently, SPA (Wang et al. 2019) dataset introduces a semi-automatic approach
to generate one high-quality clean image from each input sequence of real rain
images, and builds the semi-real paired dataset. RainRobust (Ba et al. 2022)
indeed establishes the real-world paired deraining dataset, _i.e._ , GT-Rain,
by enforcing rigorous selection criteria to minimize the environmental
variations in time multiplexed pairs. Unfortunately, it requires significant
human and financial resources, which limits the size of the dataset. To reduce
manpower requirements, WeatherStream (Zhang et al. 2023) then introduce an
automatic pipeline capturing real-world weather effects and their clean image
pairs, which can be typically regarded as a real-world de-weathering dataset.

Figure 2: Comparison of mutual information in different training periods. At
different training epochs, we test the RainRobust model trained on the GT-
Rain-Snow dataset, and measure the mutual information between the output and
the target as well as between the output and the input. The model was trained
in total $20$ epochs, and we calculate it every $2$ epochs.

##  3 Proposed Method

Figure 3: Overview of the proposed pipeline. Left: In training stage 1, we
pre-train a Consistent Label Constructor (CLC), which utilizes multiple
degraded frames to produce a pseudo-label $\mathbf{\tilde{G}}_{i}$ consistent
with the input degraded image $\mathbf{I}_{i}$. In training stage 2, we
leverage both the original label $\mathbf{G}_{i}$ and pseudo-label
$\mathbf{\tilde{G}}_{i}$ to constrain the output $\mathbf{\hat{G}}_{i}$ of the
de-weathering model. Right: During testing, only the de-weathering model is
used to remove weather-related degradations. Better viewed in color.

###  3.1 Motivation and Overall Pipeline

Denote by $\mathbf{I}$ and $\mathbf{G}$ a degraded image caused by severe
weather (_e.g._ , rain, fog, and snow) and the corresponding clean ground-
truth (GT) image. The learning-based de-weathering model $\mathcal{D}$
predicts output $\mathbf{\hat{G}}$ without weather-related degradation from
$\mathbf{I}$, _i.e._ ,

$\mathbf{\hat{G}}=\mathcal{D}(\mathbf{I};\Theta_{\mathcal{D}}).$ (1)

$\Theta_{\mathcal{D}}$ denotes the model parameters, whose optimization
objective can be defined by,

$\Theta_{\mathcal{D}}^{\ast}=\arg\min_{\Theta_{\mathcal{D}}}\mathbb{E}_{\mathbf{I},\mathbf{G}}[\mathcal{L_{\mathcal{D}}}(\mathbf{\hat{G}},\mathbf{G})],$
(2)

where $\mathcal{L}_{\mathcal{D}}$ denotes the loss functions.

Generally, the content of GT image $\mathbf{G}$ should remain the same as that
of input $\mathbf{I}$, except for the weather-related degradation to be
removed. But for real-world de-weathering, $\mathbf{G}$ and $\mathbf{I}$
cannot be captured at the same time, and they have to be collected
successively. During this collection process, changes in weather and scenes
are unavoidable and uncontrollable. As a result, there are inconsistencies in
illumination, spatial position, and texture between $\mathbf{G}$ and
$\mathbf{I}$. Although recent works (Ba et al. 2022; Zhang et al. 2023) have
set strict filtering and selection conditions to alleviate this problem, some
subtle inconsistencies still exist, as shown in Fig. 1. Intuitively, these
irregular and diverse inconsistencies will inevitably affect the optimization
of parameters $\Theta_{\mathcal{D}}$, harming the performance of the de-
weathering model.

It is worth mentioning some other real-world low-level vision (_e.g._ super-
resolution (Zhang et al. 2022; Wang et al. 2023), deblurring (Li et al. 2023),
learnable ISP (Zhang et al. 2021b), and etc (Feng et al. 2023)) tasks face
this problem as well, and the corresponding solutions vary. For inconsistent
illumination, guided filter (He, Sun, and Tang 2012) is adopted to correct
output color to target one, and then calculate the loss value (Wei et al.
2020). For spatial misalignment, optical flow estimation (Sun et al. 2018) is
widely used for aligning pairs, and some works propose more robust loss
functions, _e.g._ , CoBi (Kim et al. 2013) and misalignment-tolerate
$\ell_{1}$ (Xia et al. 2023) loss. Regrettably, these single and coarse-
grained inconsistency mitigation methods are less effective for real-world de-
weathering pairs whose inconsistencies are diverse and fine-grained, as
evidenced by the experimental results presented in Sec.5.1.

In this work, we aim to explore a unified approach for alleviating the adverse
impact caused by a variety of inconsistencies in real-world de-weathering
datasets. Inspired by information bottleneck (IB) theory (Tishby and Zaslavsky
2015), we propose a novel perspective for addressing the problem.
Specifically, the IB principle has observed that the mutual information
between the network features and target goes up monotonically, while that
between the network features and input goes up first and then goes down during
the training, as shown in Fig. 2. In other words, for the target information,
the network continuously learns. For the input information, the network
absorbs first and then eliminates the part that is irrelevant to the target.
Thus, it gives us an opportunity to construct a pseudo-label as consistent as
possible with the input. Simultaneously, for de-weathering with non-ideal
supervision, it may be more difficult to fit weather-irrelevant irregular
disturbance from the target than to remove weather-related degradations. So
the pseudo-label is promising to remove most of the degradations while not
fitting the imperfect parts of the original GT.

Furthermore, we develop a Consistent Label Constructor (CLC) for generating
pseudo-labels. Multiple neighboring frames of the current degraded image are
also fed into CLC to enhance the consistency between the pseudo-label and the
degraded image. The details of CLC are described in Sec. 3.2. Additionally,
noted that it is not feasible when only taking the pseudo-label as supervision
for the de-weathering model, as the pseudo-label has to make a trade-off
between consistency preservation and degradation removal. Instead, we combine
the pseudo-label and original imperfect labels to re-train a de-weathering
model by the proposed Information Allocation Strategy (IAS), which is detailed
in Sec. 3.3. Please see Fig. 3 for the overview of the proposed pipeline.

###  3.2 Consistent Label Constructor

The CLC model $\mathcal{C}$ learns pseudo-label $\mathbf{\tilde{G}}_{i}$ from
input $i$-th degraded image $\mathbf{I}_{i}$, _i.e._ ,

$\mathbf{\tilde{G}}_{i}=\mathcal{C}(\mathbf{I}_{i};\Theta_{\mathcal{C}}).$ (3)

The parameters $\Theta_{\mathcal{C}}$ can be optimized by the constraints
between the original label $\mathbf{G}_{i}$ and $\mathbf{\tilde{G}}_{i}$,
which is written as,

$\Theta_{\mathcal{C}}^{\ast}=\arg\min_{\Theta_{\mathcal{C}}}\mathbb{E}_{\mathbf{I}_{i},\mathbf{G}_{i}}[\mathcal{L_{\mathcal{C}}}(\mathbf{\tilde{G}}_{i},\mathbf{G}_{i})],$
(4)

where $\mathcal{L}_{\mathcal{C}}$ denotes the loss functions of CLC model.

Moreover, given that real-world de-weathering datasets are usually collected
from live streams, multiple adjacent frames of degraded images can be easily
obtained. Thus, we can naturally feed the neighboring $2n$ frames into the CLC
model. Eqn. (3) can be modified as,

$\mathbf{\tilde{G}}_{i}=\mathcal{C}(\mathbf{I}_{i-n},...,\mathbf{I}_{i-1},\mathbf{I}_{i},\mathbf{I}_{i+1},...,\mathbf{I}_{i+n};\Theta_{\mathcal{C}}).$
(5)

On the one hand, it does not hinder CLC to produce a pseudo-label consistent
with $\mathbf{I}_{i}$, as the weather-independent content is also consistent
between frames captured in a short amount of time. On the other hand,
sometimes complementary multi-frames help CLC to generate a clearer pseudo-
label, as the rain streaks and snowflakes cannot always be fixed at the same
position in the frame.

For the loss function in Eqn. (4), we adopt $\ell_{1}$ as well as structural
similarity (SSIM) (Wang, Simoncelli, and Bovik 2003) distance between the
output $\mathbf{\tilde{G}}_{i}$ and the original label $\mathbf{G}_{i}$, which
can be written as,

$\mathcal{L_{\mathcal{C}}}(\mathbf{\tilde{G}}_{i},\mathbf{G}_{i})=||\mathbf{\tilde{G}}_{i}-\mathbf{G}_{i}||_{1}+(1-\mathcal{L}_{SSIM}(\mathbf{\tilde{G}}_{i},\mathbf{G}_{i})),$
(6)

where $\mathcal{L}_{SSIM}$ denotes the multi-scale SSIM function.

The network structure of CLC depends on that of the de-weathering model used
in our experiments. For example, when the de-weathering adopts Restormer
(Zhang et al. 2023), we simply modified its single encoder to multiple ones as
the CLC model. The details about the CLC architecture can be seen in the
suppl. In addition, we set $n$ to $2$ for experiments.

###  3.3 Information Allocation Strategy

Table 1: Quantitative improvements of de-weathering models when applying the proposed method. Experiments are conducted with Restormer and RainRobust networks on GT-Rain-Snow and WeatherStream datasets, respectively. Dataset | Mothod | Rain | Fog | Snow | Overall
---|---|---|---|---|---
PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
GT-Rain-Snow (Ba et al. 2022) (Zhang et al. 2023) |  Restormer (Zamir et al. 2022) |  22.63 |  0.7940 |  20.14 |  0.7700 |  21.62 |  0.8123 |  21.51 |  0.7913
Ours-Restormer |  23.10 |  0.7971 |  21.46 |  0.7860 |  21.78 |  0.8178 |  22.17 |  0.7995
RainRobust (Ba et al. 2022) |  22.83 |  0.7887 |  20.95 |  0.7691 |  22.17 |  0.8058 |  22.01 |  0.7871
Ours-RainRobust |  23.24 |  0.7980 |  21.52 |  0.7860 |  22.31 |  0.8107 |  22.40 |  0.7977
WeatherStream (Zhang et al. 2023) |  Restormer (Zamir et al. 2022) |  23.50 |  0.7990 |  22.89 |  0.8034 |  22.29 |  0.8175 |  22.94 |  0.8059
Ours-Restormer |  23.64 |  0.8055 |  23.01 |  0.8062 |  22.66 |  0.8209 |  23.15 |  0.8102
RainRobust (Ba et al. 2022) |  23.46 |  0.7957 |  22.61 |  0.7962 |  22.11 |  0.8143 |  22.78 |  0.8013
Ours-RainRobust |  23.68 |  0.7987 |  22.71 |  0.7971 |  22.54 |  0.8163 |  23.02 |  0.8033

Although the pseudo-label $\mathbf{\tilde{G}}_{i}$ is more consistent with the
input $\mathbf{I}_{i}$ than the original label $\mathbf{G}_{i}$, it may
perform worse on degradation removal. Thus, It is not appropriate to treat
$\mathbf{\tilde{G}}_{i}$ as the only supervision of the de-weathering model.
Naturally, we can also include $\mathbf{G}_{i}$ as additional guidance, and
combine the advantages of $\mathbf{\tilde{G}}_{i}$ and $\mathbf{G}_{i}$ for
supervising the de-weathering model. Thus, the loss function in Eqn. (2) can
be written as,

$\mathcal{L_{\mathcal{D}}}(\mathbf{\hat{G}}_{i},\mathbf{\tilde{G}}_{i},\mathbf{G}_{i})\\!=\\!\mathcal{L}_{pse}(\mathbf{\hat{G}}_{i},\mathbf{\tilde{G}}_{i})\\!+\\!\lambda_{ori}\mathcal{L}_{ori}(\mathbf{\hat{G}}_{i},\mathbf{G}_{i}),$
(7)

where $\mathcal{L}_{pse}$ and $\mathcal{L}_{ori}$ denote the optimization
objectives with pseudo-label and the original label as supervision,
respectively. $\lambda_{ori}$ denotes a weight coefficient. The proper design
of $\mathcal{L}_{pse}$ and $\mathcal{L}_{ori}$ is essential, and we propose
the following Information Allocation Strategy (IAS) for this.

First, IAS should take advantage of the fact that the pseudo-label is more
consistent with the input. Thus, for $\mathcal{L}_{pse}$, we directly adopt
the image-level constraint between the output $\mathbf{\hat{G}}_{i}$ and the
pseudo label $\mathbf{\tilde{G}}_{i}$, _i.e._ ,

$\mathcal{L}_{pse}(\mathbf{\hat{G}}_{i},\mathbf{\tilde{G}}_{i})=||\mathbf{\hat{G}}_{i}-\mathbf{\tilde{G}}_{i}||_{1}.$
(8)

Second, IAS should efficiently leverage clean information from the original
label $\mathbf{G}_{i}$ while mitigating the impact of its inconsistencies as
much as possible. At this time, image-level or pixel-level loss terms are not
suitable. Instead, we impose feature-level and distribution-level constraints
because of their robustness to spatial misalignment and color variations
between image pairs. On the one hand, we adopt rain-robust loss (Ba et al.
2022), which is a variant of contrastive loss and is calculated between
intermediate features of the de-weathering networks. On the other hand, we
utilize Sliced Wasserstein (SW) (Zhang et al. 2022) loss, which is a
probability distribution distance between VGG (Simonyan and Zisserman 2014)
features. In short, the optimization objective with the original label can be
written as,

$\mathcal{L}_{ori}=\mathcal{L}_{Robust}+\lambda_{SW}\mathcal{L}_{SW},$ (9)

where $\mathcal{L}_{Robust}$ and $\mathcal{L}_{SW}$ denote rain-robust and SW
loss terms, respectively. $\lambda_{SW}$ is the weight for SW loss and set to
$0.08$. $\lambda_{ori}$ in Eqn. (7) is set to $0.1$. The details and formulas
of $\mathcal{L}_{Robust}$ and $\mathcal{L}_{SW}$ are shown in the suppl.

##  4 Experiments

Figure 4: Qualitative testing results of the de-weathering models trained with
GT-Rain-Snow dataset. Figure 5: Qualitative testing results of the de-
weathering models trained with WeatherStream dataset. Figure 6: Visualizations
of pseudo-labels.

###  4.1 Datasets

(Zhang et al. 2023) offers datasets constructed in two ways. One of which,
named GT-Rain-Snow, follows the protocol of (Ba et al. 2022) and extends to
multiple weather types by manual collection. On the other hand, the default
WeatherStream dataset is obtained by automatic collection. Specifically, GT-
Rain-Snow (Ba et al. 2022; Zhang et al. 2023) comprises $129$ scenes and
WeatherStream (Zhang et al. 2023) contains a larger set of $424$ scenes. Each
scene consists of approximately $300$ aligned degraded images and one clean
image. We conduct experiments with the two training datasets, respectively.
And experiments are all evaluated on the testing dataset proposed by (Zhang et
al. 2023), which consists of $45$ scenes and covers the sub-tasks of
deraining, dehazing and desnowing.

###  4.2 Implementation Details

We integrate our method into the CNN-based RainRobust (Ba et al. 2022) and
transformer-based Restormer (Zamir et al. 2022) to validate the effectiveness.
During training, we follow (Ba et al. 2022) to employ various augmentation
techniques, including random rotation, padding, cropping, Rain-Mix (Guo et al.
2021) and Snow-Mix (Zhang et al. 2023). For optimizing the models, we adopt
the Adam (Li et al. 2018b) with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. A warm-
up strategy is employed to gradually increase the learning rate from $5\times
10^{-5}$ to $2\times 10^{-4}$, followed by the cosine annealing strategy
(Loshchilov and Hutter 2016) to decrease the learning rate from $2\times
10^{-4}$ to $10^{-6}$. Experiments based on RainRobust are conducted on a
single RTX A$6000$ GPU. The corresponding patch size is set as $256\times
256$, and the batch size is set to $8$. Experiments based on Restormer are
conducted on two RTX A$6000$ GPUs. The corresponding patch and batch sizes are
set to $168\times 168$ and $6$, respectively. All experiments are implemented
with the PyTorch framework.

###  4.3 Comparison with State-of-the-Arts

#### Quantitative Comparisons.

In Table 1, we report the PSNR and SSIM results of the models trained with GT-
Rain-Snow and WeatherStream datasets, respectively. It can be seen that our
method consistently improves the performance of baseline de-weathering models
regardless of whether a CNN or Transformer backbone, thereby demonstrating its
satisfactory capability.

In particular, when training models with GT-Rain-Snow dataset, our
transformer-based Restormer obtains $0.66$dB PSNR improvements while CNN-based
RainRobust achieves $0.39$dB PSNR gains. Although the inconsistencies in the
WeatherStream are significantly reduced in comparison with GT-Rain-Snow. When
training models with WeatherStream, our Restormer and RainRobust still obtain
$0.21$dB and $0.24$dB PSNR gains, respectively. Moreover, our method
universally boosts the capability of de-weathering across different weather
conditions.

#### Qualitative Comparisons.

Fig. 4 and Fig. 5 show visual comparisons on various weather scenarios.
Overall, our method brings visible improvements with fewer degradations and
more consistent color. For instance, in the bottom-right example of Fig. 4,
our method promisingly recovers the color of heavily foggy images, while
vanilla Restormer only removes some streaks. In the top-left example of Fig.
5, our de-weathered image contains fewer snow streaks than vanilla Rain-
Robust.

###  4.4 Visualization of Pseudo-Labels

As mentioned above, our pseudo-labels can have greater consistency with the
degraded images compared to the original label, encompassing color, position,
and other aspects. Here we give some visual examples in Fig. 6. Therein, the
first two rows showcase the superior color and texture consistency of pseudo-
labels with the degraded images, respectively. Notably, the utilization of
multiple frames in CLC enhances the visual quality in the last row, with
pseudo-labels exhibiting more intricate textures, while the original label
still retains rain streaks.

##  5 Ablation study

###  5.1 Comparison with Other Inconsistency- Handling Methods

Table 2: Quantitative results when comparing with other inconsistency-handling methods. The baseline represents the model trained directly with the original label. The results on three sub-tasks are attached in the suppl. Methods |  PSNR$\uparrow$ |  SSIM$\uparrow$
---|---|---
Baseline |  22.01 |  0.7871
Guided Filter |  22.14 |  0.7959
Optical Flow |  22.15 |  0.7932
Misalign-Tolerate $\ell_{1}$ |  22.18 |  0.7951
Guided Filter + Optical Flow |  22.21 |  0.7935
Guided Filter + Misalign-Tolerate $\ell_{1}$ |  22.24 |  0.7976
Ours |  22.40 |  0.7977

Here we compare our method with alternative ones from other works that can
also alleviate the effect of inconsistencies between input and GT images. We
select two types of methods that are robust to color-inconsistent supervision
and misaligned supervision, respectively. Among them, guided filter (He, Sun,
and Tang 2012) is adopted for aligning the color between the output and GT
(Wei et al. 2020; Wang et al. 2023). The optical flow (Sun et al. 2018) is
used in some works (Zhang et al. 2021b; Li et al. 2023) for aligning the
content spatially. And the misalignment-tolerate $\ell_{1}$ (Xia et al. 2023)
loss is also present to tackle the misalignment.

During experiments, we train the de-weathering model with the single or
combined methods by following their ways. The results are shown in Table 2.
From the table, although these methods can also improve performance compared
to the baseline, our method achieves better performance than them.
Furthermore, even combining two complementary techniques (_e.g._ , Guided
Filter + Optical Flow), our method still performs better than them.

###  5.2 Effect of Number of Frames in CLC

Figure 7: Results of the CLC model that takes the different number of frames as input. Table 3: Effect of feeding the different number of frames into the CLC model. The results of the CLC and corresponding de-weathering models are both reported. Frame Number | CLC | De-Weathering
---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
1 | 22.01 | 0.7871 | 22.19 | 0.7971
3 | 22.84 | 0.8199 | 22.34 | 0.7972
5 | 23.30 | 0.8282 | 22.40 | 0.7977
7 | 23.20 | 0.8277 | 22.39 | 0.7976
Table 4: Quantitative results of de-weathering model trained with different supervisions. Pseudo-Label | Original Label | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---
$\times$ | ✓ | 22.01 | 0.7871
✓ | $\times$ | 22.18 | 0.7903
✓ | ✓ | 22.40 | 0.7977

Here we investigate the effect of feeding different number of frames into
Consistent Label Constructor (CLC). Table 3 evaluates the performance of both
CLC and its enhanced de-weathering model. It can be observed that their
performance generally increases as the number of frames increases. When the
number of frames is larger than 5, the performance tends to be saturated.
Meanwhile, it can be seen in Fig. 7 that more input frames will improve the
visual quality of pseudo-labels, _i.e._ , properer texture and color, and
fewer weather-related artifacts.

###  5.3 Effect of Supervision Setting

Here we conduct an ablation experiment to investigate the impact of different
supervisions when training the de-weathering model. As illustrated in Table 4,
benefiting from CLC’s meticulous design, taking pseudo-labels for supervising
the de-weathering model is more effective than taking the original labels for
that. Nevertheless, combining the two labels can achieve better performance
gain. This result also indicates the complementary nature of the two
supervisions, and shows the effectiveness of our proposed Information
Allocation Strategy (IAS).

##  6 Conclusion

In this work, we propose a unified method to mitigate the adverse effects of
various inconsistencies presenting in real-world de-weathering pairs.
Initially, we develop a Consistent Label Constructor (CLC) that generates a
pseudo-label consistent with the input degraded image while effectively
eliminating the most weather-related degradations. To further enhance the
pseudo-label, we incorporate multiple adjacent frames of the input into the
CLC. Then we propose an Information Allocation Strategy (IAS) to combine the
original imperfect labels and pseudo-labels, thereby providing comprehensive
and complementary supervision for the de-weathering networks. Experimental
results on two real-world de-weathering datasets demonstrate that our method
significantly improves the performance of existing de-weathering models.

## Supplementary Material

##  Appendix A Content

The content of this supplementary material involves:

  * •

Details of Sliced Wasserstein (SW) loss in Sec. B.

  * •

Details of Rain-Robust loss in Sec. C.

  * •

Architecture of Consistent Label Constructor (CLC) in Sec. D.

  * •

Detailed results of the ablation study in Sec. E.

  * •

More visual comparisons of de-weathering in Sec. F

  * •

More visual results of pseudo-labels in Sec. G

##  Appendix B Sliced Wasserstein Loss

As shown in Alg. A, to calculate the Sliced Wasserstein (SW) loss between the
output image $\mathbf{\hat{G}}$ and the target image $\mathbf{G}$, we first
get the $2$-dimensional features from VGG19 (Simonyan and Zisserman 2014),
following by reshaping them into $1$-dimensional through random liner
projection. Then, we calculate the Wasserstein distance between the output and
the target $1$-dimensional probability distributions, which is defined as the
element-wise $\ell_{1}$ distance over sorted $1$-dimensional distributions.

##  Appendix C Rain-Robust Loss

Rain-Robust loss is proposed by (Ba et al. 2022), which is inspired by the
advance in contrastive learning (Chen et al. 2020a). The details of Rain-
Robust loss are shown in Alg. B. For each image in
$\\{(\mathbf{I}_{i},\mathbf{G}_{i})\\}_{i=1}^{N}$, we first utilize a feature
extractor to extract features. After that, we follow InfoNCE (Oord, Li, and
Vinyals 2018) criterion to measure the Rain-Robust loss for every positive
pair. Finally, we return the mean value of the Rain-Robust loss for $N$ image
pairs. In the training process of the de-weathering network, we use $N$ pairs
of degraded-clean images from the same batch as the input for the Rain-Robust
loss, and select the encoder $\mathcal{D_{E}}(\cdot,\Theta_{\mathcal{E}})$ of
de-weathering network $\mathcal{D}(\cdot,\Theta_{\mathcal{D}})$ as the feature
extractor. The temperature coefficient $\tau$ is set to $0.25$.

##  Appendix D Consistent Label Constructor

Figure A: The overall architecture of CLC model and the corresponding de-
weathering model.

Our CLC model corresponds to the de-weathering model, which adopts Restormer
(Zamir et al. 2022) and RainRobust (Ba et al. 2022) in the experiments,
respectively. Both the architectures can be considered as U-Net (Ronneberger,
Fischer, and Brox 2015) framework, which consists of an encoder (left path)
and a decoder (right path). As shown in Fig. A, we maintain the original
decoder unchanged, while expanding the single encoder into multiple ones,
ensuring that each input frame corresponds to a separate encoder. Furthermore,
to enable the successful integration of the outputs from multiple encoders
into the original decoder, we concatenate the outputs together and utilize
simple convolutional layers to adjust their channel dimensions.

##  Appendix E Detailed Results of the Ablation Study

In this section, we present supplementary results of the ablation study. Table
A, Table B and Table C represent the detailed results of Table $2$, Table $3$
and Table $4$ in the main text, respectively.

Algorithm A Pseudo code of SW loss

0: $\mathbf{\hat{G}}$: output image, $\mathbf{G}$: target image,
$\mathbf{M}\in\mathbb{R}^{C^{\prime}\times C}$: random projection matrix.

1: obtain VGG19 features: $\mathbf{U}=\text{VGG}(\mathbf{\hat{G}})$,
$\mathbf{V}=\text{VGG}(\mathbf{G})$, $\mathbf{U}$ and $\mathbf{V}$ are both in
$\mathbb{R}^{C\times H\times W}$.

2: reshape the feature $\mathbf{U}$ and $\mathbf{V}$ to
$\mathbf{U_{r}}\in\mathbb{R}^{C\times HW}$ and
$\mathbf{V_{r}}\in\mathbb{R}^{C\times HW}$, respectively.

3: project the feature $\mathbf{U}$ and $\mathbf{V}$ to
$\mathbb{R}^{C^{\prime}\times HW}$ using $\mathbf{M}$:
$\mathbf{U_{p}}=\mathbf{M}\mathbf{U_{f}}$,
$\mathbf{V_{p}}=\mathbf{M}\mathbf{V_{f}}$.

4: sort $\mathbf{U_{p}}$ and $\mathbf{V}_{p}$:
$\mathbf{U_{s}}=\text{Sort}(\mathbf{U_{p}},\text{dim}=1)$,
$\mathbf{V_{s}}=\text{Sort}(\mathbf{V_{p}},\text{dim}=1)$.

5: return $\|\mathbf{U_{s}}-\mathbf{V_{s}}\|_{1}$.

Algorithm B Pseudo code of Rain-Robust loss

0: $\\{(\mathbf{I}_{i},\mathbf{G}_{i})\\}_{i=1}^{N}$: $N$ degraded-clean image
pairs from different scenes,$\mathcal{D_{E}}(\cdot,\Theta_{\mathcal{E}})$:
encoder of the current de-weathering model,$\tau$: temperature coefficient.

1: for $i$ from $1$ to $N$ do

2: $\mathbf{U}_{i}=\mathcal{D_{E}}(\mathbf{I}_{i},\Theta_{\mathcal{E}})$ and
$\mathbf{V}_{i}=\mathcal{D_{E}}(\mathbf{G}_{i},\Theta_{\mathcal{E}})$,

3: end for

4: for $i$ from $1$ to $N$ do

5:
$\mathcal{L}_{\mathbf{VU}_{i}}=-\text{log}\frac{\text{exp}(\text{sim}_{\text{cos}}(\mathbf{U}_{i},\mathbf{V}_{i})/\tau)}{\Sigma_{j=1,j\neq
i}^{N}(\text{exp}(\text{sim}_{\text{cos}}(\mathbf{U}_{i},\mathbf{U}_{j})/\tau)+\text{exp}(\text{sim}_{\text{cos}}(\mathbf{U}_{i},\mathbf{V}_{j})/\tau))}$,$\mathcal{L}_{\mathbf{UV}_{i}}=-\text{log}\frac{\text{exp}(\text{sim}_{\text{cos}}(\mathbf{V}_{i},\mathbf{U}_{i})/\tau)}{\Sigma_{j=1,j\neq
i}^{N}(\text{exp}(\text{sim}_{\text{cos}}(\mathbf{V}_{i},\mathbf{U}_{j})/\tau)+\text{exp}(\text{sim}_{\text{cos}}(\mathbf{V}_{i},\mathbf{V}_{j})/\tau))}$,

6: end for

7: return
$\frac{\Sigma_{i=1}^{N}(\mathcal{L}_{\mathbf{VU}_{i}}+\mathcal{L}_{\mathbf{UV}_{i}})}{N}$.

Table A: Quantitative results when comparing with other inconsistency-handling methods. The baseline represents the model trained directly with the original label. Methods | Rain | Fog | Snow | Overall
---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
Baseline | 22.83 | 0.7887 | 20.95 | 0.7691 | 22.17 | 0.8058 | 22.01 | 0.7871
Guided Filter | 23.00 | 0.7960 | 20.91 | 0.7791 | 22.43 | 0.8148 | 22.14 | 0.7959
Optical Flow | 22.81 | 0.7928 | 21.35 | 0.7818 | 22.09 | 0.8065 | 22.15 | 0.7932
Misalign-Tolerate $\ell_{1}$ | 22.86 | 0.7919 | 21.26 | 0.7806 | 22.40 | 0.8135 | 22.18 | 0.7951
Guided Filter + Optical Flow | 23.06 | 0.7962 | 21.11 | 0.7776 | 22.36 | 0.8080 | 22.21 | 0.7935
Guided Filter + Misalign-Tolerate $\ell_{1}$ | 22.89 | 0.7955 | 22.21 | 0.7914 | 21.45 | 0.8072 | 22.24 | 0.7976
Ours | 23.24 | 0.7980 | 21.52 | 0.7860 | 22.31 | 0.8107 | 22.40 | 0.7977
Table B: Effect of feeding the different number of frames into the CLC model. The results of the CLC and corresponding de-weathering models on three sub-tasks are both reported. Frame Number | CLCN | Rain | Fog | Snow | Overall
---|---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
1 | 22.01 | 0.7871 | 22.73 | 0.7965 | 21.81 | 0.7866 | 21.95 | 0.8095 | 22.19 | 0.7971
3 | 22.83 | 0.8198 | 23.40 | 0.7994 | 21.26 | 0.7852 | 22.19 | 0.8080 | 22.34 | 0.7972
5 | 23.30 | 0.8281 | 23.24 | 0.7980 | 21.52 | 0.7860 | 22.31 | 0.8107 | 22.40 | 0.7977
7 | 23.20 | 0.8276 | 23.25 | 0.7998 | 21.52 | 0.7840 | 22.29 | 0.8101 | 22.39 | 0.7976

##  Appendix F Visual Results of De-Weathering

In this section, we provide more qualitative comparisons between our proposed
method and the baselines. Fig. B and Fig. C respectively illustrate the
results of taking RainRobust and Restormer as de-weathering models when
training with the GT-Rain-Snow dataset. Fig. D and Fig. E respectively
illustrate the results of taking RainRobust and Restormer as de-weathering
models when training with the WeatherStream dataset.

##  Appendix G Visual Results of Pseudo-Labels

In this section, we provide more visual results of pseudo-labels generated by
our Consistent Label Constructor (CLC). As shown in Fig. F, despite the
inconsistency between the degraded images and the clean images, our CLC can
generate pseudo-labels that exhibit stronger consistency with the degraded
images.

Table C: Quantitative results of de-weathering model trained with different supervisions. Pseudo-Label | Original Label | Rain | Fog | Snow | Overall
---|---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
$\times$ | ✓ | 22.83 | 0.7887 | 20.95 | 0.7691 | 22.17 | 0.8058 | 22.01 | 0.7871
✓ | $\times$ | 22.87 | 0.7883 | 21.93 | 0.7820 | 21.57 | 0.8021 | 22.18 | 0.7903
✓ | ✓ | 23.24 | 0.7980 | 21.52 | 0.7860 | 22.31 | 0.8107 | 22.40 | 0.7977
Figure B: Qualitative testing results of the RainRobust model trained with the
GT-Rain-Snow dataset. Figure C: Qualitative testing results of the Restormer
trained with the GT-Rain-Snow dataset. Figure D: Qualitative testing results
of the RainRobust model trained with the WeatherStream dataset. Figure E:
Qualitative testing results of the Restormer trained with the WeatherStream
dataset. Figure F: Visualizations of pseudo-labels. Pseudo-labels from top to
bottom sequentially show better consistency with the degraded images in color,
illumination, spatial position, and moving content.

## References

  * Ancuti et al. (2019) Ancuti, C. O.; Ancuti, C.; Sbert, M.; and Timofte, R. 2019.  Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images.  In _2019 IEEE international conference on image processing (ICIP)_ , 1014–1018. IEEE.
  * Ancuti et al. (2021) Ancuti, C. O.; Ancuti, C.; Vasluianu, F.-A.; and Timofte, R. 2021.  NTIRE 2021 nonhomogeneous dehazing challenge report.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 627–646.
  * Ba et al. (2022) Ba, Y.; Zhang, H.; Yang, E.; Suzuki, A.; Pfahnl, A.; Chandrappa, C. C.; de Melo, C. M.; You, S.; Soatto, S.; Wong, A.; et al. 2022.  Not Just Streaks: Towards Ground Truth for Single Image Deraining.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII_ , 723–740. Springer.
  * Chen et al. (2023a) Chen, S.; Ye, T.; Liu, Y.; Liao, T.; Jiang, J.; Chen, E.; and Chen, P. 2023a.  MSP-Former: Multi-Scale Projection Transformer for Single Image Desnowing.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1–5.
  * Chen et al. (2020a) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020a.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607. PMLR.
  * Chen et al. (2020b) Chen, W.-T.; Fang, H.-Y.; Ding, J.-J.; Tsai, C.-C.; and Kuo, S.-Y. 2020b.  JSTASR: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_ , 754–770. Springer.
  * Chen et al. (2021) Chen, W.-T.; Fang, H.-Y.; Hsieh, C.-L.; Tsai, C.-C.; Chen, I.; Ding, J.-J.; Kuo, S.-Y.; et al. 2021.  All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 4196–4205.
  * Chen et al. (2022a) Chen, W.-T.; Huang, Z.-K.; Tsai, C.-C.; Yang, H.-H.; Ding, J.-J.; and Kuo, S.-Y. 2022a.  Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 17653–17662.
  * Chen et al. (2023b) Chen, X.; Li, H.; Li, M.; and Pan, J. 2023b.  Learning A Sparse Transformer Network for Effective Image Deraining.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5896–5905.
  * Chen et al. (2022b) Chen, X.; Pan, J.; Jiang, K.; Li, Y.; Huang, Y.; Kong, C.; Dai, L.; and Fan, Z. 2022b.  Unpaired deep image deraining using dual contrastive learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017–2026.
  * Chen and Hsu (2013) Chen, Y.-L.; and Hsu, C.-T. 2013.  A generalized low-rank appearance model for spatio-temporally correlated rain streaks.  In _Proceedings of the IEEE international conference on computer vision_ , 1968–1975.
  * Feng et al. (2023) Feng, R.; Li, C.; Chen, H.; Li, S.; Gu, J.; and Loy, C. C. 2023.  Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5013–5022.
  * Fu et al. (2017) Fu, X.; Huang, J.; Zeng, D.; Huang, Y.; Ding, X.; and Paisley, J. 2017.  Removing rain from single images via a deep detail network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3855–3863.
  * Garg and Nayar (2005) Garg, K.; and Nayar, S. K. 2005.  When does a camera see rain?  In _Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1_ , volume 2, 1067–1074. IEEE.
  * Garg and Nayar (2007) Garg, K.; and Nayar, S. K. 2007.  Vision and rain.  _International Journal of Computer Vision_ , 75: 3–27.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27.
  * Guo et al. (2022) Guo, C.-L.; Yan, Q.; Anwar, S.; Cong, R.; Ren, W.; and Li, C. 2022.  Image dehazing transformer with transmission-aware 3d position embedding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5812–5820.
  * Guo et al. (2021) Guo, Q.; Sun, J.; Juefei-Xu, F.; Ma, L.; Xie, X.; Feng, W.; Liu, Y.; and Zhao, J. 2021.  Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 1487–1495.
  * He, Sun, and Tang (2010) He, K.; Sun, J.; and Tang, X. 2010.  Single image haze removal using dark channel prior.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(12): 2341–2353.
  * He, Sun, and Tang (2012) He, K.; Sun, J.; and Tang, X. 2012.  Guided image filtering.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(6): 1397–1409.
  * Hu et al. (2019) Hu, X.; Fu, C.-W.; Zhu, L.; and Heng, P.-A. 2019.  Depth-attentional features for single-image rain removal.  In _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition_ , 8022–8031.
  * Huang, Yu, and He (2021) Huang, H.; Yu, A.; and He, R. 2021.  Memory oriented transfer learning for semi-supervised image deraining.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 7732–7741.
  * Jiang et al. (2020) Jiang, K.; Wang, Z.; Yi, P.; Chen, C.; Huang, B.; Luo, Y.; Ma, J.; and Jiang, J. 2020.  Multi-scale progressive fusion network for single image deraining.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8346–8355.
  * Kim et al. (2013) Kim, J.; Zhang, H.; André, P.; Chilton, L. B.; Mackay, W.; Beaudouin-Lafon, M.; Miller, R. C.; and Dow, S. P. 2013.  Cobi: A community-informed conference scheduling tool.  In _Proceedings of the 26th annual ACM symposium on User interface software and technology_ , 173–182.
  * Li et al. (2018a) Li, B.; Ren, W.; Fu, D.; Tao, D.; Feng, D.; Zeng, W.; and Wang, Z. 2018a.  Benchmarking single-image dehazing and beyond.  _IEEE Transactions on Image Processing_ , 28(1): 492–505.
  * Li et al. (2018b) Li, G.; He, X.; Zhang, W.; Chang, H.; Dong, L.; and Lin, L. 2018b.  Non-locally enhanced encoder-decoder network for single image de-raining.  In _Proceedings of the 26th ACM international conference on Multimedia_ , 1056–1064.
  * Li, Cheong, and Tan (2019) Li, R.; Cheong, L.-F.; and Tan, R. T. 2019.  Heavy rain image restoration: Integrating physics model and conditional adversarial learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 1633–1642.
  * Li, Tan, and Cheong (2020) Li, R.; Tan, R. T.; and Cheong, L.-F. 2020.  All in one bad weather removal using architectural search.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 3175–3185.
  * Li et al. (2022) Li, W.; Zhang, Q.; Zhang, J.; Huang, Z.; Tian, X.; and Tao, D. 2022.  Toward Real-world Single Image Deraining: A New Benchmark and Beyond.  _arXiv preprint arXiv:2206.05514_.
  * Li et al. (2019) Li, Y.; Miao, Q.; Ouyang, W.; Ma, Z.; Fang, H.; Dong, C.; and Quan, Y. 2019.  LAP-Net: Level-aware progressive network for image dehazing.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 3276–3285.
  * Li et al. (2023) Li, Y.; Ren, D.; Shu, X.; and Zuo, W. 2023.  Learning single image defocus deblurring with misaligned training pairs.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 1495–1503.
  * Li et al. (2016) Li, Y.; Tan, R. T.; Guo, X.; Lu, J.; and Brown, M. S. 2016.  Rain streak removal using layer priors.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2736–2744.
  * Liu et al. (2018) Liu, Y.-F.; Jaw, D.-W.; Huang, S.-C.; and Hwang, J.-N. 2018.  DesnowNet: Context-aware deep network for snow removal.  _IEEE Transactions on Image Processing_ , 27(6): 3064–3073.
  * Loshchilov and Hutter (2016) Loshchilov, I.; and Hutter, F. 2016.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_.
  * Oord, Li, and Vinyals (2018) Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Qian et al. (2018) Qian, R.; Tan, R. T.; Yang, W.; Su, J.; and Liu, J. 2018.  Attentive generative adversarial network for raindrop removal from a single image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2482–2491.
  * Qin et al. (2021) Qin, Q.; Yan, J.; Wang, Q.; Wang, X.; Li, M.; and Wang, Y. 2021.  Etdnet: An efficient transformer deraining model.  _IEEE Access_ , 9: 119881–119893.
  * Rai et al. (2022) Rai, S. N.; Saluja, R.; Arora, C.; Balasubramanian, V. N.; Subramanian, A.; and Jawahar, C. 2022.  Fluid: Few-shot self-supervised image deraining.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 3077–3086.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , 234–241. Springer.
  * Simonyan and Zisserman (2014) Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Sun et al. (2018) Sun, D.; Yang, X.; Liu, M.-Y.; and Kautz, J. 2018.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 8934–8943.
  * Tishby and Zaslavsky (2015) Tishby, N.; and Zaslavsky, N. 2015.  Deep learning and the information bottleneck principle.  In _2015 ieee information theory workshop (itw)_ , 1–5. IEEE.
  * Valanarasu, Yasarla, and Patel (2022) Valanarasu, J. M. J.; Yasarla, R.; and Patel, V. M. 2022.  Transweather: Transformer-based restoration of images degraded by adverse weather conditions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2353–2363.
  * Wang et al. (2023) Wang, R.; Liu, X.; Zhang, Z.; Wu, X.; Feng, C.-M.; Zhang, L.; and Zuo, W. 2023.  Benchmark Dataset and Effective Inter-Frame Alignment for Real-World Video Super-Resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1168–1177.
  * Wang et al. (2019) Wang, T.; Yang, X.; Xu, K.; Chen, S.; Zhang, Q.; and Lau, R. W. 2019.  Spatial attentive single-image deraining with a high quality real rain dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12270–12279.
  * Wang, Ma, and Liu (2023) Wang, Y.; Ma, C.; and Liu, J. 2023.  SmartAssign: Learning a Smart Knowledge Assignment Strategy for Deraining and Desnowing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3677–3686.
  * Wang et al. (2022) Wang, Y.; Yan, X.; Wang, F. L.; Xie, H.; Yang, W.; Wei, M.; and Qin, J. 2022.  UCL-Dehaze: Towards Real-world Image Dehazing via Unsupervised Contrastive Learning.  _arXiv preprint arXiv:2205.01871_.
  * Wang, Simoncelli, and Bovik (2003) Wang, Z.; Simoncelli, E. P.; and Bovik, A. C. 2003.  Multiscale structural similarity for image quality assessment.  In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, volume 2, 1398–1402. Ieee.
  * Wei et al. (2020) Wei, Y.; Liu, M.; Wang, H.; Zhu, R.; Hu, G.; and Zuo, W. 2020.  Learning flow-based feature warping for face frontalization with illumination inconsistent supervision.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_ , 558–574. Springer.
  * Wei et al. (2021) Wei, Y.; Zhang, Z.; Wang, Y.; Xu, M.; Yang, Y.; Yan, S.; and Wang, M. 2021.  Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking.  _IEEE Transactions on Image Processing_ , 30: 4788–4801.
  * Wu et al. (2021) Wu, H.; Qu, Y.; Lin, S.; Zhou, J.; Qiao, R.; Zhang, Z.; Xie, Y.; and Ma, L. 2021\.  Contrastive learning for compact single image dehazing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10551–10560.
  * Xia et al. (2023) Xia, Y.; Monica, J.; Chao, W.-L.; Hariharan, B.; Weinberger, K. Q.; and Campbell, M. 2023.  Image-to-Image Translation for Autonomous Driving from Coarsely-Aligned Image Pairs.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 7756–7762. IEEE.
  * Yang et al. (2017) Yang, W.; Tan, R. T.; Feng, J.; Liu, J.; Guo, Z.; and Yan, S. 2017.  Deep joint rain detection and removal from a single image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1357–1366.
  * Yasarla, Sindagi, and Patel (2021) Yasarla, R.; Sindagi, V. A.; and Patel, V. M. 2021.  Semi-supervised image deraining using gaussian processes.  _IEEE Transactions on Image Processing_ , 30: 6570–6582.
  * Ye et al. (2021) Ye, Y.; Chang, Y.; Zhou, H.; and Yan, L. 2021.  Closing the loop: Joint rain generation and removal via disentangled image translation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2053–2062.
  * Ye et al. (2022) Ye, Y.; Yu, C.; Chang, Y.; Zhu, L.; Zhao, X.-l.; Yan, L.; and Tian, Y. 2022.  Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5821–5830.
  * Zamir et al. (2022) Zamir, S. W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F. S.; and Yang, M.-H. 2022\.  Restormer: Efficient transformer for high-resolution image restoration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 5728–5739.
  * Zhang et al. (2023) Zhang, H.; Ba, Y.; Yang, E.; Mehra, V.; Gella, B.; Suzuki, A.; Pfahnl, A.; Chandrappa, C. C.; Wong, A.; and Kadambi, A. 2023.  Weatherstream: Light transport automation of single image deweathering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13499–13509.
  * Zhang and Patel (2018) Zhang, H.; and Patel, V. M. 2018.  Density-aware single image de-raining using a multi-stream dense network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 695–704.
  * Zhang, Sindagi, and Patel (2019) Zhang, H.; Sindagi, V.; and Patel, V. M. 2019.  Image de-raining using a conditional generative adversarial network.  _IEEE transactions on circuits and systems for video technology_ , 30(11): 3943–3956.
  * Zhang et al. (2021a) Zhang, K.; Li, R.; Yu, Y.; Luo, W.; and Li, C. 2021a.  Deep dense multi-scale network for snow removal using semantic and depth priors.  _IEEE Transactions on Image Processing_ , 30: 7419–7431.
  * Zhang et al. (2021b) Zhang, Z.; Wang, H.; Liu, M.; Wang, R.; Zhang, J.; and Zuo, W. 2021b.  Learning raw-to-srgb mappings with inaccurately aligned supervision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 4348–4358.
  * Zhang et al. (2022) Zhang, Z.; Wang, R.; Zhang, H.; Chen, Y.; and Zuo, W. 2022.  Self-supervised learning for real-world super-resolution from dual zoomed observations.  In _European Conference on Computer Vision_ , 610–627. Springer.
  * Zhu et al. (2017) Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , 2223–2232.