Multichannel Nonlinear Equalization in Coherent WDM Systems based on
  Bi-directional Recurrent Neural Networks
Kerr nonlinearity in the form of self- and cross-phase modulation imposes a
fundamental limitation to the capacity of wavelength division multiplexed (WDM)
optical communication systems. Digital back-propagation (DBP), that requires
solving the inverse-propagating nonlinear Schr\"odinger equation (NLSE), is a
widely adopted technique for the mitigation of impairments induced by Kerr
nonlinearity. However, multi-channel DBP is too complex to be implemented
commercially in WDM systems. Recurrent neural networks (RNNs) have been
recently exploited for nonlinear signal processing in the context of optical
communications. In this work, we propose multi-channel equalization through a
bidirectional vanilla recurrent neural network (bi-VRNN) in order to improve
the performance of the single-channel bi-VRNN algorithm in the transmission of
WDM M-QAM signals. We compare the proposed digital algorithm to full-field DBP
and to the single channel bi-RNN in order to reveal its merits with respect to
both performance and complexity. We finally provide experimental verification
through a QPSK metro link, showcasing over 2.5 dB optical signal-to-noise ratio
(OSNR) gain and up to 43% complexity reduction with respect to the
single-channel RNN and the DBP.
Stavros Deligiannidis, Kyle R. H. Bottrill, Kostas Sozos, Charis Mesaritakis, Periklis Petropoulos, Adonis Bogris
Multichannel Nonlinear Equalization in Coherent WDM Systems based on Bi-directional Recurrent Neural Networks

###### Abstract

Kerr nonlinearity in the form of self- and cross-phase modulation imposes a fundamental limitation to the capacity of wavelength division multiplexed (WDM) optical communication systems. Digital back-propagation (DBP), that requires solving the inverse-propagating nonlinear Schrodinger equation (NLSE), is a widely adopted technique for the mitigation of impairments induced by Kerr nonlinearity. However, multi-channel DBP is too complex to be implemented commercially in WDM systems. Recurrent neural networks (RNNs) have been recently exploited for nonlinear signal processing in the context of optical communications. In this work, we propose multi-channel equalization through a bidirectional vanilla recurrent neural network (bi-VRNN) in order to improve the performance of the single-channel bi-VRNN algorithm in the transmission of WDM M-QAM signals. We compare the proposed digital algorithm to full-field DBP and to the single channel bi-RNN in order to reveal its merits with respect to both performance and complexity. We finally provide experimental verification through a QPSK metro link, showcasing over 2.5 dB optical signal-to-noise ratio (OSNR) gain and up to 43% complexity reduction with respect to the single-channel RNN and the DBP.

 Optical Fiber Communication, Nonlinear Signal Processing, Recurrent Neural Networks, Cross-Phase Modulation, Coherent Communication

## I Introduction

Communication rates exploiting optical fiber capacity has shown a remarkable exponential growth for more than 30 years, with the latest deployed systems operating even below a decibel from Shannon's limit [1, 2]. This pace is attributed to the evolution of optoelectronic devices that led to the adoption of coherent technology operating at very high bandwidth per channel and the advanced digital signal processing (DSP), in the form of probabilistic shaping [3], forward error correction codes [4] and digital compensation of the transmission impairments [5]. Periodically amplified, wavelength division multiplexed (WDM) systems covering medium to long distances are characterized by distributed non-linear effects, the most dominant of which arising from the intensity dependent refractive index (Kerr effect). The Kerr effect manifests itself as self-phase modulation (SPM) [6], cross-phase modulation (XPM) [7] or four wave mixing (FWM) [8] and is responsible for intra-channel distortions and inter-channel crosstalk. Intra-channel nonlinearity can be ideally compensated for either by digital back propagation (DPB) using the split-step Fourier method [9] or through the nonlinear Fourier transform [10].

Assuming ideal compensation of all intra-channel effects through DBP, XPM appears to be the principal source of impairments that fundamentally limits the information capacity of an optical communication system. The various techniques for nonlinear equalization treat XPM as a time-varying intersymbol interference (ISI) process and concentrate mainly in the zeroth-order XPM contribution [11, 12], known as phase and polarization-rotation noise. In this way, by tracking the temporal changes of the ISI, linear or turbo equalizers [11, 13] can partially mitigate its effects. On the other hand, higher order XPM contributions cannot be efficiently equalized unless very complex multi-channel full-field DBP [14] or Volterra equalizers [15] are employed. However, these multi-channel equalizers increase significantly the power consumption of the DSP and thus, they are considered inapplicable. The coupled-channel DBP [16] aims to reduce the complexity of full-field DBP by separately representing the WDM channels and explicitly accounting for their interaction during propagation. However, even with this simplification and the subsequent performance sacrifice, the complexity issue remains. Many works have concentrated on complexity savings by reducing the computational steps per span in DBP [17, 18], but this may be the case only for single-channel DBP, while the multi-channel implementations remain prohibitively complex. Maximum a posteriori [19] and maximum likelihood [20] decoding can also enhance the receiver performance, though they sacrifice simplicity. In recent years, machine learning (ML) techniques have penetrated the area of nonlinear channel equalization with their ability to track intersymbol dependencies [21, 22, 23]. Amongst other models, bidirectional Recurrent Neural Networks (bi-RNN) have proved efficient mitigation performance at moderate complexity [24, 25]. Furthermore, ML can also be used to improve the DBP method by
introducing a learned digital backpropagation technique that relies on parameterizing the split-step Fourier method [26].

In this work, we propose the merge of multi-channel equalization with ML in a multi-channel bidirectional Vanilla RNN (bi-VRNN) equalizer. The equalizer takes advantage of multi-input multi-output (MIMO) processing and decoding of the WDM channels. In this way, it exploits useful information from adjacent channels for a better tracking of inter-channel dependencies. Although MIMO equalization based on deep learning has been proposed in the past [27], this previous work aimed at compensating mainly for FWM at orthogonal frequency division multiplexed signals, and demonstrated marginal complexity improvement with respect to DBP. Furthermore, the complexity increased proportionally with the parallel processing of adjacent channels. Our proposition aims both at performance gains through the increase of useful information from adjacent WDM channels and complexity reduction when a multi-channel RNN equalizer is used instead of a single-channel one. We prove through extensive numerical simulations and experimental results that the proposed approach outperforms both typical multi-channel equalization in the form of adaptive equalizers and DBP and the single-channel bi-RNN, offering bit-error-rate (BER) improvement and/or Optical Signal to Noise Ratio (OSNR) gain compared to single channel bi-RNN and DBP. Moreover, we keep the complexity within competitive levels, illustrating more than 43% complexity reduction compared to to single channel bi-RNN and single-channel DBP.

The paper is organized as follows: In section II, we describe the architecture and operation principles of the RNN and particularly introduce the multi-channel equalization approach. In section III, we present simulation results that demonstrate the multichannel RNN equalizers' BER performance and show the advantages over the single channel RNN and DBP. In Section IV, we verify the improvement of multichannel RNN performance based on an experimental QPSK transmission system. Section V discusses further the aspects of computational complexity. Finally, we conclude the paper in Section VI.

## II Model Description

### _Multi-channel bi-VRNN_

In this work, the simulations consider transmission in typical single-mode fibers (SMFs) in the C-band. Since optical fiber is a nonlinear dispersive channel, we also include simulations where dispersion is lower than that of typical SMFs. Although this case is not frequently met in commercially deployed links and could be only served by non-zero dispersion shifted fibers, it is scientifically important to identify how dispersion values affect multi-channel equalization performance, especially if one takes into account the prospect of expanding long-haul transmission systems to other bands, such as O-band where lower dispersion than that of C-band is the case. Performance is evaluated by means of BER which is calculated through error counting at the receiver. The schematic of our model transmission system depicted in Fig.1, corresponds to 1200 km transmission, simulated with the integration of Nonlinear Schrodinger equation (NLSE), considering a fiber with attenuation coefficient \(a{=}0.2\) dB/km, second order dispersion coefficient \(\beta{=}\)-20ps\({}^{2}\)/km and Kerr nonlinear parameter \(\gamma=\)1.3 W\({}^{1}\)km\({}^{-1}\). Fiber propagation is modelled based on Manakov's equations using the split-step Fourier method [28]. We consider 9-channel WDM transmission on a 75-GHz wavelength grid and dual polarization 16-QAM modulation at 64 Gbaud. Lumped periodic amplification is modelled with a noise figure of 5 dB and a span length of 50 km. Prior to any post-processing or demodulation, we perform chromatic dispersion compensation with the use of an ideal frequency domain equalizer (FDE) in the case of RNN equalizer. Polarization demultiplexing and carrier synchronization are ideally carried out as well so as to focus solely on the nonlinear impairments. The simulations are conducted with pulse shaping incorporating root-raised-cosine (RRC) shaping, with a roll-off factor of 0.1 and matched-filtering at the receiver. Finally, one sample per symbol (sps) is sent to the RNN processing unit.

The layered diagram of the bi-VRNN equalizer is shown in Fig. 2 for three co-processed WDM channels. We use both polarization components of 3 (or more) adjacent channels as an input to our neural network in order to apply joint equalization of multiple WDM channels. In [29], we had used only the two polarization components of the channel

Fig. 1: The simulated 9-channel WDM transmission system containing a multi-channel coherent receiver. The DSP stage applies the two different equalizing techniques, DPB and RNN. In the case of RNN we first perform dispersion compensation using the frequency domain ideal equalizer (FDE)
to the neural network model instead, which provides less information to the equalization process. The input \(x\) to the Bi-VRNN is a \(N\)x\(L\)x12 vector representing the multi-channel 16QAM signal, where \(N\) denotes the total number of input symbols, and \(L=2k+1\) stands for the length of the input. The input \(x\) has a total of 12 features; the I and Q for both polarizations of each one of the 3 channels. At time \(t\), the input can be expressed as

\[x_{t}^{L}=\left[\begin{array}{cc}x_{t-k}\,,\ldots,x_{t-1},x_{t},x_{t+1}, \ldots,x_{t+k}\end{array}\right] \tag{1}\]

denoting that \(k\) preceding and \(k\) succeeding symbols to the current symbol \(x_{t}\) vector are used to track the inter-symbol dependencies. The length \(L\) depends on the foreseen channel memory, which relates to the accumulated dispersion in the SMF and the bandwidth limitation of the transceiver. The input \(x\) goes through the bi-VRNN with \(H\) hidden units, which process the input in both the forward and backward directions and each hidden output \(h_{i}\) is given by:

\[h_{t}=\tanh(W\cdot h_{t-1}\,+\,U\cdot x_{t}) \tag{2}\]

where the \(W\) and \(U\) matrices contain the weights of connections; \(x_{i}\), \(h_{i}\), \(h_{i-l}\) are the input, hidden output, and previous hidden output, respectively; and _tanh_ denotes the hyperbolic tangent activation function. The same output of the bi-VRNN layer, i.e., \(h_{i}\), is then sent in parallel to six (6) Fully Connected Layers (FCL), which correspond to both polarizations of each one of the 3 channels jointly detected. Each FCL consists of two-state neurons, representing the \(I\) and \(Q\) of 16-QAM and its output \(y\) is a \(N\)x\(L\)x2 vector. The FCL adopts the regression approach, which significantly reduces the implementation complexity without sacrificing BER performance, compared to the symbol-wise classification approach [29]. The bi-VRNN is trained using the _many-to-many_ approach which produces simultaneously the same number of symbols \(y\) as those at the input \(x\) and is capable of extracting multiple symbols concurrently [29]. The bi-VRNN equalizer was built, trained and evaluated in Keras with Tensorflow 2.10 GPU backend. Mean square error (MSE) and Adam were chosen as the loss function and optimizer for BER evaluation, respectively. We considered 100,000 symbols for training, 50,000 symbols for validation, and 100,000 symbols for testing with unknown data. The training stage was executed with batches of 512 words of symbols and 1000 epochs for the single channel or 2000 for the multi-channel cases. We note that the bi-VRNN model is adopted in this work, because it offers similar performance compared to other RNN models (e.g., LSTM and GRU) while exhibiting lower computational complexity [29].

### _Digital Backpropagation and linear regression_

As a well-established benchmark of the proposed RNN-based equalization system, we employ the multi-channel full field DBP along with a MIMO adaptive equalizer. It is known that an adaptive equalizer is necessary after the FDE in order to compensate for residual linear effects, while in [30], it is shown that even after DBP, an adaptive MIMO algorithm offers a slight performance improvement. The full-field DBP is considered impractical for real-life implementations as it

Fig. 3: BER as a function of launched optical power for single-channel and multi-channel processing, for RNN vs DBP in the C-band (a). In (b) RNN performance is evaluated at two dispersion regimes (b), respectively. Only the central’s channel BER performance is plotted. Adjacent channels exhibit similar performance.

Fig. 2: Diagram of the Bidirectional Vanilla-RNN equalizer in the case of joint training and equalization of 3 WDM channels
requires a receiver with the bandwidth of the entire multi-channel spectrum [14]. However, this is the optimal benchmarking scenario for the simulation results and any other DBP variant would be inferior performance-wise. In our case, for adjusting the number of channels in the back-propagated field, we simply select the channels of interest through an optical filtering process. We then sample the entire received field at 4 sps applying the DBP with 20 steps-per-span. While 2 sps and 2 or 4 steps-per-span would be more realistic, we use these unfeasible parameters in order to compare bi-RNN approach with an almost ideal, channel-aware and greedy in terms of bandwidth and simulation steps equalizer. Following the DBP, the channels are separated with digital filters, downconverted in baseband, downsampled to 1 sps. Matched filtering is applied to each one. Finally, the linear adaptive algorithm with 21 taps is implemented separately for the real and the imaginary part of each channel.

## III Simulation Results

### _Bi-VRNN versus DBP results_

In Fig. 3, the equalization results presented in the form of BER values as a function of the launched optical power are depicted after having accomplished training of the RNN model under different conditions. We consider 9-channel dense WDM transmission with 75 GHz wavelength grid for a typical SMF in the C-band (second order dispersion parameter \(\beta\)=-20 ps\({}^{2}\)/km (Fig 3.a) and a low-dispersion fiber (\(\beta_{2}\)=- 2 ps\({}^{2}\)/km) over 1200km of fiber (Fig 3.b). We compare the BER improvement offered by the multi-channel detection scheme for 3 and 5 neighboring channels versus the application of the DBP equalization technique using a corresponding number of channels. In Fig.3 only the BER of the central channel is depicted. Similar behavior is recorded for all co-processed channels. Despite the fact that the low-dispersion channel memory is much smaller than that of the typical C-band regime, for the sake of equal comparison, the length of the input word (\(L\)) is set to 51 for both bands.

The utilization of multi-channel equalization through joint multi-channel training of the RNN model is observed to deal with the inter-channel nonlinear crosstalk caused by XPM and FWM effects, as bi-VRNN exhibits better performance than in the single-channel detection. The results show that for the C-band, the minimum bit error rate (BER) for single-channel detection is 1.3x10\({}^{\text{-2}}\)@ -4 dBm, while multi-channel processing yields a minimum BER of 8.6x10\({}^{\text{-3}}\) @ -4 dBm and 7.3x10\({}^{\text{-3}}\) @ -3.5 dBm for 3 or 5 co-processed channels, respectively, as illustrated in Fig. 3a. Compared to DBP equalizers, which demonstrate improved performance with an increasing number of channels (minimum BER of 1.9x10\({}^{\text{-2}}\) @ -6 dBm, 1.1x10\({}^{\text{-2}}\) @ -6 dBm and 6.9x10\({}^{\text{-3}}\) @ -4 dBm, for 1, 3 and 5 channel respectively), the training and evaluation of 5 adjacent channels using bi-VRNN performs slightly better than the use of 3 channels in DBP which is operated almost ideally here in terms of the samples per symbol and transmission steps, as discussed in Section II. Therefore, the multi-channel bi-RNN equalizer achieves almost ideal equalization performance similar to that offered by a greedy multi-channel DBP implementation.

While the equalization performance shows a fair improvement in the high dispersion regime when multi-channel equalization is used, in the case of low-dispersion regime, the contribution of neighboring channels to the equalization performance is less beneficial as shown in fig. 3b. Thus, besides the significant impact of adjacent channels as the dispersion decreases, the coherence time of the channel also decreases resulting in inter-channel crosstalk with a high frequency content comparable to the symbol rate. In such scenarios, the RNN model is not able to learn from inter-channel interference and incorporate it into the equalization process. This finding is similar to what had been observed in [29] when the BER performance of bi-RNN was investigated at different dispersion regimes for single-channel equalization. It had been become evident that the nonlinear equalization performance of bi-RNN was superior as channel memory increased due to dispersion. Consequently, the burden of training the model with the neighboring channels in a low-dispersion environment

Fig. 4: BER as a function of launched optical power for single-channel and multi-channel processing for bi-RNN and a simplified Linear Regression equalizer (LR).

Fig. 5: BER as a function of a) bi-VRNN’s hidden units and b) training epochs
compared to the C-band scenario, results in a marginal improvement in terms of BER. In specific, a minimum BER of 1.5x10\({}^{2}\) @ -4 dBm, 1.3x10\({}^{2}\) @ -4 dBm and 1.2x10\({}^{2}\) @ -4 dBm, for 1, 3 and 5 channels respectively, was observed.

We also test the case of pure linear equalization by means of linear regression, indicatively in the C-band scenario. We consider the network of fig. 2, by completely removing the bi-VRNN layer and using solely linear regression applied to the FDE output. The effect of the linear algorithm, applying it to 1, 3 or 5 channels in a MIMO configuration is negligible, as can be seen in fig. 4, proving that bi-VRNN not only improves the BER performance in single-channel detection, but it also offers further improvement if simultaneously trained with the sequences of multiple co-propagating channels. The higher gain is observed when moving from 1 to 3 channels, than from 3 to 5 channels in the joint equalization process. This finding is reasonable as a WDM channel is mostly impaired by its two neighboring channels.

### Dependence of the RNN on hidden units and epochs

In the optimization of the RNN equalization capacity, expressed by the number of its hidden units (h.u.) \(H\), we validate the reasonable fact, that the gradual addition of h.u is necessary as the number of training channels increases. In fig. 5a it can be concluded that 16, 18 and 20 h.u are required for reaching the minimum BER performance for the single and multi-channel case of 3 or 5 channels respectively. Providing a larger network through increasing the number of h.u. does not improve the BER performance any further. Similarly, training the multi-channel network requires a larger number of epochs compared to the single-case scenario, fig. 5b. This can be explained by the fact that the additional channels increase the information at the input, and therefore a larger number of weights must be utilized to perform satisfactory equalization. This is the reason why we trained the single-channel scenario with 1000 epochs and the multi-channel scenario with 2000 epochs. It must be noted that the slight increase in h.u is not a severe shortcoming in terms of complexity as it will be shown in section V. Moreover, for static point to point WDM transmission systems where co-propagating channels remain stable over time, the substantial increase in training complexity is not a major issue, as training is not expected to take place very frequently in a real-life scenario [29].

## IV Experimental Results

### Experimental Setup

In this section we present experimental results which confirm the numerically derived findings. The experimental transmission system depicted in Fig. 6 consists of three independent transmitters, each composed of a \(\sim\)25 kHz linewidth laser, followed by IQ-modulators that were themselves driven by arbitrary waveform generators (AWGs). The end result is three channels, CH1, CH2 and CH3, sitting at frequencies of 193.5 THz, 193.6 THz and 193.7 THz, respectively. Each channel is delivering 2\({}^{16}\) symbols of uncorrelated, random data using single-polarization, 22.5 Gbit/s QPSK signaling. We carefully selected the process for providing pseudorandom unrepeated sequences using rng('shuffle') and the very long period of 2\(\times\)19937-1 Mersenne Twister generator (as in simulations), so as to avoid bi-RNN to predict the next symbol of the pseudorandom sequence and overestimate the nonlinearity mitigation results. After signal generation, the three channels are co-polarized with polarization controllers before being multiplexed using an arrayed waveguide grating and amplified with an erbium doped fiber amplifier (EDFA). An attenuator allows for the control of the power launched into the transmission link. Transmission is carried out over a part of the UK's National Dark Fibre Facility (NDFF) and comprises two 91.9 km spans of field deployed SMF28e+ fiber on the route Southampton-Reading-Southampton. The signals are amplified with an EDFA mid-link such that the total power launched into the first and the second span of SMF28e+ will be the same. The receiver contains a variable optical attenuator to facilitate receiver characterization, a demultiplexing stage to select out the channel under measurement, an EDFA to optically pre-amplify the signal and an optical bandpass filter (OBF) in order to reject

Fig. 6: The experimental transmission system

Fig. 7: Brute-force technique for temporally aligning the three independently captured (20 dBm launched power, -10 dBm receiver power). When temporal alignment was achieved, joint equalization improved BER from an average of 3.6x10\({}^{2}\) to a sharp dip of 2.5x10\({}^{2}\).
out-of-band noise. The signal is finally captured using a coherent receiver of typical construction, consisting of an optical hybrid followed by 4 balanced photoreceivers whose output is measured by a 40 GSa/s digital storage oscilloscope. The local oscillator and the carrier for each channel were sourced from the same original lasers. Once the waveforms had been captured (one at a time) for a range of powers launched into the link, they could be processed offline.

### _DSP Processing_

Prior to any nonlinear postprocessing or demodulation, we perform polarization alignement, bulk chromatic dispersion compensation with the use of a FDE and adaptive equalization using constant modulus algorithm (CMA) with 15 taps. We then perform carrier synchronization, clock extraction, and finally resampling. Eventually, one sample per symbol is sent to the bi-RNN processing unit. The only difference in the case of DBP post-processing is the removal of the FDE block, which instead is embedded in each step of the DBP process. The DBP is performed with 2 sps and 20 steps-per-span, before the CMA.

We use the 3 neighboring single-polarization channels as an input to our neural network in order to apply joint equalization. Its input \(x\) is a \(N\times L\times 6\) vector of QPSK symbols where \(N\) denotes the total number of input symbols, and \(L=2k+1\) stands for the length of the input, similarly to what we followed in the numerical analysis, with a proper modification from dual polarization 16-QAM to single-polarization QPSK. The same output of the bi-VRNN layer is then sent in parallel to 3 Fully Connected Layers (FCL), which correspond to each one of the 3 channels. Each FCL performs regression and consists of two neurons, representing the I and Q of QPSK. We considered 50,000 symbols for training, 5,000 symbols for validation, and 10,000 symbols for testing with unknown data.

The experimental process provided data for a wide range of input and received power values per channel. This data enables the direct derivation of BER performance for each channel when only linear equalization is performed or after the bi-RNN single channel equalization. Multi-channel equalization is not a trivial task to be carried out experimentally, since the data to be processed offline by the bi-RNN algorithm were collected in an asynchronous manner from the oscilloscope. In order to temporally align the three independently captured channels to the model, we had to use a time-consuming brute-force technique to identify the exact timing by performing joint equalization, initially for two out of three channels, for all possible combinations of their relative temporal position. For instance, as depicted in the example of fig. 7, by focusing on

Fig. 8: BER as a function of received optical power for single- and multi-channel processing when the transmitted power per channel is 16 dBm (a), (c) and 20 dBm (b), (d) respectively. In (a) and (b) the performance of all transmitted and detected channels is depicted, whilst in (c) and (d) the performance for RNN, DBP or without any non-linear equalization (w/o NLE) considering the central channel is depicted.

Fig. 9: Number of multiplications vs hidden units for single and multi-channel equalization.
inter-channel effects between ch2-ch3 and by applying joint bi-VRNN equalization for different time shifts, we can identify the point where the BER is evidently improved as a result of a better temporal alignment of the two channels in terms of the inter-channel effects recognized by the RNN model. This process had to be repeated several times for fine-tuning (in fig. 7 the time-shift step is 100 symbols while at a second stage shifts of up to 10 symbols were carried out for precise synchronization) and for all the operating conditions of the optical power at the input and output. Through this brute-force method it was proven that there is a relative temporal position among all channels which results in a BER improvement for multi-channel processing (fig. 7), thus experimentally confirming the capability of the bi-RNN model to track inter-channel inter-dependencies. Indicatively, at a launch power of 20 dBm and a received power of -10 dBm, we observe a noticeable decrease in the BER from an average value of 3.6x10\({}^{-2}\) in temporally unrelated channels to 2.5x10\({}^{-2}\). It must be noted that in a practical system, the detection of co-propagating channels will be synchronous, therefore the data will be directly sent to bi-RNN for training and inference without the need of temporal alignment.

### _Multi-channel versus single-channel bi-VRNN and DBP_

In Fig. 8, one can study the performance of bi-VRNN in single-channel and multi-channel equalization based on experimental data. The equalization performance exhibits BER improvement of up to an order of magnitude compared to single-channel processing. For instance, considering ch2 (central) at a launched power of 16 dBm (fig. 8a), the BER improves from 1x10\({}^{-3}\) to 4x10\({}^{-4}\). Even at the operating point of stronger nonlinear interaction (20 dBm launched power, fig. 8b), the joint equalizer can still provide a fair improvement in BER performance at all the received power values. In fig. 8a and 8b we notice that all 3 channels benefit almost equally from the joint processing. In the scenario of a launched power of 20 dBm and a received power of -10 dBm, we notice a reduction in the BER from 2.8x10\({}^{-2}\) to 1.8x10\({}^{-2}\) for Ch1 and from 2.7x10\({}^{-2}\) to 8x10\({}^{-3}\) for Ch3, respectively, upon applying single-channel and multi-channel equalization.

The fact that the single-channel RNN is very close to a DBP with multiple steps-per-span (fig. 8c-d), indicates that it constitutes a near-optimum nonlinear equalizer of intra-channel effects. The further improvement that the 3-channel RNN offers, can therefore be attributed to the efficient XPM mitigation, rather than in residual SPM compensation. In fig. 8d, for the 20 dBm launched power the optical signal-to-noise ratio (OSNR) gain compared to single-channel bi-RNN is almost 7.5 dB _@_ BER=2x10\({}^{-2}\), whilst for the 16 dBm launched power (fig. 8c), the OSNR gain is close to 2.5 dB _@_ BER=10\({}^{-3}\), indicating that the proposed equalizer performs better in highly nonlinear environments.

## V Complexity Analysis

In order to evaluate the computational complexity of multichannel equalization, we calculate the number of multiplications. The inference computational complexity, expressed as the number of multiplications following the many-to-many approach is expressed as [28]:

\[bi-VRNN_{mult}=2(FH+H^{2})L+2H\ L\ y \tag{3}\]

where \(F\), \(y\) are the input features and output vectors respectively. The parameters \(F\) and \(y\) are equal to \(2m\), since we have 2 inputs and outputs (I/Q components) for each channel, where \(m\) is the number of jointly equalized channels. We use \(H\)=16, 18 and 20 h.u. for single and joint detection of 3 and 5 neighbouring channels. Based on the fact that at least 80% of neighbouring symbols exhibit a similar BER in the many-to-many approach [29] and since we have used a training word length of 51 symbols, we utilize the 41 central ones to calculate the number of multiplications in total. In fig. 9 we plot the total number of inference multiplications for the single and the multichannel cases versus the bi-VRNN's hidden units. One can see that although the number of multiplications increases with the addition of more input/output channels (features), eventually the number of multiplications per symbol decreases drastically. We calculate 478 multiplications per detected symbol (mps) for the single channel of 16 h.u, 313 mps for 3 extracted channels of 18 h.u. (34.38% complexity reduction compared to single channel), and 299 mps for 5 extracted channels of 20 h.u. (37.5% complexity reduction compared to single channel). The complexity reduction of multi-channel equalization over the single-channel counterpart in the scenarios we have considered can reach up to 58.7% in the case of 22 h.u with 821mps for the 1-channel relative to 339 mps for the 5-channel detection (fig. 9). The interpretation of the reduction of computational complexity in the case of multichannel RNN equalizer is found in (3) which reveals that most multiplications relate to the number of h.u. In fact, by taking advantage of the structure of the network and by adding a reasonable overhead (additional input and extracting channels), we manage to reduce the DSP processing per symbol whilst significantly improving the BER. Figures 4 and 5 demonstrate that the use of more than 3 channels contributes less to the improvement of the BER. Although it is out of the scope of this study, the hardware implementation of RNN equalizers in optical communication systems is a challenge that already concerns the research community [31]. We believe that the DSP implementation of a three-channel bi-RNN equalizer requiring a rather reasonable number of h.u. is feasible. This will be the subject of future work.

Concerning the computational complexity of the RNN in the experimental setup, we calculate 796 mps for the single extracted channel with 16 h.u. and 448 mps for the 3 extracted channels with 18 h.u. which is equivalent to 43.75% complexity reduction compared to the single channel RNN.

For the DBP algorithm the complexity per bit can be evaluated according to [32]:

\[C_{DBP}\ =\ 4N_{\text{Sspsp}}N_{\text{spans}}\left[\frac{N(log_{2}N+1)n_{s}}{(N -N_{4}+1)log_{2}M}+n_{s}\right] \tag{4}\]

where \(N_{\text{Ssp}}\) is the number of spans, \(N_{\text{Sspsp}}\) is the number of steps per span, \(N\) is the FFT size (depending on accumulated dispersion per span), \(n_{s}\) are the sps, \(M\) is the constellation order and \(N_{d}=n_{\text{T}}\upsilon/T\), where \(\upsilon\) corresponds to the dispersive channel impulse response and \(T\) is the symbol duration. We multiply the complexity by 4 since for DBP we use complex numbers and one complex multiplication is equal to four real multiplications.
Considering _N_=256 and _N_=30, with 20 steps-per-span for the 2 spans, we calculate 4528 mps. Even if we reduce the steps-per-span to 4 and the size of the FFT to 128, significantly sacrificing performance, the DBP would require at least 918 mps for a single channel.

In order to achieve a fair comparison between the bi-RNN and the DBP approach, it is imperative to incorporate into the former the 81 mps necessitated by the FDE, according to:

\[C_{FDE}\ =\ 4\left[\frac{N(log_{2}N+1)n_{g}}{(N-N_{d}+1)}\right] \tag{5}\]

Taking into account FDE, the multi-channel bi-RNN approach with 18 h.u (requires 529 mps) would offer over 42,37% complexity reduction per channel with respect to a simplified DBP implementation (918mps), when 3 channels are incorporated, and 88,32% reduction compared to the greedy 20 step-per-span DBP (4528mps) considered in this paper. It is important to highlight that the computational complexity of the bi-RNN is not affected by the transmission length, whereas the DBP experiences a significant escalation in complexity with an increase in the number of spans. Additionally, both DBP and FDE equalizer require a minimum of 2 sps, in contrast to bi-RNN, which demonstrates satisfactory performance with only 1sps.

## VI Conclusion

The paper investigates a bi-RNN based equalizer for joint training and equalization of adjacent channels of a WDM coherent system. We provided simulation results considering a polarization multiplexed 16-QAM transmitter operating at 64GBd and experimental results with a WDM QPSK coherent system operating at 22.5 Gb/s. The simulation results revealed that multi-channel nonlinear equalization by means of bi-RNN is meaningful and can provide a clear performance advantage especially in high dispersion regimes. The theoretical findings were confirmed by the experimental process incorporating 3-channel equalization. The proposed multi-channel equalization scheme exhibits BER improvement of up to an order of magnitude compared to a single channel detection equalizer while reducing the computational complexity per symbol beyond 43%. We believe this multi-channel equalizer can offer a practical solution for enabling reach extension in point-to-point digital coherent transmission systems, while requiring a moderate increase in receiver complexity. Next steps of our research will be directed to the implementation of a 3-channel bi-RNN equalizer in a field-programmable gate array platform.
