Optical signal processing (OSP) technology is a crucial part of the optical switching node in the modern optical-fiber communication system, especially when advanced modulation formats, e.g., quadrature amplitude modulation (QAM), are applied. However, the conventional on–off keying (OOK) signal is still widely used in access or metro transmission systems, which leads to the compatibility requirement of OSP for incoherent and coherent signals. In this paper, we propose a reservoir computing (RC)-OSP scheme based on nonlinear mapping behavior through a semiconductor optical amplifier (SOA) to deal with the non-return-to-zero (NRZ) signals and the differential quadrature phase-shift keying (DQPSK) signals in the nonlinear dense wavelength-division multiplexing (DWDM) channel. We optimized the key parameters of SOA-based RC to improve compensation performance. Based on the simulation investigation, we observed a significant improvement in signal quality over 10 dB compared to the distorted signals on each DWDM channel for both the NRZ and DQPSK transmission cases. The compatible OSP achieved by the proposed SOA-based RC could be a potential application of the optical switching node in the complex optical fiber communication system, where incoherent and coherent signals meet.