Multi-Criteria Evaluation of Bi-LSTM Recurrent Neural Network for Nonlinearity Compensation in Digital Coherent Optical Systems

Abstract

This paper investigates Bidirectional Long Short-Term Memory (Bi-LSTM) recurrent neural networks for nonlinear compensation in digital coherent optical systems. Simulations were conducted for a 112 Gbps DP-16QAM system over a 140 km link, exploring various Bi-LSTM configurations with different memory cell and tap counts. Within the tested parameter range, results show that increasing the number of taps improves bit error ratio (BER) performance by mitigating nonlinear intersymbol interference, but also raises computational complexity. The findings emphasize the need for multi-objective optimization during the hyperparameter selection to achieve a trade-off between computational cost and nonlinear compensation effectiveness in practical optical systems.