NSGA-II-Based Multi-Objective Optimization of Neural Networks for Nonlinear Compensation in Digital Coherent Interconnects

Abstract

This work proposes a multi-objective optimization framework using NSGA-II to tune Multilayer Percetron (MLP)based nonlinear equalizers for digital coherent optical systems. A 400 Gbps DP-16QAM link, compliant with the OIF 400ZR standard, was simulated over 120 km. Hyperparameters such as normalization, activation functions, and network size were optimized to balance bit error ratio (BER) and computational complexity measured in FLOPs. Results show that significant complexity reductions are achievable with minimal BER penalty. The study demonstrates the effectiveness of multi-objective approaches for MLP equalizer design, providing insights into optimal configurations suitable for practical high-speed optical interconnects under resource constraints.