This paper examines artificial intelligence (AI) methods for compensating the distortion experienced in optical communication systems resulting from fiber nonlinearity. To identify the resulting degree of improvement afforded by machine learning methods, the procedures are applied to a model of a typical single frequency optical communication system with, a 3200km fiber length, double polarization, and a 16-QAM modulation format. The performance of transmitters and receivers that incorporate Neural Networks (NNs) are in particular examined for different values of the nonlinear coefficient \(\left(\gamma \right)\). Both of these are found to improve the system Q-factor for all values of\(\gamma\) although the degree of enhancement is dependent on the signal to noise ratio. The structures studied include Siamese neural networks (SNN) implemented at the receiver end and two-stage architectures that employ NNs at the transmitter together with a classifier at the receiver side. Here classifiers ranging from simple decision tree structures to boosting, forests, extra trees, and Multi-layer perceptron (MLP) were examined and found to provide significant enhancement for \(\gamma >\) \(4{W}^{-1}k{m}^{-1}\). The optimal performance for highly nonlinear systems was achieved with two-stage systems with random forest or extra tree AI methods at the receiver. Empirical equations are also presented for each AI technique that relates the Q-factor enhancement to \(\gamma\) and the computational resource requirements (the number of included triplet terms).