With the rapid advances in Internet of Things (IoT) technologies, the number of smart objects connected to IoT networks is increasing day by day. Parallel to this exponential growth, attacks against IoT networks are also increasing rapidly. Various Intrusion Detection Systems (IDS) have been proposed by researchers to improve accuracy in detecting attacks with different behaviors and reduce intrusion detection time. This work presents a novel IDS based on the combination of the Principal Component Analysis and Mayfly Optimization methods (PCA-MAO) for dimensionality reduction, the Borderline Synthetic Minority Oversampling Technique (BSMOTE) for data balancing, and the Long Short-Term Memory (LSTM) method for classification. A new dataset was created by combining IoTID20, CIC-ToN-IoT and USB-IDS-1 datasets to be used in the performance test of the proposed model. Thus, the performance evaluation of the proposed model was performed for more attack types with different behaviors. As a result of classification using the proposed hybrid PCA-MAO based LSTM model, an accuracy of 99.51% was obtained. It has been observed that the proposed IDS provides superior intrusion detection performance for high-dimensional, complicated, and imbalanced data compared to classical machine learning (ML) methods.