The vibration signals of rolling bearings are inevitably influenced by noise as well as by working conditions. The use of one-dimensional original signals converted into images for rolling bearing fault diagnosis has achieved good results, but ignores the large size of the model and the speed of diagnosis, so it is not suitable for practical fault diagnosis. To address this problem, we propose a neural network based on Ghost modules and dynamic attention mechanisms. The method uses the Ghost module and coordinate attention to compress model size and reduce computational effort while improving the network's perception of the input signals. In addition, to enable efficient use of similar feature maps generated by convolution, an adaptive weighting module is proposed to further simplify the learning process and reduce network training time. The validity of the proposed method was verified on a dataset provided by Case Western Reserve University and the Mechanical Failure Prevention Technology Society. The results show that the proposed method not only has good noise immunity in noisy environments, but also has good diagnostic performance and good generalization performance under different loads. Under the same experimental conditions, the method proposed in this paper is compared with some methods and the results show the superiority of the proposed method in the diagnosis of rolling bearing faults.