Accurate segmentation of polyps is crucial in the field of medical image recognition. Attention mechanisms have been widely applied in medical image segmentation, but attention mechanisms implemented by convolution are limited in capturing multi-scale information due to the constraints of convolution kernel sizes. This linear aggregation method restricts the network's adaptability to various complex situations and has limitations in handling multi-scale information. Therefore, this paper proposes PASK-Net, which performs serial computations in both channel and spatial dimensions. In the channel dimension, a nonlinear approach is introduced to achieve neuron-adaptive receptive field sizes, addressing the problem of fixed convolutional processing range in attention mechanisms that hinders effective acquisition of multi-scale information. Meanwhile, in the spatial dimension, Channel-Prioritized Convolutional Attention (CPCA) is introduced to enhance the network's feature representation capability by aggregating multi-scale information from different-sized convolutional kernel branches, while ensuring computational efficiency. Comparative experiments on the Kvasir dataset demonstrate that, compared to other networks, the PASK module performs well on six evaluation metrics including Dice and IOU, achieving a Dice value of 87.54% and an IOU value of 80.6%, effectively improving the accuracy of polyp segmentation. Results from ablation experiments also validate the effectiveness of the module. The codes are available at https://github.com/LvYamKun/PASK-Net